Repository navigation
Explore an interface description language (IDL) #71
Description
Discussed in #519
At the July 2025 design retreat, we decided to explore whether an interface description language (IDL) could be used for supporting data products that need to have multiple in-memory representations (C++, C++ SOA, Python, etc.). This issue is the parent issue for IDL explorations.
Relevant items that were discussed at the retreat:
- We want to be able to use some existing C++ types currently used in LArSoft with possibly minor modifications. This means that not all data products types will be created from a DDL and code generation.
- We want to support data-product concepts and automatic translation from one concrete type to another concrete type for the same data-product concept.
- We want to support GPU-only concrete data products.
- We believe an IDL will likely help the generation of language translators and alternative in-memory layouts.
Activity
marcpaterno commentedon Oct 27, 2025
Related Requirements
These results were captured on 2025-10-26, from the DUNE stakeholder requirements Jama site.
Data Product Representation.
Support C++ algorithms.
Support Python algorithms.
Persist user-defined metadata. This one is questionable, but it seems possible to use the IDL to define user-defined metadata structures, if they are to be written to output files in some fashion.
Framework output files readable outside the framework. DUNE has been asked to remove this requirement; as of 2025-10-26, this is still under review.
Safely executing thread-safe and non-thread-safe algorithms. I believe this means that data products given as algorithm input must be save to access simultaneously from different threads.
Specification of algorithm's CPU memory usage. This seems to imply the ability to determine the memory footprint of individual concrete data products.
Optimize memory management for data products. This seems relevant to decisions on how one data product has-some elements from another data product (e.g. a Track has-some Hits).
Efficient execution of algorithms with specific CPU memory requirements. This seems to imply the ability to determine a logical size (number of Hit objects in a collection of hits, or even in a collection of Track objects that reference those Hit objects).
Composable workflows using GPU algorithms and CPU algorithms. This implies the framework's translation from CPU- to GPU-representations and vice-versa.
I/O backend for ROOT.
I/O backend for HDF5.
Data product backward compatibility.