Define interaction between core framework and IO backends #54

Activity

1. knoepfel

   self-assigned this

   on Jul 30, 2025
2. knoepfel

   added

   F1 milestoneItem necessary for achieving F1 milestoneItem necessary for achieving F1 milestone

   on Jul 30, 2025
3. knoepfel commented on Aug 25, 2025

   

   knoepfel

   on Aug 25, 2025

   MemberAuthor

   More actions

   Terminology

   The Phlex type includes:

   • Creator name (CHOF name)
   • Product name
   • Programming type [or should this be the data-product concept?]
   • Phase name (equivalent to art's process name)
   • Data layer(s)

   The Phlex type does not contain the data-cell index, but only the data layer(s) to which the data product belongs.

   Note

   A data product is unambiguously identified by its data-product identifier, which is its data-cell index and its Phlex-type instance.

   Preserving data

   • FORM will receive data along one or more graph edges from HOF graph nodes—i.e., there is no explicit form::write call in a (e.g.) transform node.
   • FORM will not write placeholder/null objects to (e.g.) RNTuples whenever an algorithm did not execute.

   FORM must store:

   1. Data products, and their product identifiers, which include the data-cell index and the Phlex type instance
   2. The provenance associated with each data product
   3. The data-layer hierarchy of the data (which may be calculable)
   4. The configuration used in producing a framework file

   Providing data

   FORM must be able to provide:

   1. Data products, and their product identifiers, which include the data-cell index and the Phlex type instance
   2. The provenance associated with each data product
   3. The data-layer hierarchy of the data in a framework file
   4. A list of the data products (and their associated data cells/layers) in a framework file
   5. The configuration used in producing a framework file
4. gemmeren

   self-assigned this

   on Aug 25, 2025
5. knoepfel commented on Aug 25, 2025

   

   knoepfel

   on Aug 25, 2025

   MemberAuthor

   More actions

   API for data exchange

   1. A user-defined data product of a specific type T will be owned by an object phlex::product<T>, which inherits from an abstract base class phlex::product_base. To preserve a data product, the core Phlex framework will pass to FORM (along a graph edge) an object whose type is the equivalent of std::shared_ptr<phlex::product_base>. To provide a data product, FORM will create an object of type std::shared_ptr<phlex::product_base> to give to the core framework.

      1. For algorithms that return multiple data products per invocation, it may be most efficient to pass a single set of std::shared_ptr<phlex::product_base> objects from the core framework to FORM.

   2. Data-product provenance will be accessible through objects of the type phlex::provenance, whose API and layout have yet to be determined.

   3. The following file-level entities do not have an obvious data-exchange API between the core framework and FORM:

      1. Job configuration
      2. Data-layer hierarchy
      3. List of data products (and their associated data cells/layers)

      Such an API will likely be developed later.
6. knoepfel

   added

   F2 milestoneItem necessary for achieving F2 milestoneItem necessary for achieving F2 milestone

   on Aug 25, 2025
7. knoepfel

   added this to the F1: Prototype release milestone on Sep 16, 2025
8. knoepfel commented on Sep 16, 2025

   

   knoepfel

   on Sep 16, 2025

   MemberAuthor

   More actions

   We are marking this issue as complete for the F1 milestone. The interaction between the core framework and IO backends will continue to be refined as part of the F2 milestone.
9. knoepfel

   closed this as completedon Sep 16, 2025