Skip to content

[Draft] Prototype TransportFrameData backend #938

Draft
X0708a wants to merge 1 commit intoAIDASoft:masterfrom
X0708a:draft-transport-framedata
Draft

[Draft] Prototype TransportFrameData backend #938
X0708a wants to merge 1 commit intoAIDASoft:masterfrom
X0708a:draft-transport-framedata

Conversation

@X0708a
Copy link

@X0708a X0708a commented Feb 24, 2026

Summary

This draft PR explores implementing a minimal FrameData backend to
validate whether frame-level transport can live at the FrameData layer
without modifying Frame or existing writers.

Scope

  • Minimal TransportFrameData class implementing the required interface
  • getIDTable(), getCollectionBuffers(), getAvailableCollections(), getParameters()
  • Compiles cleanly against core podio
  • Verified that a podio::Frame can be constructed from this backend without modifying Frame

Out of scope

  • Writer side (intentionally excluded — writers currently assume fully
    materialized collections as noted in Frame serialization/deserialization #565)
  • Actual serialization format
  • Network transport implementation

Relation to #565

This prototype is motivated by the discussion in #565 and the recent
suggestion that the reading side may already be solvable via arbitrary
FrameData construction. This is an attempt to validate that boundary.

Open question

Should getCollectionBuffers() eventually own the Arrow/transport buffers
directly, or should a separate buffer ownership layer sit between the
transport and CollectionReadBuffers?

Feedback welcome on the architectural direction before going further.

@andresailer andresailer marked this pull request as draft February 24, 2026 15:34
@tmadlener
Copy link
Collaborator

From a quick glance this looks good from an interface point of view. It should be possible to construct a Frame with this, have you tried?

Should getCollectionBuffers() eventually own the Arrow/transport buffers
directly, or should a separate buffer ownership layer sit between the
transport and CollectionReadBuffers?

That is actually a good question to which I don't have a clear answer yet, resp. which will probably require some evaluation.

Currently the situation is as follows:

  • When reading the FrameData actually take ownership of the data that is provided by the reader. If a user wants to access a collection via the Frame that has been constructed with this FrameData, the Frame takes ownership (and the collection data is in fact removed from the FrameData). The FrameData also ensures that any unconsumed collection data are destroyed once it is destroyed (see Make it possible to dispose of unconsumed collection buffers #502)
  • When writing the ownership of the buffers (that are created from the collection) stays with the Frame and the Writer only gets what is essentially a view to them.

So at least at the moment it looks taking ownership is necessary.

@X0708a
Copy link
Author

X0708a commented Feb 24, 2026

Yes , I have verified that a podio::Frame can be constructed from TransportFrameData, and the standard lazy materialization path works as expected.
After reviewing the current read-side flow (e.g. ROOTFrameData and the FrameModel::doGet path), it seems clear that FrameData is expected to own raw buffers until they are handed out via getCollectionBuffers(), at which point ownership effectively transfers to Frame.
Given that, it makes sense that a future TransportFrameData implementation should mirror this behavior and take ownership of the transport buffers so that it fits cleanly into the existing ingestion contract without introducing a new lifetime model.
I agree that any broader symmetry with the writer side would likely require separate evaluation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants