feat: Initial Arrow conversion and parquet output

[paulgessinger]

Adds an Apache Arrow / Parquet output path for the examples framework, so event data (particles, sim hits, tracks) can be written to a columnar, sharded Parquet dataset and read back per-event — from both C++ and Python/pyarrow.

What

• New ActsPluginArrow plugin wrapping Apache Arrow. Arrow is linked
  with hidden visibility so none of its symbols leak across the .so
  boundary (enforced by exported-symbol lists). Two visibility-exported,
  pybind-friendly handles — ArrowSchemaHandle and ArrowTable — move
  schemas and tables across library and language boundaries via the Arrow
  C Data Interface, giving ABI-safe pyarrow interop without exposing
  arrow's own typeinfo.

• Parquet I/O (Examples/Io/Parquet):

  • ParquetWriter: sharded dataset, one directory per collection. Events
    are routed to shard files by event_id, so each shard owns a disjoint
    event-id range and footer min/max statistics stay tight — letting the
    reader prune to a single fragment per lookup. Row-group buffering bounds
    peak memory; per-collection schemas are validated on write.
  • ParquetReader: dataset reader with per-event lookup via filter
    pushdown + footer-statistics pruning, and added-column schema evolution
    against an optional target schema.

• Output converters (Examples/Io/Arrow) for particles, sim hits and
  tracks. Each builds a per-event nested table: one row per event, every
  field a list<T> whose single element holds that event's values, with
  event_id as the outer routing key.

• Python bindings for the converters, reader and writer, with the arrow
  schema bridged into Python so the producing and consuming sides share one
  schema handle.

• Wired into full_chain_odd.py (new --output-parquet flag); covered by
  test_arrow.py and an ABI-isolation test (test_arrow_isolation.py).

Why

ACTS examples lacked a standard columnar output. Parquet gives compact, typed, schema-stable files that downstream ML/analysis tooling consumes directly, and the C-Data-interface design lets the same buffers be used from pyarrow with no copies and no arrow-symbol clashes.

Blocked by:

• feat: Make ScopedTimer threadsafe #5439
• feat: add parent id to existing SimParticle EDM #5440

[github-actions]

📊: Physics performance monitoring for b74b718

Full contents

physmon summary

• ✅ Particles fatras
• ✅ Particles geant4
• ✅ Particles ttbar
• ✅ Vertices ttbar
• ✅ Truth tracking (KF)
• ✅ Truth tracking (GSF)
• ✅ Truth tracking (GX2F)
• ✅ Truth tracking (KF refit)
• ✅ Truth tracking (GSF refit)
• ✅ CKF finding performance | trackfinding | single muon | truth smeared seeding
• ✅ CKF fitting performance | trackfinding | single muon | truth smeared seeding
• ✅ CKF track summary | trackfinding | single muon | truth smeared seeding
• ✅ Seeding trackfinding | single muon | truth estimated seeding
• ✅ CKF finding performance | trackfinding | single muon | truth estimated seeding
• ✅ CKF fitting performance | trackfinding | single muon | truth estimated seeding
• ✅ CKF track summary | trackfinding | single muon | truth estimated seeding
• ✅ Seeding trackfinding | single muon | default seeding
• ✅ CKF finding performance | trackfinding | single muon | default seeding
• ✅ CKF fitting performance | trackfinding | single muon | default seeding
• ✅ CKF track summary | trackfinding | single muon | default seeding
• ✅ Seeding trackfinding | single muon | orthogonal seeding
• ✅ CKF finding performance | trackfinding | single muon | orthogonal seeding
• ✅ CKF fitting performance | trackfinding | single muon | orthogonal seeding
• ✅ CKF track summary | trackfinding | single muon | orthogonal seeding
• ✅ Seeding trackfinding | 4 muon x 50 vertices | default seeding
• ✅ CKF finding performance | trackfinding | 4 muon x 50 vertices | default seeding
• ✅ CKF fitting performance | trackfinding | 4 muon x 50 vertices | default seeding
• ✅ CKF track summary | trackfinding | 4 muon x 50 vertices | default seeding
• ✅ Ambisolver finding performance | trackfinding | 4 muon x 50 vertices | default seeding
• ✅ IVF notime | trackfinding | 4 muon x 50 vertices | default seeding
• ✅ AMVF gauss notime | trackfinding | 4 muon x 50 vertices | default seeding
• ✅ AMVF grid time | trackfinding | 4 muon x 50 vertices | default seeding
• ✅ Seeding trackfinding | ttbar with 200 pileup | default seeding
• ✅ CKF finding performance | trackfinding | ttbar with 200 pileup | default seeding
• ✅ CKF fitting performance | trackfinding | ttbar with 200 pileup | default seeding
• ✅ CKF track summary | trackfinding | ttbar with 200 pileup | default seeding
• ✅ Ambisolver finding performance | trackfinding | ttbar with 200 pileup | default seeding
• ✅ ML Ambisolver | trackfinding | ttbar with 200 pileup | default seeding
• ✅ AMVF gauss notime | trackfinding | ttbar with 200 pileup | default seeding
• ✅ AMVF grid time | trackfinding | ttbar with 200 pileup | default seeding
• 🔵️ Comparison - Truth tracking (GX2F vs KF)

[benjaminhuth]

I have some comments, but not fully finished the review. One of the main points is the question if we need to bundle the arrow-interface and the Calo EDM together in one PR

[benjaminhuth]

Hmm maybe we need to revise the flexibility of the dataset reading: a download of the ColliderML dataset with the colliderml library looks like this:

.
└── CERN__ColliderML-Release-1
    ├── ttbar_pu200_particles
    │   ├── data
    │   │   └── ttbar_pu200_particles
    │   │       └── train-00000-of-01000.parquet
    │   └── metadata.json
    └── ttbar_pu200_tracker_hits
        ├── data
        │   └── ttbar_pu200_tracker_hits
        │       └── train-00000-of-01000.parquet
        └── metadata.json

I think we cannot map this at the moment...

[murnanedaniel]

@benjaminhuth @paulgessinger

Hmm maybe we need to revise the flexibility of the dataset reading: a download of the ColliderML dataset with the colliderml library looks like this:

.
└── CERN__ColliderML-Release-1
    ├── ttbar_pu200_particles
    │   ├── data
    │   │   └── ttbar_pu200_particles
    │   │       └── train-00000-of-01000.parquet
    │   └── metadata.json
    └── ttbar_pu200_tracker_hits
        ├── data
        │   └── ttbar_pu200_tracker_hits
        │       └── train-00000-of-01000.parquet
        └── metadata.json

I think we cannot map this at the moment...

Is this a blocker though? One can output to a location then re-arrange as needed. Or do you specifically mean allowing different shard sizes?

[andiwand]

This adds quite some complexity, I remember some of it after you explained it to me. Can we summarize this in the PR description? LLM summary would be enough from my side which could also guide the review a bit

[benjaminhuth]

What flexibility is missing @benjaminhuth ?

For the record: I think I was not correct, I don't really remember what exact issue I was encountering...

[benjaminhuth]

LGTM!

[paulgessinger]

@andiwand added a PR description now.