openPMD Output

[ax3l]

Implement a new diagnostics element that writes out openPMD data.

Close #98

Checklist

• build logic
• openPMD element:
  • C++
  • Python
  • docs
• configure options:
  • file format / backend
  • iteration encoding
• GPU pinned memory support instead of managed/GPU pointers
• CI: w/ HDF5 CI: Clean Up & HDF5 #324 and/or ADIOS2
• remove ASCII draft diagnostics that wrote the full beam
• update examples to use openPMD, esp. in analysis & plot scripts
• update Python examples requirements.txt and other script & deploy logic to request openPMD-api
• update cluster docs to load HDF5 and/or ADIOS2 modules
• match to MAD-X monitor: https://mad.web.cern.ch/mad/webguide/manual.html#Ch11.S19
  • technically, this is meta-data w/o particles

Performance Measurement

Measured on my laptop, 12th Gen Intel(R) Core(TM) i9-12900H processor & PM9A1 Samsung 2TB SSD drive.
Software: GCC 11.3.0, MPICH 4.0.3, ADIOS2 2.8.3.

Running all ImpactX tests.

# for PR only: patch example I/O backend to ADIOS2 BP (our first CI version uses HDF5)
git grep h5 examples/ | awk -F: '{print $1}' | xargs -I{} sed -i 's/h5/bp/g' {}
sed -i 's/h5/bp/g' src/python/impactx/madx_to_impactx.py

# build
cmake -S . -B build -DImpactX_PYTHON=ON
cmake --build build -j 12

# run
ctest --output-on-failure --test-dir build/ -E AMReX -j 6

Ran 90 test targets each (incl. analysis & plotting):

• prior to this PR: 59.14 sec
• now, with ADIOS2 openPMD: 26.03 sec
• now, with HDF5 openPMD: 27.33 sec

Follow-Up

• more docs on the output format
  • as in WarpX to https://impactx.readthedocs.io/en/latest/dataanalysis/dataanalysis.html
  • example: simple way to export particle data from openPMD via pandas to ASCII table
• run openPMD-validator on created files Fix openPMD Meta-Data #349
• add reference particle
• add more meta-data
  • add reduced diagnostics
• configure options:
  • data and engine options, e.g., compression: getSeriesOptions
  • s or t and do transformation if needed
  • filter
  • disable particle output (only meta-data)
  • disable meta-data (only particles)
• new openPMD-api features:
  • span-based API to reduce memory footprint Optimizations for BP5 engine of ADIOS2: Use span-based API and avoid flushing before closing an Iteration BLAST-WarpX/warpx#3200
  • use steps to reduce memory footprint openPMD: Use Steps #337
  • Series::close() from 0.15.0+

[ax3l]

Can use span-based API here:

• Optimizations for BP5 engine of ADIOS2: Use span-based API and avoid flushing before closing an Iteration BLAST-WarpX/warpx#3200

[cemitch99]

I have two questions about this element. First, is it useful to include an ascii option for backward-compatibility? (Some users are simply more comfortable with plain ascii text.) Second, should we include the option of specifying a filename for output?

[ax3l]

I think no and yes :-) No to the first part for heavy particle data, because it's not scalable and what counts for users is mainly an API to read the data in their scripting language of choice. Also lacks a ton of meta-data that we want to preserve.

We can document an easily usable export to ascii table nonetheless - but in post.
It's a one liner via .to_csv() in Python:
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html

Let's discuss tomorrow :)

[cemitch99]

Looks good to me. I made some comments to follow-up on later.