Current best practice for block structured AMR

[pgrete]

I'm in the process of adding support for OpenPMD output to our block structured AMR code https://github.com/parthenon-hpc-lab/parthenon

In light of the open PR on the standard for mesh refinement openPMD/openPMD-standard#252 I'm wondering what the current best practice is (also with regard to achieving good performance at scale).

In our case, each rank owns variable number of meshblocks (though their size is fixed) with a (potentially variable) number of variables and at variable levels (depending on the chosen ordering of blocks).

The most straightforward approach approach would be to create one mesh record per bock and variable.
Alternatively, I imagine pooling record by level (so that the coordinate information is shared wrt to dx) to increase the size of the output buffer.
Are there other approaches/recommendations?

And what's the impact on performance when we write one chunk per block (which at the api level would effectively be a serial "write" as each block/variable combination is unique)?
Are the actual writes on flush optimized/pooled/...?

For reference, our current HDF5 output wrote the data of all blocks in parallel for each variable with the corresponding offsets.
The coordinate information was stored separately, so that this large output buffer didn't need to handle different dx.
This approach is currently not compatible with the OpenPMD standard for meshes with varying dx as each record has a tight connection to fixed coordinates, correct?

Thanks,

Philipp

Software Environment:
Have you already installed openPMD-api?
If so, please tell us which version of openPMD-api your question is about:

• version of openPMD-api: [0.15.2]
• installed openPMD-api via: [from source]

[BenWibking]

I think all existing AMR codes that use openPMD have one field per variable per AMR level. This is true for WarpX, Quokka, and CarpetX (e.g., https://github.com/quokka-astro/quokka/blob/development/src/openPMD.cpp#L108 and https://github.com/eschnett/CarpetX/blob/main/CarpetX/src/io_openpmd.cxx). There's a full list of simulation codes here: https://github.com/openPMD/openPMD-projects?tab=readme-ov-file#scientific-simulations

Because ADIOS2 has built-in sparsity support, any part of the domain that is not covered by grids can simply be left un-written without any penalty.

AFAIK the writes will be pooled on flush in order to minimize filesystem metadata overhead (this is what the docs say: https://openpmd-api.readthedocs.io/en/latest/usage/firstwrite.html#flush-chunk)

Since I had discussed this with @ax3l awhile ago re both Quokka and Parthenon, he might be able to chime in, particularly what the analysis tools assume about AMR ;)