BTD: Values for Unfilled Lab Stations

[ax3l]

Currently, we have undefined values in BTD lab stations that are not fully filled.

E.g., the example in Examples/Physics_applications/laser_acceleration/inputs_2d_boost
will with openPMD-viewer read back NaNs in the unwritten parts of the openPMD BP output if the BTD flush is incomplete (i.e., run with fewer timesteps).

Analysis:

from openpmd_viewer import OpenPMDTimeSeries

ts = OpenPMDTimeSeries('./diags/btd/')
for it in ts.iterations:
    Ex = ts.get_field(field='E', coord='z', iteration = it)
    print (it,Ex[0])

This is no problem for matplotlib, which we use in openPMD-viewer, since it handles NaN's gracefully.
But with other plotting tools, e.g., one that @jlvay use from WARP, this can cause issues.

reporter: @jlvay
cc: @RevathiJambunathan

[ax3l]

First approach: resize the fields during output #4704 by @RevathiJambunathan and me.

This turned out to not be workable, because neither HDF5 nor ADIOS support resizing the index space in "negative" direction, which is the natural filling order for BTD.

(Side note: we can still use the resize for z-plane / recording fields #4119, because there we resize in "positive" index direction.)

If we want to re-approach this for BTD and like to resize perfectly via #4704, we would need to rotate all our data chunks before writing, so we can resize ("grow") in positive index direction. That works, but then funnily a WarpX BTD simulation will appear to "move down" in visualization tools, while a non-BTD WarpX simulation will "move up" (for the same visualization tool). Also, left-right will be swapped for everything.

[ax3l]

Next approach: keep the current full allocation of fields, but use:

• HDF5 fill values (of zero): this is a bit costly but workable, needs extension to openPMD-api, and
• ADIOS2 read by block in openPMD-viewer (already done), padding with zeros what is not present: easy fix, we currently mask with NaNs

[ax3l]

ADIOS2 fill value in openPMD-viewer: NaN -> Zero openPMD/openPMD-viewer#419

Note that we return NaNs often where technically sensible. E.g., LPA diagnostics and particle tracker return NaN if a particle selection does not apply (for a requested index). I am not yet convinced that using zero is clearer / I do not yet understand where NaNs are creating issues.
(Update: not an openPMD-viewer vis issue, but an issue with another vis tool that used openPMD-viewer data APIs to get values.)

[ax3l]

Preferred solution for BTD:
Crop away the NaN regions and resize the meta-data to what was truly written.

Might require an extra openPMD meta-data attribute to track.
(And support for variable attributes in ADIOS2.)

Could also be done at read time, based on HDF5: fill value analysis (costly?), ADIOS2: chunk analysis (extra attribute that this is a BTD data set, so we do not also chop away data for AMR levels?).

[RevathiJambunathan]

I agree for unfilled snapshots, NaNs are better than zeros.
But I like the ideal solution you posted above, to crop the domain to where the snapshots are written!
Whenever there is support for the from ADIOS/HDF5/openPMD to work that out, I ll be up for providing the necessary BTD meta-data!

[RevathiJambunathan]

For now, whatever users find useful, is good
May be, we can add a documentation about there being NaNs for unfilled snapshots, so they know what to expect. (we already have a warning message when snapshots are not going to be filled. we could include the fact that there will be NaNs in this unfilled cells of the snapshot)