Diagnostics output when using mesh refinement

[noreasonatall1]

Hi!

I'm new to WarpX and running some test simulations with mesh refinement. The patch is refined x8 times, so think LxL=800x800 coarse grid and 0.25Lx0.25L=1600x1600 refined parch in the centre. My Diagnostics output format is OpenPMD and for every quantity X it creates two datasets: X (800x800) and X_lvl1 (6400x6400 - padded with zeros like it's spanning the whole domain). The latter leads to the output files being way larger then they should be. How do you fix that? I just want to output quantities on their inital grids.

Here's the Diagnostics bit of the input file:

diagnostics.diags_names = diag
diag.intervals = 2000
diag.diag_type = Full
diag.fields_to_plot = Bz rho_beam
diag3.format = openpmd

Thanks in advance!

[RemiLehe]

Thanks for reporting this.

Could you let us know whether the output files are in HDF5 format (filename ends with .h5) or ADIOS format (filename ends with .bp, .bp4 or .bp5)? In principle, with ADIOS, the size on file should still be reasonable (i.e. it actually only saves data for the 1600x1600 grid, but the metadata on the shape contains information about the full refined level, even if only partially filled). openPMD-viewer combines the data automatically where there is overlap.

[RemiLehe]

OK, thanks.
Could you try to recompile WarpX with ADIOS support? In principle, ADIOS is the default format for openPMD in WarpX, assuming WarpX was compiled with ADIOS support. To enable ADIOS, you only need to make sure that the ADIOS library has been installed, before compiling WarpX. See https://warpx.readthedocs.io/en/latest/install/dependencies.html#install-dependencies

[noreasonatall1]

I work on Viking cluster which doesn't have an adios module, so it took a bit of time to install and recompile everything. I ran the script with a bp5 output and it produced 2 output folders of reasonable size ~57 MB each. Visualization in ParaView looks ok. So I guess it's an hdf5 problem?
Edit: double-checked, on the same recompiled version h5 files are still ~750 MB

[RemiLehe]

Yes, it is indeed a limitation of HDF5. Thanks for checking.

[noreasonatall1]

Do you think it can be fixed?

[noreasonatall1]

And I guess a connected question: is there a way to output a X_lvl1 array without NaN padding i.e. on it's original grid? I'm using OpenPMD-viewer in Python, and the plot_range attribute doesn't seem to do anything:

B_lvl1, info_B_lvl1 = ts.get_field(iteration=250,field='B_lvl1',coord='z',plot_range=[[-1e-6,1e-6],[-50e-6,50e-6]])
returns the same arrays as
B_lvl1, info_B_lvl1 = ts.get_field(iteration=250,field='B_lvl1',coord='z')