JSON file backend reserializes whole file for each call to storeChunk()

[dantargz]

Describe the bug
If JSON backend is used, and calling code makes N calls to storeChunk() and then calls flush(), the produced JSON file will be deleted and rewritten to N times. Expected behavior is one JSON serialization per flush() call regardless of the number of storeChunk() calls.

To Reproduce

Build & run WarpX against the laser accelerator example with default build options

1. Get WarpX source
2. Compile with default configuration (description here)
3. Run WarpX against the 3d laser accelerator example warpx.3d inputs_test_3d_laser_acceleration
4. Observe that it takes 60 seconds (or longer) to generate a ~50MB JSON file with ~2 million doubles

Expected behavior
Expected behavior is one JSON serialization per flush() call regardless of the number of storeChunk() calls.

Software Environment

• version of openPMD-api: 0.16.1
• installed openPMD-api via: WarpX cmake build system dependency
• operating system: OSX Sequoia 15.6.1
• machine: Mac Book Pro 2024 / M4 Pro
• name and version of Python implementation: N/A
• version of HDF5: N/A
• version of ADIOS2: N/A
• name and version of MPI: N/A

Additional context
Removing the call to putJsonContents(file); in JSONIOHandlerImpl::writeDataset mostly solves the issue -- the JSON file is still re-serialized about 8 times per time the WarpX application goes to serialize, but that is a significant improvement over 1000+ it currently does. Adding a configuration option or flag for "only serialize to JSON on flush" would be acceptable for my use case.

Code flow:

• WarpX stores data in openPMD via storeChunk/storeChunkRaw calls (I think the main caller is here https://github.com/BLAST-WarpX/warpx/blob/development/Source/Diagnostics/WarpXOpenPMD.cpp#L942) -- the data is not all in a single contiguous buffer for particle data in default mode for many simulations
• Each call to storeChunk creates a WRITE_DATASET IOTask
• WarpX calls openPMD flush()
• flush() iterates through IOTasks and handles each one-by-one
• When handling WRITE_DATASET IO tasks, it eventually invokes JSONIOHandlerImpl::writeDataset
• Each call to JSONIOHandlerImpl::writeDataset re-serializes entire full json file by invoking putJsonContents

[franzpoeschel]

We recently fixed a similar issue in #1704, but that is already part of the 0.16.1 release which you use. I will take a look.

[franzpoeschel]

Thank you for narrowing down this behavior; yes, those calls should have never been part of writeDataset() (and some other methods) in the first place.

the JSON file is still re-serialized about 8 times per time the WarpX application goes to serialize

That is probably the number of calls to Series::flush() then? Our JSON backend has never been greatly optimized, so we currently have no distinction if a flush to disk is truly necessary or if a flush to internal buffers would suffice. Since we can (at least easily) write the current version of a JSON dataset to disk only by entirely replacing the existing file, that means one rewrite per flush and file.

Options for a finer distinction might include:

1. A preferred_flush_target option similar to what we have implemented for ADIOS2 with possible values buffer and disk
2. Adding such an optimization to the newly-added WRITE_LINEAR access mode (wherein Iterations are not randomly-accessed, but written one after another): Data would be dumped to disk upon ending an Iteration

Fix is in #1782

[dantargz]

the JSON file is still re-serialized about 8 times per time the WarpX application goes to serialize

That is probably the number of calls to Series::flush() then?

Makes sense, I will investigate options on the WarpX side. I will circle back if it doesn't seem fixable in that code.

Fix is in #1782

Thank you! Removing these extra serializations makes running the WarpX examples much faster.