Unlike previous engines in ADIOS2, BP5 can avoid copying data to internal buffers if one uses an optimized workflow. The following sequence of operations should be avoided:

engine.Put(dataPtr, …, adios2::Mode::Deferred);
// …
engine.PerformPuts(); // this provokes a copy to internal buffers
// …
engine.EndStep();

This PR uses two optimizations to avoid that workflow:

1. Use the Span API of openPMD/ADIOS2. Instead of creating buffers inside WarpX, this asks the IO backend to create a buffer for us to write into it. The problem of data copies is avoided entirely since the data is created at the right place already.
   This optimization benefits all ADIOS2 engines that support the Span API (e.g. BP4), openPMD-api automatically switches to a fallback otherwise.
2. In other places, the data already exists in WarpX in the right format as static data, or needs to be downloaded into pinned memory which is not supported by the Span API. The "normal" storeChunk/Put API must be used, so the above workflow should be avoided. Do this by deleting instances of Series::flush() between RecordComponent::storeChunk() and Iteration::endStep().

Notes:

• For creating a memory profile and seeing if this PR makes a difference: https://github.com/KDE/heaptrack
• Optimization (2) is hard to maintain. There is nothing that stops someone to put a Series::flush() somewhere in these routines at a later point, this way destroying the optimization again.
  Suggestion: Add something like Series::promiseNoFlushesUntilEndstep() to the openPMD-api that throws an error if a flush does indeed happen.
• Another idea we had was to add a unique_ptr overload to RecordComponent::storeChunk(). This way, the backend knows that it is the unique owner of the data and can autonomously decide to delay showing the data to ADIOS2 until directly before EndStep. This is semantically only applicable to the pinned memory instance here in WarpX, the other places are static data, not unique data.

Question to Axel: The following shared_ptr does correctly handle destroying the pinned memory again, right? There is no custom destructor given, so I'm wondering.

#ifdef AMREX_USE_GPU
                if (fab.arena()->isManaged() || fab.arena()->isDevice()) {
                    amrex::BaseFab<amrex::Real> foo(local_box, 1, amrex::The_Pinned_Arena());
                    std::shared_ptr<amrex::Real> data_pinned(foo.release());
                    amrex::Gpu::dtoh_memcpy_async(data_pinned.get(), fab.dataPtr(icomp), local_box.numPts()*sizeof(amrex::Real));
                    // intentionally delayed until before we .flush(): amrex::Gpu::streamSynchronize();
                    mesh_comp.storeChunk(data_pinned, chunk_offset, chunk_size);
                } else
#endif

Close #3133