Series Open Performance

[ax3l]

We should investigate and profile our performance of Series open (especially with READ_ONLY and READ_WRITE, but also with CREATE on truncation: #483).

Most crucial with file-based series, I have the subjective feeling they are slower than they need to be, already in serial mode.

We might need to clearly define also in this step, what we check and assume when opening a series (aka contract). The less assumptions the better, if you ask me. For example, I would not expect the data of each iteration nor it's names or layouts to be required to match. (Although some applications, such as our openPMD-viewer GUI, will benefit if we later on provide an auxiliary scan-like list summary for records in all iterations.)

Resources

Maybe some useful resources:

• scandir() in ag
• benchmarking idea
• tracefile (strace maybe expand to measure time spend; or even easier: add time log [WIP] spdlog for multi-level multi-sink logging #462 to profile our auxiliary::Filesystem operations)

[franzpoeschel]

I have noticed a similar potential for optimization while working on streaming mode. This issue is very relevant in this scenario, since in streaming mode the Series (or at least parts of it) need to be re-read after a new packet has arrived.
Often enough, this incurs a much more massive performance penalty than moving the actual datasets which is hardly tolerable.

A first step that could massively help boost reading performance (or maybe hide it, rather) would be to read in a Series lazily. For example, upon opening a Series object, only check what iterations are present, but do not continue parsing the iterations recursively until one is actually opened. Each application would then ideally only parse those parts of the openPMD tree that are of interest for it.

[nischild]

I also encountered a performance problem when opening a Series in READ_WRITE mode multiple times in a single program.

In my implementation a Series is destroyed after diagnostic data has been written to file after an iteration.
In the next iteration a new series is opened. The time required to open a Series increases every time a new Series is created by opening an existing h5-file. This continues until the walltime of an iteration is dominated by opening the Series.

I use the HDF5 backend. An excerpt of my measurement with the timer placed around the series constructor

Operation t [s]
Create: 0.000638468
Open: 0.00791107
Open: 0.0107065
Open: 0.0141802
Open: 0.0196517
Open: 0.0209107
Open: 0.0401462
Open: 0.0274309
Open: 0.0306679
Open: 0.0341617
Open: 0.037439
Open: 0.0408647
...

[franzpoeschel]

@DerNils-git Can you share the part of your code that deals with opening and closing the Series?

My suspicion on what is happening here:

You repeatedly close and open the same Series. When e.g. using Series("data_%T.h5, Access::READ_WRITE) to write iterations, each iteration will be written to separate files data_0.h5, data_100.h5, ...
When you open the Series again, all of these files are parsed again. So, when more and more files are created, that parsing step takes longer and longer.

In this case, the easiest solution would be:

• to simply keep the Series open, there is (from our side) no need to close it.

Alternatively, you could:

• either use Access::APPEND if you wish to only append new iterations and do not need to modify old ones. The parsing step is skipped in that case. This is also more portable, since ADIOS2 supports appending, but not modifying.
• or use Series("data_%T.h5", Access::READ_WRITE, R"({"defer_iteration_parsing": true})") to not read all iterations upon opening (the third parameter is a JSON-formatted string). In order to access existing iterations, you will need to first call Iteration::open() then.

[nischild]

@DerNils-git Can you share the part of your code that deals with opening and closing the Series?

Unfortunately not yet.

You repeatedly close and open the same Series. When e.g. using Series("data_%T.h5, Access::READ_WRITE) to write iterations, each iteration will be written to separate files data_0.h5, data_100.h5, ...

Yes that is what's happening. (But using group-based encoding).

What does parsing means in this context specifically? I would not expect reading the meta-data of a Series would be that expensive for a few hundred iterations.

I am testing the second solution you gave in alternatives.
Parsing around a single Series does not seem to work since MPI processes interacting with the file change depending on the output data.

[franzpoeschel]

(But not using group-based encoding).

My example above assumes that you use file-based encoding. Is this assumption correct?

I would not expect reading the meta-data of a Series would be that expensive for a few hundred iterations.

Reading the metadata is likely not that expensive, opening a few hundred files from the file system for each newly created file is.

[nischild]

My example above assumes that you use file-based encoding. Is this assumption correct?

No sorry. Above was a type, which I removed. I use group-based encoding. (At least I assume since it is the default encoding in the documentation.) So it is only a single file.

Using R"({"defer_iteration_parsing": true})" gives a better performance but results in the same problem. So it still does not solve the problem.

[franzpoeschel]

If you are measuring an improvement of performance, but still see the underlying problem, then I assume that the following happened:

1. There is barely any metadata parsing happening in openPMD now any longer with deferred iteration parsing, explaining the improved performance.
2. Opening a huge (by amount of metadata) HDF5 file still has performance implications inside HDF5.

I'd assume that the defer_iteration_parsing option helped optimize away the parsing time of openPMD, but you still see the file-opening and parsing time of HDF5. (Note that I'm no expert in HDF5 tuning, so this is mainly an assumption on my side)

If you want to investigate this further, I suggest using Google Perftools https://gperftools.github.io/gperftools/cpuprofile.html
This profiler can be used without recompiling, by just LD_PRELOADing it into the runtime environment.

On the other hand, it's not really recommended to put a very large amount of iterations into a single file. If any data corruption happens anywhere, then the entire dataset is affected. In your workflow, it's worth considering if file-based encoding makes more sense.

[ax3l]

As discussed, the code should be here now:
https://gitlab.mpcdf.mpg.de/bsl6d/bsl6d/-/tree/master/src/io/data

[ax3l]

On the other hand, it's not really recommended to put a very large amount of iterations into a single file. If any data corruption happens anywhere, then the entire dataset is affected. In your workflow, it's worth considering if file-based encoding makes more sense.

This depends. For ADIOS2, this is no problem and will not corrupt written data.

For HDF5, there is a similar journaling mode. It seems it was slated for 1.10+ but might have arrived later:
https://stackoverflow.com/questions/31287744/corrupt-files-when-creating-hdf5-files-without-closing-them-h5py
I have to check if that is the same as the SWMR mode, I think that is what I used in the past. Will check with @jeanbez