Unified Snapshots API, HDF5 Compression, Dataset-Specific Backend Configuration, fully supported Variable Encoding in ADIOS2, Chunk Distribution Strategies & Performance

This release unifies the Streaming API and the Random-Access File API into a unified Snapshots API, with improved support for lazy parsing. Filtering and compression features are now also available for HDF5 (previously only ADIOS2), configured via JSON/TOML. JSON/TOML configuration can now target individual datasets for an improved tuning of compression operations. Variable Encoding is now fully supported in ADIOS2 (one ADIOS2 step per Iteration), as an alternative for File Encoding (one file per Iteration). Advanced chunk distribution strategies are now available in the API and in openpmd-pipe for data organization in parallel read operations. Multiple performance optimizations have been added throughout.

Upgrade Guide: https://github.com/openPMD/openPMD-api/blob/0.17.0/NEWS.rst
Full Changelog: https://github.com/openPMD/openPMD-api/blob/0.17.0/CHANGELOG.rst

Thanks to @franzpoeschel, @ax3l, @BenWibking, @guj, @nischild and @ikbuibui for contributing to this release!

New Contributors

• @BenWibking made their first contribution in #1798
• @ikbuibui made their first contribution in #1828

New Backends, Extension, Perf. & Memory

This release fixes backend and build bugs of the 0.16.0 release.

Thank you to Franz Poeschel @franzpoeschel and Axel Huebl @ax3l for contributing to this release!

New Backends, Extension, Perf. & Memory

ADIOS2 Joined arrays, API simplification, HDF5 subfiling, TOML backend, wildcard file extensions, Performance & Memory: This release adds support for additional I/O features in all backends, and additionally a completely new TOML backend, useful for openPMD-formatted configuration files in scientific workflows. The ADIOS1 backend, previously deprecated, has been removed in favor of ADIOS2.

The openPMD-api has been simplified by no longer requiring explicit specification for scalar components via RecordComponent::SCALAR in its object model. The addition of wildcard filename extensions (e.g. simData.%E or simData_%T.%E) simplifies the generic implementation of backend-independent logic, such as post-processing routines.

Thanks to Franz Pöschel @franzpoeschel, Axel Huebl @ax3l, David Grote @dpgrote, Davide Terzani @titoiride, Ilian Kara-Mostefa @IlianCS, Junmin Gu @guj, Luca Fedeli @lucafedeli88, Nils Schild @DerNils-git, Paweł Ordyna @pordyna and Ulrik Günther @skalarproduktraum for contributing to this release!

New Contributors

• @dpgrote made their first contribution in #1442
• @skalarproduktraum made their first contribution in #1364
• @pordyna made their first contribution in #1477
• @IlianCS made their first contribution in #1510
• @lucafedeli88 made their first contribution in #1532
• @titoiride made their first contribution in #1558

Python, ADIOS2 and HDF5 Fixes

This release fixed regressions in the Python frontend as well as the ADIOS2 and HDF5 backends.
Supported macOS versions are now 11.0+ and Python versions are 3.8+.

Thanks to Franz Poeschel, Axel Huebl, David P. Grote, Ulrik Günther and Ilian Kara-Mostefa for contributing to this release!

Build Regressions

This release fixes build regressions and minor documentation updates for the 0.15.0 release.

Thanks to Axel Huebl and Franz Poeschel for contributing to this release!

C++17, Error Recovery, ADIOS2 BP5, Append & Read-Linear Modes, Performance & Memory

This release adds error recovery mechanisms, in order to access erroneous datasets, created e.g. by crashing simulations. The BP5 engine of ADIOS2 v2.9 is fully supported by this release, including access to its various features for more fine-grained control of memory usage. Various I/O performance improvements for HDF5 are activated by default. Runtime configuration of openPMD and its backends, e.g. selection of backends and compression, is now consistently done via JSON, and alternatively via TOML for better readability. The data storage/retrieval API now consistently supports all common C++ pointer types (raw and smart pointers), implementing automatic memory optimizations for ADIOS2 BP5 if using unique pointers.

The miminum required C++ version is now C++17. Supported Python versions add Python 3.10 and 3.11.

Thanks to Axel Huebl, Franz Poeschel, Jean Luc Bez, Junmin Gu, Bernhard Manfred Gruber, and Nils Schild for contributing to this release!

Improve Series Parsing, Python & Fix Backend Bugs

This release improves reading back iterations that overflow the specified zero-pattern. ADIOS1, ADIOS2 and HDF5 backend stability and performance were improved. Python bindings got additional wheel platform support and various smaller issues were fixed.

Thanks to Axel Huebl, Franz Poeschel, and Jean Luca Bez for contributing to this release!

Increased Compatibility & Python Install Bug

This release fixes various read/parsing bugs and increases compatibility with upcoming versions of ADIOS and old releases of Intel icpc. An installation issue for pip-based installs from source in the last release was fixed and Python 3.10 support added. Various documentation and installation warnings have been fixed.

Thanks to Axel Huebl and Franz Poeschel for contributing to this release!

Read Bugs, C++17 Mixing and HDF5 Performance

This release makes reads more robust by fixing small API, file-based parsing and test bugs. Building the library in C++14 and using it in C++17 will not result in incompatible ABIs anymore. HDF5 1.10.1+ performance was improved significantly.

Thanks to Axel Huebl, Franz Poeschel and Jean Luca Bez for contributing to this release!

Various Reader Fixes

This releases fixes regressions in reads, closing files properly, avoiding inefficient parsing and allowing more permissive casts in attribute reads. (Inofficial) support for HDF5 vlen string reads has been fixed.

Thanks to Franz Poeschel and Axel Huebl for contributing to this release!