v2.11.0

What's new in 2.11?

This is a major release with new features and lots of bug fixes. The main
highlights include enhanced derived variables, remote data access, GPU
improvements, campaign management overhaul, and advanced compression operators.

Summary

Derived Variables

Support for reader-side computed variables with expression evaluation. This
release adds comprehensive functionality including new mathematical operations
(trigonometric functions, multiplication, division, power) and the
ability to apply derived variables to aggregated arrays.

Remote Data Access Infrastructure

Implementation of XRootD and remote server capabilities enabling distributed
file access. This includes support for remote streaming, metadata caching, and
optimized query handling across networked systems.

GPU and Memory Space Improvements

Enhanced GPU backend support with memory selection adjustments, Kokkos
integration improvements, and bindings for cuPy pointers and torch tensors.
Addresses layout mismatches between user code and device memory.

Campaign Management System Overhaul

Significant updates to campaign tracking using SQLite databases replacing JSON
files, configuration file support, and attribute handling. Includes multi-host
support and improved metadata organization.

CampainReader is a new engine to read .aca files created by hpc-campaign 0.6.
https://adios2.readthedocs.io/en/latest/advanced/campaign_management.html, https://hpc-campaign.readthedocs.io

Compression and Data Format Advances

New operators including BigWhoop compression and MGARD for compressing complex arrays. ZFP now
supports 4D arrays, and improved MGARD efficiency through zstd compression of
lossless data portions.

OpenSSF Best Practices

ADIOS2 now adopts OpenSSF (Open Source Security Foundation) best practices
including enhanced security measures, automated vulnerability scanning, and
adherence to industry-standard development practices.

Other features

• BP5 has a new aggregator, Data size based aggregation, that performs best for highly imbalanced IO situations, i.e., when different processes produce wildly different amount of data.
  https://adios2.readthedocs.io/en/latest/advanced/aggregation.html
• TimeSeries is an engine to handle a series of individual BP files as one. Enumerate entries in .ats YAML file.

What's Changed

• python,cmake: update version script to support rc by @vicentebolea in #3979
• BP5internal - Switch to name based record lookup by @eisenhauer in #3981
• Fix compilation errors on strict compiler by @eisenhauer in #3983
• Update the tutorial with links to external tutorials by @anagainaru in #3978
• ASAN cleanup by @eisenhauer in #3982
• Fix read request length on BP5 Local arrays with operators by @eisenhauer in #3989
• Backport master 3980 3994 by @pnorbert in #3996
• Backport 3984 to master: Python API: Adios() new argument 'config_file', Stream() new argument… by @pnorbert in #3999
• update CODEOWNERS by @vicentebolea in #4003
• update readthedocs deps by @vicentebolea in #4001
• Merge upstream dill changes with master by @eisenhauer in #4006
• Implementing sqlite database instead of json file. Writing part. by @dmitry-ganyushin in #4000
• A simpler way to deal with the mismatching layouts between the user code and the memory space by @anagainaru in #3976
• fix BP5 parameter parsing for size values with units by @pnorbert in #4012
• Merge 3769 to master by @pnorbert in #4011
• Replace the only PERFSTUBS_SCOPED_TIMER_FUNC() call in BP5 to get rid… by @pnorbert in #4013
• Python updates by @pnorbert in #4014
• FFS upstream by @eisenhauer in #4016
• Posix transport should fail if you try to read past EOF by @eisenhauer in #3997
• Make POSIX transport work on Windows, default to it by @eisenhauer in #4023
• Campaign fix by @dmitry-ganyushin in #4024
• readthedocs: enable query docs by @vicentebolea in #4030
• Don't force initialization for buffers whose contents we will immediately destroy by @eisenhauer in #4031
• fix IO.available_attributes(), remove a debug print from Stream, fix … by @pnorbert in #4021
• campaign management: update sqlite database after another run by @dmitry-ganyushin in #4038
• Fixed campaign python script. by @dmitry-ganyushin in #4040
• Fix Windows Testing build by @eisenhauer in #4027
• Tweak POSIX fail on EOF to kill heisenbug by @eisenhauer in #4035
• ci: add macos static build by @vicentebolea in #4041
• Python write scalars by @pnorbert in #4036
• Add a static windows build to CI by @eisenhauer in #4037
• Add arguments to IO.AvailableAttributes() and io.available_attributes… by @pnorbert in #4043
• Return a 0-dim numpy array of size 1 when reading a scalar variable … by @pnorbert in #4044
• Add timeout parameter to stream.steps(). Necessary for controlling ti… by @pnorbert in #4045
• blosc2: do not export blosc2 lib by @vicentebolea in #4055
• Fix segfault when asking for BlocksInfo for a variable name that does… by @pnorbert in #4047
• Reading of campaign configuration parameters from ~/.config/adios2 by @dmitry-ganyushin in #4048
• do not flush io and adios in read mode. BP5 reader does not like it. by @pnorbert in #4065
• Allow plugin operators to take advantage of the estimated size API by @rupertnash in #4046
• Add minmax and shape functions to CampaignReader, so that per-block i… by @pnorbert in #4067
• fixes to still be able to build with gcc 4.8.2. by @pnorbert in #4063
• added support to read back from H5T_STRING VARIABLES by @guj in #4060
• Using the correct flag to detect the CUDA backend in ZFP by @anagainaru in #4066
• Fix Reord to use MinBlocksInfo where appropriate by @eisenhauer in #4071
• • Restructure python API doc in separate main topic, add working exam… by @pnorbert in #4075
• C bindings for the memory space related functions by @anagainaru in #4074
• Actually look at return value for read in remote_server by @eisenhauer in #4078
• Fortran bindings for memory space related functions by @anagainaru in #4077
• Raise an exception if remote open fails by @eisenhauer in #4069
• Python: add the same treatment to attributes as to variables before: … by @pnorbert in #4076
• Update documentation for 2.10 changes to the GPU-backend by @anagainaru in #4083
• BlockIndex.Evaluate() made branch based on BP5 vs BP4. To support Cam… by @pnorbert in #4084
• Fix typo in the links to external tutorials by @anagainaru in #4086
• adios2_define_derived_variable C/Fortran API. C is compiled condition… by @pnorbert in #4088
• Add setup for Aurora (load adios2 as e4s package) by @pnorbert in #4089
• Do not create adios-campaign/ unless there is something to record by @pnorbert in #4096
• Bison 3.8 Parser by @lizdulac in #4062
• WIP: Make Fortran tests fail with a non-zero exit code by @eisenhauer in #4097
• Fix typo in Fortran API docs by @JamieJQuinn in #4102
• ci: add ccache job summary by @vicentebolea in #4101
• Fix static blosc2 build by @vicentebolea in #4093
• Don't run derived test in MPI mode, it's not written for that by @eisenhauer in #4104
• Pull in DILL Upstream that avoids libFFI except on demand. by @eisenhauer in #4091
• Update campai...

v2.11.0-rc1

What's new in 2.11?

This is a major release with new features and lots of bug fixes. The main new feature is ...

New/updated notable features

• CampainReader is a new engine to read .aca files created by hpc-campaign 0.6.
  https://adios2.readthedocs.io/en/latest/advanced/campaign_management.html, https://hpc-campaign.readthedocs.io
• TimeSeries is an engine to handle a series of individual BP files as one. Enumerate entries in .ats YAML file.
• BP5 has a new aggregator, Data size based aggregation, that performs best for highly imbalanced IO situations, i.e., when different processes produce wildly different amount of data.
• mgard_complex is a new operator to compress complex and complex arrays

New experimental features

What's Changed

• python,cmake: update version script to support rc by @vicentebolea in #3979
• BP5internal - Switch to name based record lookup by @eisenhauer in #3981
• Fix compilation errors on strict compiler by @eisenhauer in #3983
• Update the tutorial with links to external tutorials by @anagainaru in #3978
• ASAN cleanup by @eisenhauer in #3982
• Fix read request length on BP5 Local arrays with operators by @eisenhauer in #3989
• Backport master 3980 3994 by @pnorbert in #3996
• Backport 3984 to master: Python API: Adios() new argument 'config_file', Stream() new argument… by @pnorbert in #3999
• update CODEOWNERS by @vicentebolea in #4003
• update readthedocs deps by @vicentebolea in #4001
• Merge upstream dill changes with master by @eisenhauer in #4006
• Implementing sqlite database instead of json file. Writing part. by @dmitry-ganyushin in #4000
• A simpler way to deal with the mismatching layouts between the user code and the memory space by @anagainaru in #3976
• fix BP5 parameter parsing for size values with units by @pnorbert in #4012
• Merge 3769 to master by @pnorbert in #4011
• Replace the only PERFSTUBS_SCOPED_TIMER_FUNC() call in BP5 to get rid… by @pnorbert in #4013
• Python updates by @pnorbert in #4014
• FFS upstream by @eisenhauer in #4016
• Posix transport should fail if you try to read past EOF by @eisenhauer in #3997
• Make POSIX transport work on Windows, default to it by @eisenhauer in #4023
• Campaign fix by @dmitry-ganyushin in #4024
• readthedocs: enable query docs by @vicentebolea in #4030
• Don't force initialization for buffers whose contents we will immediately destroy by @eisenhauer in #4031
• fix IO.available_attributes(), remove a debug print from Stream, fix … by @pnorbert in #4021
• campaign management: update sqlite database after another run by @dmitry-ganyushin in #4038
• Fixed campaign python script. by @dmitry-ganyushin in #4040
• Fix Windows Testing build by @eisenhauer in #4027
• Tweak POSIX fail on EOF to kill heisenbug by @eisenhauer in #4035
• ci: add macos static build by @vicentebolea in #4041
• Python write scalars by @pnorbert in #4036
• Add a static windows build to CI by @eisenhauer in #4037
• Add arguments to IO.AvailableAttributes() and io.available_attributes… by @pnorbert in #4043
• Return a 0-dim numpy array of size 1 when reading a scalar variable … by @pnorbert in #4044
• Add timeout parameter to stream.steps(). Necessary for controlling ti… by @pnorbert in #4045
• blosc2: do not export blosc2 lib by @vicentebolea in #4055
• Fix segfault when asking for BlocksInfo for a variable name that does… by @pnorbert in #4047
• Reading of campaign configuration parameters from ~/.config/adios2 by @dmitry-ganyushin in #4048
• do not flush io and adios in read mode. BP5 reader does not like it. by @pnorbert in #4065
• Allow plugin operators to take advantage of the estimated size API by @rupertnash in #4046
• Add minmax and shape functions to CampaignReader, so that per-block i… by @pnorbert in #4067
• fixes to still be able to build with gcc 4.8.2. by @pnorbert in #4063
• added support to read back from H5T_STRING VARIABLES by @guj in #4060
• Using the correct flag to detect the CUDA backend in ZFP by @anagainaru in #4066
• Fix Reord to use MinBlocksInfo where appropriate by @eisenhauer in #4071
• • Restructure python API doc in separate main topic, add working exam… by @pnorbert in #4075
• C bindings for the memory space related functions by @anagainaru in #4074
• Actually look at return value for read in remote_server by @eisenhauer in #4078
• Fortran bindings for memory space related functions by @anagainaru in #4077
• Raise an exception if remote open fails by @eisenhauer in #4069
• Python: add the same treatment to attributes as to variables before: … by @pnorbert in #4076
• Update documentation for 2.10 changes to the GPU-backend by @anagainaru in #4083
• BlockIndex.Evaluate() made branch based on BP5 vs BP4. To support Cam… by @pnorbert in #4084
• Fix typo in the links to external tutorials by @anagainaru in #4086
• adios2_define_derived_variable C/Fortran API. C is compiled condition… by @pnorbert in #4088
• Add setup for Aurora (load adios2 as e4s package) by @pnorbert in #4089
• Do not create adios-campaign/ unless there is something to record by @pnorbert in #4096
• Bison 3.8 Parser by @lizdulac in #4062
• WIP: Make Fortran tests fail with a non-zero exit code by @eisenhauer in #4097
• Fix typo in Fortran API docs by @JamieJQuinn in #4102
• ci: add ccache job summary by @vicentebolea in #4101
• Fix static blosc2 build by @vicentebolea in #4093
• Don't run derived test in MPI mode, it's not written for that by @eisenhauer in #4104
• Pull in DILL Upstream that avoids libFFI except on demand. by @eisenhauer in #4091
• Update campaign manager script to handle config file, ... by @pnorbert in #4111
• Use EXPECT_EQ instead of assert() in tests by @eisenhauer in #4108
• Fix warnings that happen in Release build by @eisenhauer in #4113
• EVpath upstream to make NO_RDMA more robust by @eisenhauer in #4116
• Different names for MPI and Serial tests by @eisenhauer in #4118
• cmake: add sqlite3 and zlib dep in adios2 cmake pkg by @vicentebolea in #4120
• Add support for user options in ~/.config/adios2/adios2.yaml by @pnorbert in #4119
• Updates to campaign management by @pnorbert in #4122
• Add attribute support to campaign by @pnorbert in #4124
• DataPlane Configuration changes by @eisenhauer in #4121
• Add -f file1 [file2...] option to process adios files from a list ins… by @pnorbert in #4126
• Add the CURL function to derived variables by @lizdulac in #4114
• Setting the derived variable support OFF by default by @anagainaru in #4127
• Fixes for FreeBSD, including upstream by @eisenhauer in #4138
• Compile FileHTTP on Windows by @eisenhauer in #4132
• Option in bpls to show the expression string for derived variables by @anagainaru in #4145
• pugixml,cmake: fix linkage when pugixml is external by @...

v2.10.2

What's Changed

• Bump version to v2.10.2 by @vicentebolea in #4383
• Backport #4379 by @vicentebolea in #4381
• fix(cmake): increase min required cmake when using bloscs2 by @vicentebolea in #4379
• Fix missing int64_t definition by @eisenhauer in #4365
• Update the memory selection layout when there is a layout mismatch by @anagainaru in #4346
• Setting the memory space for Spans to Host by @anagainaru in #4343
• Compress the lossless part of MGARD data with Zstd. by @pnorbert in #4339
• Fix an issue when reading blocks in streaming mode by @pnorbert in #4332
• Fix in python API: by @pnorbert in #4331
• Update the script to build adios with Kokkos for Frontier by @anagainaru in #4303
• Fixing parser/lexer related errors by @anagainaru in #4293
• [Bug fix] Read side computation of derived variables should not be triggered when data is stored by @anagainaru in #4247
• Update ReadMe.md by @vicentebolea in #4243
• [Bug fix] Switch from ROCM_VERSION_MAJOR to HIP_VERSION_MAJOR by @anagainaru in #4222
• Limit the exposed symbols related to derived variables by @anagainaru in #4207
• Patch for xrootd with derived variables by @anagainaru in #4200
• Add macro defines for BP3, BP4 and BP5 by @franzpoeschel in #4191
• Overload NdCopy instead of adding a new argument for the 2.10 release by @anagainaru in #4377
• Prepare release branch by @vicentebolea in #4378
• bp5dbg parse records and check sizes during it for mmd.0 and md.0. No… by @pnorbert in #3759
• Backport #3759 to Release 2.10 branch. by @pnorbert in #4341
• ci: backport ci changes from master by @vicentebolea in #4356
• Backport 4305 by @vicentebolea in #4349
• bp5dbg parse records and check sizes during it for mmd.0 and md.0. No… by @pnorbert in #3759
• Backport #3759 to Release 2.10 branch. by @pnorbert in #4341
• kokkos: support ROCM >=6 by @vicentebolea in #4214
• Dill Upstream in release_210 by @eisenhauer in #4198

Full Changelog: v2.10.1...v2.10.2

v2.10.1

What's Changed

• pugixml: correct target name by @vicentebolea in #4135

• Dill Upstream by @eisenhauer in #4136

• Fix python char by @pnorbert in #4144

• Test hpdf5 fix on 2.10 by @eisenhauer in #4177

• Deactivate campaign recording by default. Use .adios-campaign/ as dir… by @pnorbert in #4174

• EVPath Upstream #4181

• Add API call to reset memory selection #4169

• Typo in documentation #4166

• Adding hooks to gather performance #4161

• Minor docs tweaks #4153

• ReadModeFlatten #4147

• pugixml,cmake: fix linkage when pugixml #4142

• Fixes for FreeBSD, including upstream #4138

• Kill outdated reference to bindings.Mode.ReadFlattenSteps #4185

• Bump version to v2.10.1 by @vicentebolea in #4187

Full Changelog: v2.10.0...v2.10.1

v2.10.0

What's new in 2.10?

This is a major release with new features and lots of bug fixes. The main new feature is the new Python API.

Python

Before, ADIOS had two separate APIs for Python. The low-level ("Full") API was written with Pybind11 and directly mimicked the C++ API. The high-level API was another, smaller, and more pythonesque API that allowed for easier scripting with Python. The main problems with these two were that they were independent, and that the high-level API was not complete. Once a developer needed a feature only available in the full API, they had to start from scratch writing a script with the full API.

In 2.10, there is officially one Python API, written in Python, which in turn uses the old Pybind11 classes. The new API combines the high-level features of the old high-level API and the full feature set of the low-level bindings.

Note:

Old scripts that used the full API can still run without almost any modification, just change the import line from import adios2 to import adios2.bindings as adios2

Old scripts that used the high-level API must be modified to make them work with the new API, see Transition from old API to new API: Transition from old API to new API

See Python API

New/updated notable features

• BP5 is supported on Windows now
• SST and DataMan staging engines are GPU-Aware now
• SYCL support added for Intel GPUs (besides CUDA and HIP for NVidia and AMD GPUs)
• the SST/libfabric data transport now works on Frontier (besides the MPI data transport)

New experimental features

• Campaign management: https://adios2.readthedocs.io/en/latest/advanced/campaign_management.html

What's Changed

• ci: migrate to almalinux, cuda base image by @vicentebolea in #3484
• ci: update kokkos to 3.7 by @vicentebolea in #3490
• codeowners: added code owners file by @vicentebolea in #3496
• Hdf5 vol support will require latest hdf5 1.14 by @guj in #3516
• Bump numpy from 1.21.6 to 1.22.0 in /docs by @dependabot in #3535
• Bump cryptography from 38.0.2 to 39.0.1 in /docs by @dependabot in #3534
• Release 29 by @dmitry-ganyushin in #3545
• Enable Linux Emulation build (Power8, IBM XL) by @vicentebolea in #3537
• Setting code owners for the GPU files by @anagainaru in #3561
• Release 29 by @vicentebolea in #3573
• Allowing Put/Get to receive Kokkos Views with HIP backend by @anagainaru in #3571
• Prototype DAOS engine by @eisenhauer in #3576
• CODEOWNERS: update codeowner file by @vicentebolea in #3581
• shellcheck: add shellcheck ci check by @vicentebolea in #3582
• Build scripts for ADIOS2 with CUDA and Kokkos backend by @anagainaru in #3579
• Disable emu tests by @vicentebolea in #3583
• cmake: set correct soname by @vicentebolea in #3585
• TestBPChangingShapeWithinStep without and with various compressors by @pnorbert in #3577
• add release files by @vicentebolea in #3570
• Heat Transfer Example: MPI Datatype by @ax3l in #3593
• test,install: paramaterize adios2_DIR by @vicentebolea in #3586
• ci,ascent: enable ascent builds by @vicentebolea in #3610
• ci: use nvcc_wrapper in adiosKokkos by @vicentebolea in #3623
• Adding SYCL support to the Kokkos backend by @anagainaru in #3621
• Bump requests from 2.28.1 to 2.31.0 in /docs by @dependabot in #3628
• Fix flaky MSMPI installation in windows builds by @vicentebolea in #3627
• docs: update whatsnew by @vicentebolea in #3634
• ci,olcf,crusher: enable Crusher CI by @vicentebolea in #3620
• ci,crusher: minor tweaks by @vicentebolea in #3635
• Ascent,CI: do not report sync script build by @vicentebolea in #3637
• replaced uint with size_t by @guj in #3640
• crusher,ci: set unique env per pipeline by @vicentebolea in #3644
• Update documentation for the GPU-aware functionality by @anagainaru in #3639
• work around with include <unistd.h> on windows by @guj in #3652
• Bug fix for Kokkos applications without GPU backend by @anagainaru in #3654
• An effort to make profiler more user friendly by @guj in #3660
• Bump cryptography from 39.0.1 to 41.0.0 in /docs by @dependabot in #3650
• ci: remove power8 emulated builds by @vicentebolea in #3667
• ci,ascent: disable install tests by @vicentebolea in #3673
• Tweaks for BP5 on windows. by @eisenhauer in #3682
• Fix memory leak when there are Joined Arrays in streaming mode by @eisenhauer in #3684
• Fix for the issue #3646. by @dmitry-ganyushin in #3687
• cmake,evpath: namespace modules install dir by @vicentebolea in #3689
• Ci rework gcc builds by @scottwittenburg in #3631
• added gitattributes file by @vicentebolea in #3698
• Update files with new EOL by @vicentebolea in #3699
• build(deps): bump cryptography from 41.0.0 to 41.0.2 in /docs by @dependabot in #3697
• ci,mpich: disable Sz|Zfp flaky tests by @vicentebolea in #3702
• build(deps): bump pygments from 2.14.0 to 2.15.0 in /docs by @dependabot in #3704
• ci,cmake: cleanup scripts subdirectory by @vicentebolea in #3625
• cmake: moved logic to enable h5vol to detectoptions.cmake by @vicentebolea in #3701
• build(deps): bump certifi from 2022.12.7 to 2023.7.22 in /docs by @dependabot in #3712
• ci: increase windows timeout defsync and DataWrite by @vicentebolea in #3726
• format,ci: update clang-format to clang16 by @vicentebolea in #3722
• Enable Flush Test for BP5 by @eisenhauer in #3683
• Use ccache in github actions builds by @scottwittenburg in #3705
• ci: Fix conditional expression syntax to fix cache save by @scottwittenburg in #3730
• ci: work around bug in actions/cache by @scottwittenburg in #3731
• build(deps): bump cryptography from 41.0.2 to 41.0.3 in /docs by @dependabot in #3733
• Version constants plus functions to retrieve list of ... by @pnorbert in #3676
• ci: add libcurl4 to all ubuntu images by @scottwittenburg in #3734
• ci: update release instructions by @vicentebolea in #3738
• ci: add downloads readthedocs by @vicentebolea in #3739
• cmake: fix evpath plugins install path by @vicentebolea in #3737
• Remote Data access via Get() or Transport by @eisenhauer in #3740
• readthedocs: remove sys pkgs; use current ADIOS2 by @vicentebolea in #3743
• Fix: Static Builds Against C-Blosc2 by @ax3l in #3715
• CI: add two static builds by @vicentebolea in #3749
• Upstream ffs by @eisenhauer in #3755
• Update Readme.md with community section by @vicentebolea in #3700
• cmake,readthedocs: update python enable instructions by @vicentebolea in #3713
• cmake: correct info.h installation path by @vicentebolea in #3758
• Korvo upstream by @eisenhauer in #3761
• cmake: resolve cmake python deprecation warning by @vicentebolea in #3764
• Update kwsys by @vicentebolea in #3765
• ci: add mgard dependency to spack builds by @vicentebolea in #375...

v2.10.0-rc1

What's Changed

• ci: migrate to almalinux, cuda base image by @vicentebolea in #3484
• ci: update kokkos to 3.7 by @vicentebolea in #3490
• codeowners: added code owners file by @vicentebolea in #3496
• Hdf5 vol support will require latest hdf5 1.14 by @guj in #3516
• Bump numpy from 1.21.6 to 1.22.0 in /docs by @dependabot in #3535
• Bump cryptography from 38.0.2 to 39.0.1 in /docs by @dependabot in #3534
• Release 29 by @dmitry-ganyushin in #3545
• Enable Linux Emulation build (Power8, IBM XL) by @vicentebolea in #3537
• Setting code owners for the GPU files by @anagainaru in #3561
• Release 29 by @vicentebolea in #3573
• Allowing Put/Get to receive Kokkos Views with HIP backend by @anagainaru in #3571
• Prototype DAOS engine by @eisenhauer in #3576
• CODEOWNERS: update codeowner file by @vicentebolea in #3581
• shellcheck: add shellcheck ci check by @vicentebolea in #3582
• Build scripts for ADIOS2 with CUDA and Kokkos backend by @anagainaru in #3579
• Disable emu tests by @vicentebolea in #3583
• cmake: set correct soname by @vicentebolea in #3585
• TestBPChangingShapeWithinStep without and with various compressors by @pnorbert in #3577
• add release files by @vicentebolea in #3570
• Heat Transfer Example: MPI Datatype by @ax3l in #3593
• test,install: paramaterize adios2_DIR by @vicentebolea in #3586
• ci,ascent: enable ascent builds by @vicentebolea in #3610
• ci: use nvcc_wrapper in adiosKokkos by @vicentebolea in #3623
• Adding SYCL support to the Kokkos backend by @anagainaru in #3621
• Bump requests from 2.28.1 to 2.31.0 in /docs by @dependabot in #3628
• Fix flaky MSMPI installation in windows builds by @vicentebolea in #3627
• docs: update whatsnew by @vicentebolea in #3634
• ci,olcf,crusher: enable Crusher CI by @vicentebolea in #3620
• ci,crusher: minor tweaks by @vicentebolea in #3635
• Ascent,CI: do not report sync script build by @vicentebolea in #3637
• replaced uint with size_t by @guj in #3640
• crusher,ci: set unique env per pipeline by @vicentebolea in #3644
• Update documentation for the GPU-aware functionality by @anagainaru in #3639
• work around with include <unistd.h> on windows by @guj in #3652
• Bug fix for Kokkos applications without GPU backend by @anagainaru in #3654
• An effort to make profiler more user friendly by @guj in #3660
• Bump cryptography from 39.0.1 to 41.0.0 in /docs by @dependabot in #3650
• ci: remove power8 emulated builds by @vicentebolea in #3667
• ci,ascent: disable install tests by @vicentebolea in #3673
• Tweaks for BP5 on windows. by @eisenhauer in #3682
• Fix memory leak when there are Joined Arrays in streaming mode by @eisenhauer in #3684
• Fix for the issue #3646. by @dmitry-ganyushin in #3687
• cmake,evpath: namespace modules install dir by @vicentebolea in #3689
• Ci rework gcc builds by @scottwittenburg in #3631
• added gitattributes file by @vicentebolea in #3698
• Update files with new EOL by @vicentebolea in #3699
• build(deps): bump cryptography from 41.0.0 to 41.0.2 in /docs by @dependabot in #3697
• ci,mpich: disable Sz|Zfp flaky tests by @vicentebolea in #3702
• build(deps): bump pygments from 2.14.0 to 2.15.0 in /docs by @dependabot in #3704
• ci,cmake: cleanup scripts subdirectory by @vicentebolea in #3625
• cmake: moved logic to enable h5vol to detectoptions.cmake by @vicentebolea in #3701
• build(deps): bump certifi from 2022.12.7 to 2023.7.22 in /docs by @dependabot in #3712
• ci: increase windows timeout defsync and DataWrite by @vicentebolea in #3726
• format,ci: update clang-format to clang16 by @vicentebolea in #3722
• Enable Flush Test for BP5 by @eisenhauer in #3683
• Use ccache in github actions builds by @scottwittenburg in #3705
• ci: Fix conditional expression syntax to fix cache save by @scottwittenburg in #3730
• ci: work around bug in actions/cache by @scottwittenburg in #3731
• build(deps): bump cryptography from 41.0.2 to 41.0.3 in /docs by @dependabot in #3733
• Version constants plus functions to retrieve list of ... by @pnorbert in #3676
• ci: add libcurl4 to all ubuntu images by @scottwittenburg in #3734
• ci: update release instructions by @vicentebolea in #3738
• ci: add downloads readthedocs by @vicentebolea in #3739
• cmake: fix evpath plugins install path by @vicentebolea in #3737
• Remote Data access via Get() or Transport by @eisenhauer in #3740
• readthedocs: remove sys pkgs; use current ADIOS2 by @vicentebolea in #3743
• Fix: Static Builds Against C-Blosc2 by @ax3l in #3715
• CI: add two static builds by @vicentebolea in #3749
• Upstream ffs by @eisenhauer in #3755
• Update Readme.md with community section by @vicentebolea in #3700
• cmake,readthedocs: update python enable instructions by @vicentebolea in #3713
• cmake: correct info.h installation path by @vicentebolea in #3758
• Korvo upstream by @eisenhauer in #3761
• cmake: resolve cmake python deprecation warning by @vicentebolea in #3764
• Update kwsys by @vicentebolea in #3765
• ci: add mgard dependency to spack builds by @vicentebolea in #3753
• Misc fixes by @vicentebolea in #3768
• Add automated vulnerability code analysis using CodeQL. by @vicentebolea in #3606
• cmake: correct ADIOS2_Blosc2_PREFER_SHARED behavior by @vicentebolea in #3767
• match type of timestep for h5 engine to size_t (same as adios Variabl… by @guj in #3776
• ci: Create static minimal build by @vicentebolea in #3760
• Remove unused SelectionType values by @eisenhauer in #3784
• Update ADIOS2 HDF5 VOL with basic set of capability flags by @jhendersonHDF in #3632
• gha,ci: update checkout to v4 by @vicentebolea in #3789
• ci: Add HDF5 to a windows build by @scottwittenburg in #3732
• Fixup local var reading by block with test, master branch by @eisenhauer in #3787
• Add MinBlocksInfo to BP5 writer engine by @eisenhauer in #3794
• yaml-cpp: support 0.8.0 version by @vicentebolea in #3799
• Delete experimental examples by @spyridon97 in #3800
• Remove BP5 BetweenStepPairs variable that hides Engine.h counterpart by @eisenhauer in #3803
• WIP: Reader-side Profiling by @eisenhauer in #3778
• Fix the ADIOS_USE_{} variable names to use ADIOS2 by @anagainaru in #3807
• Pass Array Order of reader to remote server for proper Get() operation by @eisenhauer in https://github.com...

ADIOS v2.9.2 Production Release

Breaking changes

BP5 is the default engine wince 2.9.0 that behaves differently from BP3/BP4. Read the notes in the 2.9.0 release.

What's Changed

• Organize the processes into groups so that the two steps of metadata … by @pnorbert in #3750
• bp5dbg parse records and check sizes during it for mmd.0 and md.0. No… by @pnorbert in #3759
• Fix ChunkV maintaining CurOffset when downsizing current chunk in All… by @pnorbert in #3769
• BP5Deserialize: modify changes to keep abi compt by @vicentebolea in #3773
• Backport readthedocs fixes by @vicentebolea in #3783
• Fix local values block reading by @eisenhauer in #3782
• Set AWS version requirement to 1.10.15 and also turn it OFF by defaul… by @pnorbert in #3804
• bpls -l with scalar string variable: print the value (since min/max … by @pnorbert in #3805
• FFS upstream by @eisenhauer in #3814
• Backport from master by @vicentebolea in #3861
• Fix data length calculation for hash by @eisenhauer in #3875
• Replace LookupWriterRec's linear search on RecList with an unordered_… by @pnorbert in #3877
• Merge pull request #3823 from eisenhauer/SstMemSel by @vicentebolea in #3871
• bp5: make RecMap an static anon namespaced var by @vicentebolea in #3881
• Merge pull request #3588 from vicentebolea/fix-mpi-dp by @vicentebolea in #3885
• Merge pull request #3878 from anagainaru/test-null-blocks by @vicentebolea in #3886
• Bump version to v2.9.2 by @vicentebolea in #3888
• Fix destdir install test by @vicentebolea in #3850
• cmake: update minimum cmake to 3.12 by @vicentebolea in #3849
• MPI: add timeout for conf test for MPI DataPlane by @vicentebolea in #3848
• MPI DataPlane: do not call MPI_Init internally by @vicentebolea in #3847
• install: export adios2 device variables by @vicentebolea in #3819
• Fix abs/rel step in BP5 DoCount by @eisenhauer in #3812
• yaml-cpp: support 0.8.0 version by @vicentebolea in #3799
• gha,ci: update checkout to v4 by @vicentebolea in #3789
• Blosc2 USE ON: Fix Module Fallback by @ax3l in #3774
• cmake: correct ADIOS2_Blosc2_PREFER_SHARED behavior by @vicentebolea in #3767
• ci: Create static minimal build by @vicentebolea in #3760
• cmake: correct info.h installation path by @vicentebolea in #3758
• ci: add downloads readthedocs by @vicentebolea in #3739
• cmake: fix evpath plugins install path by @vicentebolea in #3737
• Fix: Static Builds Against C-Blosc2 by @ax3l in #3715
• Fix: Static Builds Against C-Blosc2 by @ax3l in #3715
• ci: add downloads readthedocs by @vicentebolea in #3739
• ci: Create static minimal build by @vicentebolea in #3760
• cmake: correct info.h installation path by @vicentebolea in #3758
• cmake: correct ADIOS2_Blosc2_PREFER_SHARED behavior by @vicentebolea in #3767
• gha,ci: update checkout to v4 by @vicentebolea in #3789
• Fix abs/rel step in BP5 DoCount by @eisenhauer in #3812

Full Changelog: v2.9.1...v2.9.2

ADIOS v2.9.1 Production Release

Breaking changes

BP5 is the default engine wince 2.9.0 that behaves differently from BP3/BP4. Read the notes in the 2.9.0 release.

Pull Requests backported

PR	Title
#3676	Version constants plus functions to retrieve list of ...
#3697	build(deps): bump cryptography from 41.0.0 to 41.0.2 in /docs
#3698	added gitattributes file
#3704	build(deps): bump pygments from 2.14.0 to 2.15.0 in /docs
#3712	build(deps): bump certifi from 2022.12.7 to 2023.7.22 in /docs
#3687	Fix for the issue #3646.
#3699	Update files with new EOL
#3690	Merge pull request #3689 from vicentebolea/fix-evpath-modules
#3689	cmake,evpath: namespace modules install dir
#3684	Fix memory leak when there are Joined Arrays in streaming mode
#3652	work around with include <unistd.h> on windows
#3650	Bump cryptography from 39.0.1 to 41.0.0 in /docs
#3673	ci,ascent: disable install tests
#3681	Fix: std::min w/ windows.h in C-Blosc2
#3657	Allow Span in files opened for Append
#3623	ci: use nvcc_wrapper in adiosKokkos
#3619	Backports bp5 bugfix
#3605	Adding missing HIP specific functionality
#3607	Backport: Heat Transfer Example: MPI Datatype #3593
#3593	Heat Transfer Example: MPI Datatype
#3591	Implement BP5 (and BP4) reader-side memory selection, do testing

Full Changelog: v2.9.0...v2.9.1

ADIOS v2.9.0 Production Release

Summary

This is a major release with new features and lots of bug fixes.

Breaking changes

BP5 is the default engine that behaves differently from BP3/BP4. Read mode is proper streaming mode, i.e., Open does not process the metadata and hence io.AvailableVariables() will return empty. Variables and attributes are only available after engine.BeginStep(), and only those that exist in the given step. Also, because of this, the pointer returned by io.InquireVariable() becomes invalid after EndStep. Note, that there is now a new ReadRandomAccess mode to process all metadata in Open and to see all the variables with all their steps at once.

General

• GPU-Aware I/O enabled by using Kokkos. Device pointers can be passed to Put()/Get() calls directly. Kokkos 3.7.x required for this release. Works with CUDA, HIP and Kokkos applications. https://adios2.readthedocs.io/en/latest/advanced/gpu_aware.html#gpu-aware-i-o
• GPU-compression. MGARD and ZFP operators can compress data on GPU if they are built for GPU. MGARD operator can be fed with host/device pointers and will move data automaticaly. ZFP operator requires matching data and compressor location.
• Joined Array concept (besides Global Array and Local Array), which lets writers dump Local Arrays (no offsets no global shape) that are put together into a Global Array by the reader. One dimension of the arrays is selected for this join operation, while other dimensions must be the same for all writers. https://adios2.readthedocs.io/en/latest/components/components.html?highlight=Joined#shapes

File I/O

• Default File engine is now BP5. If for some reason this causes problems, manually specify using "BP4" for your application.
• BP5 is semantically identical to the staging engines, and any code that works with BP5 will work in situ without changes.
• BP5 engine supports multithreaded reading to accelerate read performance for low-core counts.
• BP5 Two level metadata aggregation and reduction reduced memory impact of collecting metadata and therefore is more scalable in terms of numbers of variables and writers than BP4.
• Uses Blosc-2 instead of Blosc for lossless compression. The new compression operator is backward compatible with old files compressed with blosc. The name of the operator remains "blosc".

Staging

• UCX dataplane added for SST staging engine to support networks under the UCX consortium
• MPI dataplane added for SST staging engine. It relies on MPI intercommunicators to connect multiple independent MPI applications for staging purposes. Applications must enable multithreaded MPI for this dataplane.

Experimental features

• Preliminary support for data structs. A struct can have single variables of basic types, and 1D fixed size arrays of basic types. Supported by BP5, SST and SSC engines.

What's Changed

• Refactor SSC to allow easy addition of new sub-routines by @JasonRuonanWang in #3092
• BP5Deserializer GenerateReadRequests fix rank fix by @eisenhauer in #3096
• Added naive engine mode for SSC by @JasonRuonanWang in #3109
• move sodium option to its right place by @JasonRuonanWang in #3112
• Bump pillow from 9.0.0 to 9.0.1 in /docs by @dependabot in #3103
• added bp file reading in MgardPlus by @JasonRuonanWang in #3113
• added zero block test for ssc by @JasonRuonanWang in #3111
• fixed a bug in ssc which blocks MPI communications by @JasonRuonanWang in #3114
• clean up SSC logging information for better debugging experience by @JasonRuonanWang in #3115
• SSC: temporary workaround for crusher MPI bug by @JasonRuonanWang in #3117
• DILL upstream by @eisenhauer in #3118
• Fix windows mpi builds by @chuckatkins in #3142
• remove template instantiation in header files by @JasonRuonanWang in #3144
• MgardPlus: check for empty mesh file by @JasonRuonanWang in #3157
• pr3153 into master by @chuckatkins in #3158
• blosc unknown parameter should be warning by @JasonRuonanWang in #3163
• Fix a template instantiation issue with GCC 10 in dataman serializer by @JasonRuonanWang in #3165
• ENET upstream by @eisenhauer in #3166
• Fix helloBPTimeWriter Python example. by @manauref in #3167
• removed VariableCompound class as it has never worked since added by @JasonRuonanWang in #3168
• move Span class to dedicated files by @JasonRuonanWang in #3169
• Add template-free C++ API by @JasonRuonanWang in #3171
• on-demand step delivery by @eisenhauer in #3170
• added missing open modes in C API by @JasonRuonanWang in #3174
• Add struct variable support by @JasonRuonanWang in #3179
• added DefineStructVariable test by @JasonRuonanWang in #3183
• moved Variable::Shape() to VariableBase by @JasonRuonanWang in #3184
• removed VariableBase::GetShape by @JasonRuonanWang in #3185
• Remove templates in SSC writer by @JasonRuonanWang in #3187
• removed most templated functions in ssc reader by @JasonRuonanWang in #3188
• added ssc test for struct variables by @JasonRuonanWang in #3189
• added IO::m_StructDefinitions to hold temporary struct definitions from parsing step metadata by @JasonRuonanWang in #3191
• fixed a bug in ssc test that fails when mpi size is large by @JasonRuonanWang in #3192
• Add MinMax in non-template API by @JasonRuonanWang in #3193
• Tweak OnDemand test by @eisenhauer in #3190
• Add BlocksInfo in SSC naive by @JasonRuonanWang in #3194
• finish up polishing tests and examples by @JasonRuonanWang in #3196
• moved ToBlocksInfoMin to private by @JasonRuonanWang in #3197
• added BlocksInfo API for VariableNT, and implementation for VariableStruct in SSC by @JasonRuonanWang in #3198
• make struct size parameter compulsory to ensure padding correctness by @JasonRuonanWang in #3199
• added freeze function to StructDefinition by @JasonRuonanWang in #3200
• added allowReorganize for InquireStructVariable by @JasonRuonanWang in #3201
• Add SSC struct definition serialization by @JasonRuonanWang in #3202
• Update documentation with correct StepDistributionMode values by @anagainaru in #3212
• Add access to internal m_BetweenStepPairs member by @eisenhauer in #3216
• Bp5 read multithreaded by @pnorbert in #3220
• Bp5 multithreaded read, dynamic version by @pnorbert in #3233
• Throw an error when GathervArrays() is asked to gather more than 2^31… by @pnorbert in #3239
• Stop using Dims in some performance critical areas by @eisenhauer in #3240
• Merge bp5 flush read from 281 by @pnorbert in #3243
• Fix cuda build by @eisenhauer in #3246
• If BP5 parameter Threads is 0 (default), calculate the number of thre… by @pnorbert in #3247
• fix api for gcc11 by @robertu94 in #3250
• BP5 Two level metadata aggregation and reduction: work in progress saved by @pnorbert in #3251
• Remove DebugMode from all external APIs by @eisenhauer in #3204
• FFS upstream by @eisenhauer in #3259
• Fix C binding problem by @eisenhauer in #3264
• BP5 subformats by @eisenhauer in #3260
• Ensure Init of Blocks Info by @eisenhauer in #3268
• Kill reader-created variables in parent IO upon engine close (for BP5) by @eisenhauer in #3270
• Changed parameters for DAOS API functions. by @dmitry-ganyushin in #3266
• Add backward compatible read with BLOSC compression to BP4 files crea… by @pnorbert in #3277
• Necessary revision to get VOL work HDF5 1.13 & BP4 by @guj in #3279
• Destructor close by @eisenhauer in #3278
• Destroy only created vars on Beg...

ADIOS v2.9.0-rc1 Production Release

Merge pull request #3476 from vicentebolea/bump-release-version

release: Bump version to v2.9.0-rc1