v2.12.0

[vicentebolea]

Release notes

C API Improvements

New two-call pattern functions for safely retrieving string data of unknown
length: adios2_get_string(), adios2_attribute_string_data(), and
adios2_attribute_string_data_array().

Selection API for Get() Operations

A new Selection class bundles spatial, block, and step parameters into a
single object passed to Get(), replacing scattered SetSelection() /
SetBlockSelection() / SetStepSelection() calls on Variable.

engine.Get(var, data, adios2::Selection::BoundingBox({0,0},{10,20}).WithSteps(0,5));

Plugin Interface v2 (Breaking Change)

Engine and operator plugins no longer inherit from internal core types.
Engine plugins inherit from PluginEngineInterface, operator plugins from
PluginOperatorInterface, and shared libraries only need to link adios2::cxx.
Update EngineCreate() / EngineDestroy() signatures; see examples/plugins/.

C++17 Minimum Requirement

The minimum C++ standard to build ADIOS2 has been raised from C++14 to C++17.

S3 Object Storage for BP5 Data

BP5 can now write data files to S3-compatible object storage (Amazon S3, MinIO,
Ceph, etc.) with metadata kept on the local filesystem. Set
DataFileTransport=awssdk, S3Endpoint, and S3Bucket as engine parameters.

Python Bindings: Nanobind Migration

The Python bindings have been migrated from pybind11 to nanobind, producing
smaller, faster extension modules and enabling stable-ABI (abi3) wheels for
Python 3.12 and later.

Asymmetric Encryption Operator

A new Encryption operator encrypts variable data using libsodium's sealed-box
construction (X25519 + XSalsa20-Poly1305). Writers only need the recipient's
public key; only the private key holder can decrypt.

SZ3 Compression Operator

A new SZ3 lossy compression operator is available, extending the existing
options alongside SZ, ZFP, MGARD, and BigWhoop.

RefactorProDM Operator

A new RefactorProDM operator integrates ADIOS2 with ProDM (Dr. Xin Liang's
decomposition library), providing decomposition-based data reduction when ProDM
is available at build time.

Python Free-Threading Support

The Python bindings are now compatible with free-threaded Python (PEP 703,
python3.13t), allowing ADIOS2 to run without the GIL.

HTTPS Transport for Remote Access

An HTTPS-based remote transport (XrootdHttpsRemote) using libcurl has been
added as an alternative to the native XRootD protocol for environments where
native XRootD connectivity is unavailable.

XRootD Server Resource Management

The XRootD server plugin now evicts idle file descriptors and metadata when
configurable limits are approached, shares a single FD across engines reading
the same tar file, and batches variable reads into a single network round-trip.

Cross-Endian Interoperability

The ADIOS2_USE_Endian_Reverse CMake option has been removed; cross-endian
file interoperability is now always enabled.

Campaign Reader Improvements

Host aliases, XRootD host entries, and in-memory metadata caching were added.
The reader now requires ACA 0.7 strictly. A rewritten SQLite join query reduces
open time from ~27 seconds to under one second for large archives.

TAR File Reading

BP5 datasets, text files, and images can now be read from TAR archives (local,
HTTPS-hosted, or via the remote server) when a TAR index is provided.

BP5 DataFileTransport Parameter Rename

The BP5 engine parameter DataTransport has been renamed to DataFileTransport.
Existing configurations must be updated.

BP5 Concurrent File Access

BP5 data reading now uses per-subpool locking over a shared file-descriptor
pool, enabling safe concurrent reads from multiple threads.

Python Stream: minmax()

adios2.Stream.minmax(name, [step], [block_info_list]) returns per-step or
global min/max values of a variable.

Inline Engine Buffering

The inline engine now supports basic output buffering, moving toward feature
parity with file-based engines for in-situ workflows.

Windows and macOS pip Wheel Support

CI now runs pip sdist and wheel integration tests on Windows and macOS,
verifying correct library loading under pip install workflows.

REUSE License Compliance

ADIOS2 now complies with the REUSE specification; every source file carries a
machine-readable SPDX license and copyright notice.

What's Changed

• Fix logic error in BP5 selective metadata aggregation by @eisenhauer in Fix logic error in BP5 selective metadata aggregation #4717
• RDMA dataplane: Share fi_fabric() and don't close until all are done by @eisenhauer in RDMA dataplane: Share fi_fabric() and don't close until all are done #4646
• Potential fix for CMake logic in using an external EVPath build by @Crivella in Potential fix for CMake logic in using an external EVPath build #4697
• backport 4722 by @vicentebolea in backport 4722 #4724
• Bump the all-actions group with 3 updates by @dependabot[bot] in Bump the all-actions group with 3 updates #4727
• cmake: define cxx11 target once by @vicentebolea in cmake: define cxx11 target once #4730
• Korvo Upstream by @eisenhauer in Korvo Upstream #4733
• Fixup XRootD plugin by @eisenhauer in Fixup XRootD plugin #4728
• ci: fix windows ci python err by @vicentebolea in ci: fix windows ci python err #4739
• ci: upload conda env description to cdash by @vicentebolea in ci: upload conda env description to cdash #4744
• use the same base handle as in extract.sh by @guj in use the same base handle as in extract.sh #4714
• Use Transports directly by @eisenhauer in Use Transports directly #4740
• ci: add backport action by @vicentebolea in ci: add backport action #4748
• ci: setup git committer in backport action by @vicentebolea in ci: setup git committer in backport action #4749
• ci: setup git committer in backport action by @vicentebolea in ci: setup git committer in backport action #4750
• ci: setup git committer in backport action by @vicentebolea in ci: setup git committer in backport action #4751
• Note SZ version operability by @eisenhauer in Note SZ version operability #4752
• BP5Writer: Subfile config for DSB works like other aggregators by @scottwittenburg in BP5Writer: Subfile config for DSB works like other aggregators #4742
• Documentation fixes for beginner users by @hahnjo in Documentation fixes for beginner users #4746
• Better documentation of collective functions by @hahnjo in Better documentation of collective functions #4755
• ci: remove temp install after test install by @vicentebolea in ci: remove temp install after test install #4754
• ci: download latest cmake for el8 imgs by @vicentebolea in ci: download latest cmake for el8 imgs #4758
• Updated profiler tags used in json by @guj in Updated profiler tags used in json #4759
• Fix Conda badge link in ReadMe.md by @vicentebolea in Fix Conda badge link in ReadMe.md #4762
• Add support to read a BP5 dataset / TEXT / IMAGE from TAR files... by @pnorbert in Add support to read a BP5 dataset / TEXT / IMAGE from TAR files... #4764
• Use specified engine rather than BPFile in tests by @eisenhauer in Use specified engine rather than BPFile in tests #4757
• Integrate ProDM changes, added new operator RefactorProDM by @AkiraFukuda in Integrate ProDM changes, added new operator RefactorProDM #4771
• Add SZ3 compression operator support by @eisenhauer in Add SZ3 compression operator support #4772
• Use pool of files for data reading in BP5 by @eisenhauer in Use pool of files for data reading in BP5 #4766
• Add CLAUDE.md guidance file for Claude Code development by @eisenhauer in Add CLAUDE.md guidance file for Claude Code development #4779
• Add automatic cleanup of generated test files in BP engine tests by @eisenhauer in Add automatic cleanup of generated test files in BP engine tests #4778
• Consolidate test utilities and improve cleanup handling by @eisenhauer in Consolidate test utilities and improve cleanup handling #4780
• Order remote replicas in order of most likely accessible: … by @pnorbert in Order remote replicas in order of most likely accessible: … #4768
• Fix SZ3 3.1+ API compatibility in decompression by @eisenhauer in Fix SZ3 3.1+ API compatibility in decompression #4785
• ci: add EXTRA_SPECS build arg to spack imgs by @vicentebolea in ci: add EXTRA_SPECS build arg to spack imgs #4786
• ci: use 0s in static jobs by @vicentebolea in ci: use 0s in static jobs #4788
• ci: fix clang static job by @vicentebolea in ci: fix clang static job #4792
• tests: sanitize path before removal by @vicentebolea in tests: sanitize path before removal #4789
• Update common tracking by @mathstuf in Update common tracking #4710
• build(deps): bump the all-actions group across 1 directory with 7 updates by @dependabot[bot] in build(deps): bump the all-actions group across 1 directory with 7 updates #4787
• tests: sanitize path before removal by @vicentebolea in tests: sanitize path before removal #4793
• refactor code to utilize c++14 features by @vicentebolea in refactor code to utilize c++14 features #4783
• ci: reduce size of clang11-static-mpich job by @vicentebolea in ci: reduce size of clang11-static-mpich job #4798
• Add adios2_get_string() to C API for safe string variable retrieval by @eisenhauer in Add adios2_get_string() to C API for safe string variable retrieval #4801
• Add C API functions for safe string attribute retrieval by @eisenhauer in Add C API functions for safe string attribute retrieval #4804
• build(deps): bump github/codeql-action from 4.31.9 to 4.31.10 in the all-actions group by @dependabot[bot] in build(deps): bump github/codeql-action from 4.31.9 to 4.31.10 in the all-actions group #4797
• Reorganize reading of ACA files for version 0.7 by @pnorbert in Reorganize reading of ACA files for version 0.7 #4809
• build(deps): bump actions/cache from 5.0.1 to 5.0.2 in the all-actions group by @dependabot[bot] in build(deps): bump actions/cache from 5.0.1 to 5.0.2 in the all-actions group #4810
• Revert "Merge pull request Update common tracking #4710 from mathstuf/update-common-tracking" by @vicentebolea in Revert "Merge pull request #4710 from mathstuf/update-common-tracking" #4811
• GTKorvo Upstream by @eisenhauer in GTKorvo Upstream #4812
• Detect the new flag HDF5_PROVIDES_PARALLEL from 2.0 by @guj in Detect the new flag HDF5_PROVIDES_PARALLEL from 2.0 #4814
• Add HTTPS transport for XRootD remote access by @eisenhauer in Add HTTPS transport for XRootD remote access #4815
• adios2.Stream.minmax() by @pnorbert in adios2.Stream.minmax() #4818
• ci: fix readthedocs job by @vicentebolea in ci: fix readthedocs job #4823
• Fix 3p update script by @mathstuf in Fix 3p update script #4816
• build(deps): bump the all-actions group across 1 directory with 3 updates by @dependabot[bot] in build(deps): bump the all-actions group across 1 directory with 3 updates #4824
• Xrootd https remote by @eisenhauer in Xrootd https remote #4822
• Fix dill config for unsupported architectures by @eisenhauer in Fix dill config for unsupported architectures #4826
• Fix BP5 cross-endian support for big-endian architectures (s390x) by @eisenhauer in Fix BP5 cross-endian support for big-endian architectures (s390x) #4803
• fix thirdparties by @vicentebolea in fix thirdparties #4827
• Stabilize CI tests by @eisenhauer in Stabilize CI tests #4829
• build(deps): bump github/codeql-action from 4.32.0 to 4.32.1 in the all-actions group by @dependabot[bot] in build(deps): bump github/codeql-action from 4.32.0 to 4.32.1 in the all-actions group #4832
• BP5: Fix segfault with DataSizeBased aggregation and async write by @scottwittenburg in BP5: Fix segfault with DataSizeBased aggregation and async write #4834
• Cleanup XRootD Docker build by @eisenhauer in Cleanup XRootD Docker build #4828
• Fix flaky tests: BPFortranToCppRead and TestOperator.py by @eisenhauer in Fix flaky tests: BPFortranToCppRead and TestOperator.py #4835
• Fix DataMan Python tests: add RUN_SERIAL by @eisenhauer in Fix DataMan Python tests: add RUN_SERIAL #4837
• Merge WriteReadADIOS2 test variants by @eisenhauer in Merge WriteReadADIOS2 test variants #4830
• Fix NULL pointer dereference in SST writer close by @eisenhauer in Fix NULL pointer dereference in SST writer close #4839
• Add accuracy parameters to XRootD HTTP protocol by @eisenhauer in Add accuracy parameters to XRootD HTTP protocol #4840
• Support host aliases by @pnorbert in Support host aliases #4841
• BP5Deserializer: capture selection state at queue time for thread safety by @eisenhauer in BP5Deserializer: capture selection state at queue time for thread safety #4842
• BP5Reader: close metadata FDs when writer is inactive by @eisenhauer in BP5Reader: close metadata FDs when writer is inactive #4846
• Cleanup files that might be lingering from earlier output by @eisenhauer in Cleanup files that might be lingering from earlier output #4845
• Campaign reader engine must call PerformGets in DoClose to enable mul… by @pnorbert in Campaign reader engine must call PerformGets in DoClose to enable mul… #4853
• ADIOSFilePool: per-SubPool locking for concurrent file access by @eisenhauer in ADIOSFilePool: per-SubPool locking for concurrent file access #4847
• Eliminate blockdiag in favor of uploaded svg and png images. by @eisenhauer in Eliminate blockdiag in favor of uploaded svg and png images. #4861
• Korvo Upstream by @eisenhauer in Korvo Upstream #4852
• Fix XRootD HTTPS test startup race by @eisenhauer in Fix XRootD HTTPS test startup race #4855
• Fix SST writer close race and KillWriter test timeout by @eisenhauer in Fix SST writer close race and KillWriter test timeout #4858
• Fix TimeSeries MPI/Serial test collision by @eisenhauer in Fix TimeSeries MPI/Serial test collision #4856
• Fix wrong sizeof in ReturnMetadataInfo format registration by @eisenhauer in Fix wrong sizeof in ReturnMetadataInfo format registration #4859
• Add Selection API for thread-safe Get() operations by @eisenhauer in Add Selection API for thread-safe Get() operations #4843
• BP5Reader: Fix a possible issue when writer map changes between steps by @scottwittenburg in BP5Reader: Fix a possible issue when writer map changes between steps #4865
• FileStdio: Fix transport issue when opening in append mode by @scottwittenburg in FileStdio: Fix transport issue when opening in append mode #4864
• Implement and Document Selection-based Get() by @eisenhauer in Implement and Document Selection-based Get() #4866
• Create parent directory of temporary .ats file. by @pnorbert in Create parent directory of temporary .ats file.  #4867
• security: publish score cards result to github security dashboard by @vicentebolea in security: publish score cards result to github security dashboard #4868
• ci: add dependabot coverage into our python deps by @vicentebolea in ci: add dependabot coverage into our python deps #4872
• Update assignees format in dependabot.yml by @vicentebolea in Update assignees format in dependabot.yml #4873
• Add support for version 2.11.x in SECURITY.md by @vicentebolea in Add support for version 2.11.x in SECURITY.md #4875
• build(deps): bump the all-actions group across 1 directory with 2 updates by @dependabot[bot] in build(deps): bump the all-actions group across 1 directory with 2 updates #4874
• Support xrootd connections in hosts.yaml, with xrootd/http/https tran… by @pnorbert in Support xrootd connections in hosts.yaml, with xrootd/http/https tran… #4870
• Eliminate old QueueGet by @eisenhauer in Eliminate old QueueGet #4869
• ci: add dependabot coverage into our conda deps by @vicentebolea in ci: add dependabot coverage into our conda deps #4877
• Pick archive hosts with 'fs' before 's3', so that xrootd is preferred… by @pnorbert in Pick archive hosts with 'fs' before 's3', so that xrootd is preferred… #4876
• ci: correct dependabot coverage of our conda deps by @vicentebolea in ci: correct dependabot coverage of our conda deps #4878
• ci: only do security updates in conda,pip,docker by @vicentebolea in ci: only do security updates in conda,pip,docker #4880
• remove thirdparty ci files by @vicentebolea in remove thirdparty ci files #4881
• [StepSecurity] Apply security best practices by @step-security-bot in [StepSecurity] Apply security best practices #4882
• build(deps): bump github/codeql-action from 4.32.3 to 4.32.4 in the all-actions group by @dependabot[bot] in build(deps): bump github/codeql-action from 4.32.3 to 4.32.4 in the all-actions group #4886
• Through-file streaming tweaks, add BP5 testing by @eisenhauer in Through-file streaming tweaks, add BP5 testing #4885
• Plugin engine interface by @eisenhauer in Plugin engine interface #4863
• Catch error thrown by Campaign engine when engine.all_blocks_info() i… by @pnorbert in Catch error thrown by Campaign engine when engine.all_blocks_info() i… #4892
• Fix segfault on empty array attribute by @eisenhauer in Fix segfault on empty array attribute #4893
• Pool curl request handles for connection reuse and batch requests. by @eisenhauer in Pool curl request handles for connection reuse and batch requests. #4888
• More complete plugin engine interface, document changes to all plugin interfaces, add whatsnew entry noting breaking change. by @eisenhauer in More complete plugin engine interface, document changes to all plugin interfaces, add whatsnew entry noting breaking change. #4894
• Add S3 write support with hybrid storage for BP5 by @eisenhauer in Add S3 write support with hybrid storage for BP5 #4831
• Better XRootD eviction and tar FD reuse by @eisenhauer in Better XRootD eviction and tar FD reuse #4895
• Minor XRootD tweaks (time-based eviction, docs and comment fix) by @eisenhauer in Minor XRootD tweaks (time-based eviction, docs and comment fix) #4897
• Refactor engine parameter parsing by @scottwittenburg in Refactor engine parameter parsing #4901
• remove spurious xrootd log message by @eisenhauer in remove spurious xrootd log message #4900
• Gray-scott docs update by @eisenhauer in Gray-scott docs update #4898
• ci: test fix sz3 by adding zstd by @vicentebolea in ci: test fix sz3 by adding zstd #4896
• Rename BP5 DataTransport parameter to DataFileTransport by @eisenhauer in Rename BP5 DataTransport parameter to DataFileTransport #4904
• Campaign in memory metadata by @pnorbert in Campaign in memory metadata #4903
• ci: add wheel integration tests by @vicentebolea in ci: add wheel integration tests #4906
• Automate backport via Copilot agent on backport:release label or PR description instruction by @Copilot in Automate backport via Copilot agent on backport:release label or PR description instruction #4909
• correct typo backport by @vicentebolea in correct typo backport #4910
• correct typo backport by @vicentebolea in correct typo backport #4912
• more capable bp5dbg (prototype) by @eisenhauer in more capable bp5dbg (prototype) #4908
• migrate to nanobind by @vicentebolea in migrate to nanobind #4887
• backport: revert migration to copilot by @vicentebolea in backport: revert migration to copilot #4917
• Rerouting aggregator by @scottwittenburg in Rerouting aggregator #4647
• build(deps): bump the all-actions group with 3 updates by @dependabot[bot] in build(deps): bump the all-actions group with 3 updates #4902
• Basic buffering in inline engine by @eisenhauer in Basic buffering in inline engine #4922
• Process and save in-memory-metadata object if file is part of a time-… by @pnorbert in Process and save in-memory-metadata object if file is part of a time-… #4923
• Add Asymmetric encryption support EncryptionOperator by @vicentebolea in Add Asymmetric encryption support EncryptionOperator #4883
• cmake: bump to c++ standard to c++17 by @vicentebolea in cmake: bump to c++ standard to c++17 #4907
• python: enable free threading python support by @vicentebolea in python: enable free threading python support #4925
• python: add windows sdist pip test by @vicentebolea in python: add windows sdist pip test #4924
• reuse: add REUSE compliance by @vicentebolea in reuse: add REUSE compliance #4928
• fix typo in dependabot config file by @vicentebolea in fix typo in dependabot config file #4931
• Remove copyright and license details from ReadMe.md by @vicentebolea in Remove copyright and license details from ReadMe.md #4934
• build(deps): bump the all-actions group with 5 updates by @dependabot[bot] in build(deps): bump the all-actions group with 5 updates #4932
• python: replace flake8/pylint/black with ruff by @vicentebolea in python: replace flake8/pylint/black with ruff #4930
• Add MPI barrier before test file cleanup to prevent race conditions by @eisenhauer in Add MPI barrier before test file cleanup to prevent race conditions #4936
• Restructure OnDemand test for accuracy and reliability by @eisenhauer in Restructure OnDemand test for accuracy and reliability #4937
• Add collective operations table for engines by @eisenhauer in Add collective operations table for engines #4940
• [StepSecurity] Apply security best practices by @step-security-bot in [StepSecurity] Apply security best practices #4942
• Various HTTP/XROOTD changes by @eisenhauer in Various HTTP/XROOTD changes #4926
• Campaign Reader engine doc. Some fixes to other files. by @pnorbert in Campaign Reader engine doc. Some fixes to other files. #4944
• build(deps): bump the all-actions group with 3 updates by @dependabot[bot] in build(deps): bump the all-actions group with 3 updates #4946
• Embed doc images rather than host at imgur by @eisenhauer in Embed doc images rather than host at imgur #4945
• only accept ACA 0.7, throw error on older and newer ACA versions by @pnorbert in only accept ACA 0.7, throw error on older and newer ACA versions #4947
• Performance improvements in null-data single-node write case by @eisenhauer in Performance improvements in null-data single-node write case #4950
• Defer hosts.yaml parsing until first access by @eisenhauer in Defer hosts.yaml parsing until first access #4951
• CI: add ubuntu/gcc12/mpich job using only external dependencies by @vicentebolea in CI: add ubuntu/gcc12/mpich job using only external dependencies #4949
• Remove GlobalServices, make Kokkos and PerfStubs lazy singletons by @eisenhauer in Remove GlobalServices, make Kokkos and PerfStubs lazy singletons #4952
• try just specifying the full path for clang in the individual job by @pnorbert in try just specifying the full path for clang in the individual job #4954
• Rewrite reading the database using a large join select to process dat… by @pnorbert in Rewrite reading the database using a large join select to process dat… #4956
• Use access/secret keys in AWS explicitly only if they are not empty, … by @pnorbert in Use access/secret keys in AWS explicitly only if they are not empty, … #4955
• update campaign management example, fix some warnings by @pnorbert in update campaign management example, fix some warnings #4957
• Restore ATOM_SERVER_HOST default for ADIOS by @eisenhauer in Restore ATOM_SERVER_HOST default for ADIOS #4958
• Bump version to v2.12.0-rc1 by @vicentebolea in Bump version to v2.12.0-rc1 #4961
• Backport pip: enable curl and openssl #4964 to release_212 by @ornladios-robot in Backport #4964 to release_212 #4966
• Fix BP5 when compression is applied to memory selections (Release Branch) by @eisenhauer in Fix BP5 when compression is applied to memory selections (Release Branch) #4971
• Backport rdma shutdown fix (Release Branch) by @eisenhauer in Backport rdma shutdown fix (Release Branch) #4972
• Suppress GCC 14 false positive -Wstringop-overflow in ReverseCopyFrom… by @eisenhauer in Suppress GCC 14 false positive -Wstringop-overflow in ReverseCopyFrom… #4975
• Remove pthread_cancel from RDMA progress thread shutdown (Release Branch) by @eisenhauer in Remove pthread_cancel from RDMA progress thread shutdown (Release Branch) #4973
• Backport evpath,pip: use internal transport libs #4993 to release_212 by @ornladios-robot in Backport #4993 to release_212 #4994
• Backport pip: enable curl #4995 to release_212 by @ornladios-robot in Backport #4995 to release_212 #4996
• Backport pip: enable curl #4997 to release_212 by @ornladios-robot in Backport #4997 to release_212 #4998
• backport from master by @vicentebolea in backport from master #5001
• Bump version to v2.12.0 by @vicentebolea in Bump version to v2.12.0 #5003

New Contributors

• @Crivella made their first contribution in Potential fix for CMake logic in using an external EVPath build #4697
• @hahnjo made their first contribution in Documentation fixes for beginner users #4746
• @AkiraFukuda made their first contribution in Integrate ProDM changes, added new operator RefactorProDM #4771
• @step-security-bot made their first contribution in [StepSecurity] Apply security best practices #4882

Full Changelog: v2.11.0...v2.12.0

---

This discussion was created from the release v2.12.0.