Skip to content

storeChunkRaw from GPU memory reverses the shape of multidimensional datasets with GPU-enabled ADIOS2 #1957

Open

Description

@BenWibking

Summary

With an ADIOS2 build that has GPU support, storeChunkRaw from a cudaMalloc pointer into a dataset with two or more dimensions goes wrong:

  1. With one chunk, the dataset is written with its extent reversed. A dataset declared {2, 3} reads back as {3, 2}.
  2. With several chunks, the second storeChunkRaw fails openPMD-api's own verifyDataset check with "Dataset access out of bounds".

The root cause is in ADIOS2 (ornladios/ADIOS2#5241). On the first Put of a device pointer, ADIOS2 switches the variable to column-major and reverses its shape, start and count. openPMD-api always describes datasets in C order and never calls adios2::Variable::SetMemorySpace or SetArrayLayout, so this default always applies. openPMD-api then reads the reversed shape back from ADIOS2, which explains case 2.

1-D records such as particle data are unaffected. Without GPU support in ADIOS2 (or with CUDA managed memory, which ADIOS2 doesn't classify as device memory) the pointer is treated as host memory and works.

Reproducer

// Store a 2x3 mesh from a cudaMalloc buffer with storeChunkRaw,
// as one chunk (case 1) and as two row chunks (case 2).
#include <cuda_runtime.h>
#include <iostream>
#include <openPMD/openPMD.hpp>
#include <vector>

int main()
{
	const std::vector<double> host{0, 1, 2, 3, 4, 5}; // 2 rows x 3 columns, row-major
	double *device = nullptr;
	cudaMalloc(&device, host.size() * sizeof(double));
	cudaMemcpy(device, host.data(), host.size() * sizeof(double), cudaMemcpyHostToDevice);
	{
		openPMD::Series series("gpu_chunks_%T.bp", openPMD::Access::CREATE);
		auto iteration = series.iterations[0];
		for (const char *name : {"one_chunk", "two_chunks"}) {
			iteration.meshes[name][openPMD::RecordComponent::SCALAR].resetDataset({openPMD::Datatype::DOUBLE, {2, 3}});
		}
		series.flush();
		iteration.meshes["one_chunk"][openPMD::RecordComponent::SCALAR].storeChunkRaw(device, {0, 0}, {2, 3});
		series.flush();
		std::cout << "case 1 (one chunk) stored\n";
		auto two = iteration.meshes["two_chunks"][openPMD::RecordComponent::SCALAR];
		try {
			two.storeChunkRaw(device, {0, 0}, {1, 3});
			series.flush();
			two.storeChunkRaw(device + 3, {1, 0}, {1, 3});
			series.flush();
			std::cout << "case 2 (two chunks) stored\n";
		} catch (const std::exception &e) {
			std::cout << "case 2 (two chunks) threw: " << e.what() << '\n';
		}
		series.close();
	}
	openPMD::Series series("gpu_chunks_%T.bp", openPMD::Access::READ_ONLY);
	auto record = series.iterations[0].meshes["one_chunk"][openPMD::RecordComponent::SCALAR];
	const auto extent = record.getExtent();
	const auto data = record.loadChunk<double>();
	series.flush();
	std::cout << "case 1 read back: extent {" << extent[0] << "," << extent[1] << "}, data =";
	for (int i = 0; i < 6; ++i) {
		std::cout << ' ' << data.get()[i];
	}
	std::cout << '\n';
	cudaFree(device);
}

Output (relevant lines)

case 1 (one chunk) stored
case 2 (two chunks) threw: [ADIOS2IOHandlerImpl::verifyDataset()] Dataset access out of bounds.
Note: Variable '/data/0/meshes/two_chunks' has shape [3, 2], is accessed from offset [1, 0] with extent [1, 3] (no joined dimension).
case 1 read back: extent {3,2}, data = 0 1 2 3 4 5

bpls shows /data/0/meshes/one_chunk as {3, 2}. The exception in case 2 also leaves the series without its mesh attributes, which causes many Could not read expected attribute warnings on read; those are a side effect of the reproducer, not part of the bug.

Suggested fix

openPMD-api knows its datasets are row-major, so it can set the layout explicitly. In VariableDefiner::call (src/IO/ADIOS/ADIOS2IOHandler.cpp, after IO.DefineVariable), call:

var.SetArrayLayout(adios2::ArrayOrdering::RowMajor);

This needs a version guard for ADIOS2 releases without ArrayOrdering. In a plain ADIOS2 test (in ornladios/ADIOS2#5241), this call keeps a device-buffer Put row-major and stores correct data.

In plain ADIOS2, reading into device memory gave correct results in our test. We haven't tested openPMD-api's read path with device buffers. verifyDataset reads Shape() from the variable, though, so setting the layout on read-side variables too may be worthwhile.

As a stopgap, documenting that raw device pointers are only safe for 1-D records with GPU-enabled ADIOS2 would help. Applications can work around the problem with the span-based storeChunk<T>(offset, extent, createBuffer) and a device-to-host copy into the returned buffer. WarpX does this, and we have adopted it in Quokka.

Environment

  • openPMD-api 0.17.0-68-gff916100 (dev branch, 2026-09-15)
  • ADIOS2 2.12.1 (git f5267290) with ADIOS2_USE_CUDA=ON; BP5 engine
  • CUDA 13.4, NVIDIA driver 615.71.09, NVIDIA RTX PRO 6000 Blackwell Max-Q (sm_120)
  • Rocky Linux 10.2, GCC 14.3.1, OpenMPI

Activity

  1. franzpoeschel commented on Oct 30, 2026

    @franzpoeschel
    Contributor

    The suggested fix makes openPMD's behavior diverge from ADIOS2's. Personally, I dislike ADIOS2's behavior in this instance, so I would tentatively agree. But we should allow an opt-out, e.g.:

    {"adios2": {"dataset": {"array_ordering": X}}}

    Where X is one of "row_major", "column_major" or "adios2_native". We should make clear that this does not entail any row/column-major semantics in openPMD (that would be a larger undertaking), but is just a forwarding option to ADIOS2.

    And then decide which one to make the default..

  2. BenWibking commented on Nov 3, 2026

    @BenWibking
    Author

    Rather than adding new behavior to work around bugs in upstream ADIOS2, I've extracted the parts of ADIOS2 that openPMD-api uses and put it in a standalone repo here with correctly-behaving device pointers (that behave identically to CPU pointers in upstream ADIOS2 today): https://github.com/BenWibking/bp5lite.

    You can link openPMD against it without any modifications, treating it as if it were ADIOS2. It is BP5-only, and does not support streaming transports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions