Best practices on writing attributes at large scale

[pgrete]

In the context of writing large outputs (several tens to hundreds of TB from 10k+ ranks) with attributes, I came across #4237 again and am now wondering about best practices.

For context: in our application (block structured adaptive mesh refinement) each attribute is known on 1 to N ranks, i.e., it may be known only on a single rank, on all ranks (guaranteed to be identical on all ranks) or on any number in between.

In the case where it's known on all ranks the solution in #4237 worked well as a single rank could just write it.

In the case were only a subset of ranks know about some attributes, I ran into an issue with this flag.
I (naively) assumed that adios2.attribute_writing_ranks = 0 would result in some internal communication so that attributes written by other ranks were transferred to the writing ranks (like the two level data write).
However, the attributes written by ranks other than the writing_ranks were just ignored and didn't make it to disk.

So I went back to "all ranks write" again just to bump into the GathervArrays restriction again.

My question is: is this sth I should fix in our code (i.e., communicate/aggregate the attributes to the known writing ranks) or what's the best approach/fix here?

[eisenhauer]

Hmm. attribute_writing_ranks seems to be an openpmd thing, not an ADIOS thing, so I don't know what that does.

Can I ask what version of ADIOS you are running with? We did make some significant improvements to metadata gathering, but unfortunately we haven't yet done a release that includes it. We hope to do the ADIOS 2.11 release in the coming weeks, but you might try running with ADIOS master from Github to see if it works better for you. If it does not, then it's something that I'd like to look at and see if I can fix.

[pgrete]

Hmm. attribute_writing_ranks seems to be an openpmd thing, not an ADIOS thing, so I don't know what that does.

Good point. I just checked the source and is indeed an openpmd option/feature. Given the adios2 prefix, I assumed that it'd be forwarded similar to other runtime parameters that are forwarded to adios2.

Can I ask what version of ADIOS you are running with? We did make some significant improvements to metadata gathering, but unfortunately we haven't yet done a release that includes it. We hope to do the ADIOS 2.11 release in the coming weeks, but you might try running with ADIOS master from Github to see if it works better for you. If it does not, then it's something that I'd like to look at and see if I can fix.

This is with 2.10.2. I'll give the latest dev version a shot and report back.
How significant is the expected overhead writing the same attributes from all ranks (or is this sth that has also been optimized in the current dev version)?

[eisenhauer]

The newest metadata aggregation methods do several things differently, including doing separate gathers on different types of information. BP5 tries hard to optimize the case where nothing much has changed from the previous timestep (I.E. no new variables written, no new or changed attributes, etc.). That often works out great, but generally it's not of any help on timestep 0 when everything is new. The metadata gathering in 2.11 splits the overall gather into 3 separate subgather's, but still manages to be more efficient than the 2.10 approach. Attributes are managed in the last of these subgather's and the overhead per new attribute is largely what you'd expect, extra bytes in the gather amounting to the length of the string name and a dozen more bytes of info, for each rank on which it is written. If you happen to do the not-uncommon thing where every rank writes the exact same set of attributes/values, we do eliminate duplicate sets before the gather happens (but this is done with a hash, so any difference between ranks makes us gather all the unique sets).