Skip to content

Conversation

franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented Dec 16, 2024

TODO:

  • MPI-aware preparsing
  • Maybe through out MPI-awareness of preparsing again
  • Edge case: What if a particle species completely vanishes after some iteration, making it appear as an empty, i.e. constant component -> unsupported case, backend will throw
  • Better testing
  • Documentation, examples
  • Replace group-based encoding with variable-based encoding more thoroughly for ADIOS2 (maybe follow-up)
  • Can we make variable-based encoding the default output for ADIOS2? (maybe follow-up for easier reversibility)
  • Check if the engine supports this mode ---> would include HDF5 engine, but it has no first-class support, so setting group-based encoding manually is fine
  • Check if the datasets have the same number of steps as there are snapshots
  • Add sth like "UnselectStepSelection" for reading /rankTable
  • follow-up: maybe pre-parse the entire thing in linear encoding and only then switch to random-access?
  • follow-up: Preparsing also in READ_LINEAR (i.e. eliminate duplicate Iterations better than now)
  • For defaulting v-encoding: Do attributes such as dt or time make sense here at all? ---> Need to preparse those, too. Check all modifiable attributes upon first READ_LINEAR open. --> follow-up, print warning for now
  • Maybe use openPMD 2.0 to introduce a more breaking default for v-based encoding in ADIOS2? --->follow-up
  • Merge ADIOS2: Require v2.9 as minimum #1711 first

@franzpoeschel franzpoeschel added backend: ADIOS2 api: new additions to the API labels Dec 16, 2024
@franzpoeschel franzpoeschel force-pushed the random-access-steps branch 2 times, most recently from 4f7284b to 585b084 Compare December 19, 2024 13:35
@franzpoeschel franzpoeschel force-pushed the random-access-steps branch 5 times, most recently from 238fcd1 to 688ffa9 Compare January 15, 2025 14:23
#endif
}

TEST_CASE("automatic_variable_encoding", "[adios2]")

Check notice

Code scanning / CodeQL

Unused static function Note test

Static function C_A_T_C_H_T_E_S_T_56 is unreachable (
autoRegistrar57
must be removed at the same time)
@ax3l ax3l requested review from ax3l and guj February 18, 2025 17:58
@ax3l ax3l self-assigned this Feb 18, 2025
@ax3l
Copy link
Member

ax3l commented Feb 18, 2025

#1711 merged, ready for rebase :)

@franzpoeschel
Copy link
Contributor Author

#1711 merged, ready for rebase :)

done ;)

@franzpoeschel
Copy link
Contributor Author

Note to self: Benchmark this on a Series with many Iterations, compare with group/file encoding and with variable encoding in linear access


struct StepSelection
{
std::optional<size_t> step;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please include <optional> for stability in this file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, I've also removed an unused <map> import

@ax3l ax3l merged commit ff52a8d into openPMD:dev Feb 20, 2025
30 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: new additions to the API backend: ADIOS2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants