IterationEncoding: variableBased

FORMAT_ADIOS.md

@@ -32,3 +32,28 @@ Output from `bpls -A` for a boolean attribute `pybool` stored in the location of
 
 There is no convention yet for a unique representation of ADIOS2 variables with boolean type.
 Thus, implementations should cast the data to and from `unsigned char` instead.
+
+## `stepBased` Encoding of Iterations
+
+The `iterationEncoding` mode `stepBased` must be implemented via ADIOS steps.
+
+## Datasets
+
+An openPMD **data set** is represented by an group prefix that contains an ADIOS variable `__data__`.
+
+**attributes** are defined further below and can also appear at the dataset's **group** prefix level.
+
+## Attributes
+
+openPMD **attributes** stored as ADIOS `Variables` at the location where they would usually be stored.
+
+Example for a mesh record `E` with record component `x` and attributes `unitDimension` and `unitSI`:
+```
+  double    /data/meshes/E/unitDimension     10*{7}
+  double    /data/meshes/E/x/__data__        10*{1000}
+  double    /data/meshes/E/x/position        10*{1}
+  double    /data/meshes/E/x/unitSI          10*scalar
+```
+
+This example uses `stepBased` iteration encoding, but other iteration encodings would work similarly with their respective `basePath` prefix.
+

STANDARD.md

@@ -48,7 +48,9 @@ for changes in keywords).
 Hierarchy of the Data File
 --------------------------
 
-The used hierarchical data file format must provide the capability to
+For simplicity, we call the storage concept of a specific data format that implements the openPMD hierarchy "files", even if they are implemented in-memory or by other means.
+
+The used hierarchical data format must provide the capability to
 
   - create groups and sub-groups (in-file directories)
   - create multi-dimensional, homogeneous array-based data structures
@@ -85,7 +87,10 @@ Each file's *root* group (path `/`) must at least contain the attributes:
                    to create a real path from it replace all occurrences
                    of `%T` with the integer value of the iteration, e.g.,
                    `/data/%T` becomes `/data/100`
-    - allowed value: fixed to `/data/%T/` for this version of the standard
+    - allowed values:
+      - see *Iterations and Time Series* below
+      - for `fileBased` and `groupBased`, this is fixed to `/data/%T/`
+      - for `stepBased` this is fixed to `/data/`
     - note: all the data that is formatted according to the present
       standard (i.e. both the meshes and the particles) is to be
       stored within a path of the form given by `basePath` (e.g. in
@@ -195,9 +200,8 @@ standard:
 Iterations and Time Series
 --------------------------
 
-Iterations can be encoded in either the file name of each master-file of a
-time step or in groups of the same file. (Here, an *iteration* refers
-to a single simulation cycle.)
+Iterations can be encoded in either the file name of each individual files, in groups of the same file, or in data sets & attributes (with supported data formats).
+(Here, an *iteration* might refer to a single measurement or simulation cycle.)
 
 The chosen style shall not vary within a related set of iterations.
 
@@ -212,6 +216,7 @@ Each file's *root* group (path `/`) must further define the attributes:
     - allowed values:
       - `fileBased` (multiple files)
       - `groupBased` (one file)
+      - `stepBased` (one file with internal encoding for iterations, if supported by the data format)
 
   - `iterationFormat`
     - type: *(string)*
@@ -222,20 +227,32 @@ Each file's *root* group (path `/`) must further define the attributes:
                    for `fileBased` formats the iteration must be included
                    in the file name;
                    the format depends on the selected `iterationEncoding` method
+    - note: it is not required that every openPMD iteration contains an update for each declared openPMD record (see below)
     - examples:
       - for `fileBased`:
         - `filename_%T.h5` (without file system directories)
-      - for `groupBased`:
+      - for `groupBased`: (fixed value)
         - `/data/%T/` (must be equal to and encoded in the `basePath`)
+      - for `stepBased`: (fixed value)
+        - data-format internal convention
+        - *slowest varying index* of data
+
+### `stepBased` Encoding of Iterations
+
+In order to correlate openPMD iterations with an index of data-format internal updates/steps or an index in the slowest varying dimension of an array, the *root* group (path `/`) must contain an additional variable once `stepBased` is chosen for `iterationEncoding`:
+
+  - `snapshot`
+    - type: 1-dimensional array containing N *(int)* elements, where N is the number of updates/steps in the data format
+    - description: for each update/step in a data format, this variable needs to be updated with the corresponding openPMD iteration.
+    - note: in some data formats, updates/steps are absolute and not every update/step contains an update for each declared openPMD record
+    - advice to implementers: an openPMD iteration might be spread over multiple updates/steps, but not vice versa.
+                              In such a scenario, an individual openPMD record's update/step must appear exactly once per iteration.
 
 
 Required Attributes for the `basePath`
 --------------------------------------
 
-In addition to holding information about the iteration, each series of
-files  (`fileBased`) or series of groups (`groupBased`) should have
-attributes that describe the current time and the last
-time step.
+In addition to holding information about the iteration, each series of files  (`fileBased`), series of groups (`groupBased`) or internally encoded iterations (`stepBased`) should have attributes that describe the current time and the last time step.
 
  - `time`
    - type: *(floatX)*