Truncate option for Engine Open mode Append

[franzpoeschel]

Why is this feature important?
Scenario:

• An application uses ADIOS for checkpointing and for regular output
• A new bp file is created per checkpoint
• Regular output keeps appending new steps to the same bp file
• Checkpointing frequency is lower than regular output checkpointing

Consequence: After restarting from a checkpoint:

1. The output file needs to be opened in Append mode
2. Some simulation steps are written for a second time

=> The output file will contain the same steps twice

Solution: Allow to truncate a number of last steps when opening a file for appending

What is the potential impact of this feature in the community?
ADIOS steps can be used more sensibly for output in simulations that use checkpointing

Is your feature request related to a problem? Please describe.
see above

Describe the solution you'd like and potential required effort
see above
effort depends on the file format

Describe alternatives you've considered and potential required effort
Alternative: Just accept that the output file will have some time steps twice. Required effort: Additional complexity in reading data produced by ADIOS

Additional context
Discussed offline with @pnorbert and @ax3l
openPMD/openPMD-api#1007

[ax3l]

Solution: Allow to truncate a number of last steps when opening a file for appending

Just to add, more precisely, we would like to specify "truncate all steps after ADIOS step <N>"

[pnorbert]

@franzpoeschel BP5 in the master branch now has a parameter AppendAfterSteps. It enables overwriting some steps as you requested. If you have 10 steps in the file,

• value 0 means starting from the beginning, truncating all existing data
• value 1 means appending after the first step, so overwrite 2,3...10
• value 10 means appending after all existing steps
• value >10 means the same, append after all existing steps (gaps in steps are impossible)
• -1 means appending after the last step, i.e. same as 10 or higher
• -2 means removing the last step, i.e. starting from the 10th
• -11 (and <-11) means truncating all existing data

Default value is MAX_INT, so it always appends after the last step. -1 would be another good default value.

We won't add this feature to BP4 because it would require implementing the entire metadata parsing process of the read side in the writing engine.

[franzpoeschel]

Great, thanks for this!

[pnorbert]

This is finally going to work properly in 2.8.1