What is the optimal workflow using pset.from_particlefile(...,restart=True) and MPI?

[PeterWolfram]

Question

Question

I am running large simulations (3Mio. particles for around 20 years) on a HPC with a walltime limit of 8h per job. I use MPI to run the simulation as 20 MPI jobs. Due to the walltime limit I have to run my simulation in chunks of 1-2 years.
Because I have been having problems with Out-Of-Memory Errors while restarting from a merged .zarr store I tried per-rank restarting, so every MPI rank restarts from its previous output. That worked in principle, but particles got shuffled around along the trajectory dimension, so one particles was attributed to a different trajectory index / value every chunk. I tried to circumvent this problem by having every rank append to its procXX.zarr (by setting "create_new_zarrfile=False" for higher chunks) instead of creating a new .zarr store. This did not solve the problem.
It should be noted, that this problem only arises if I release my particles at different times, even if all particles are being released within the first chunk.

My question now is: what would be a clean workflow for this kind of simulation? Are there any Parcels tools and functionalities I am missing?

I am grateful for any advice.

Supporting code

I perform the per-rank restarting using

pset_old = ps.ParticleSet.from_particlefile(fieldset,
                                          pclass=pclass,
                                          filename=input_dir,
                                          partition_function=False,
                                          restart=True)

where input_dir is the rank specific .zarr store. The output ParticleFile is

if init_globals.CHUNK_IDX == 1:
        create_new_zarrfile = True
else:
        create_new_zarrfile = False
output = pset.ParticleFile(
        name=fname, 
        outputdt=out_dt, 
        chunks=cs_out, 
        create_new_zarrfile=create_new_zarrfile
    )

[erikvansebille]

Thanks for asking, @PeterWolfram. I don't have much experience with output and MPI, but @JamiePringle has so might be able to weigh in?