Crash in restarting from a checkpoint with different number of MPI ranks

[chongchonghe]

Describe the bug
When restarting from a sim with particles (e.g. sink particles) and the particles are all at finest level 3, the code crashes with the following error. It tried to load particles from level 0, 1 and 2 and failed. This only occurs when using different number of MPI processes when restarting.

  54 Reading last_chk/metadata.yaml...
  55     quokka_version = 25.03
  56     git_hash_quokka = a86c5e8272c06df95f9bc31354faec31e6d8df41
  57     git_hash_amrex = c4fcda1fb2ea238c3e6fa273dc9cfcde54b4cc51
  58     units = (complex type)
  59     constants = (complex type)
  60     SFH_StochasticStellarPop_particles = (complex type)
  61     SFH_Sink_particles = (complex type)
  62 amrex::Error::0::Couldn't open file: last_chk/Sink_particles/Level_2/DATA_00013 !!!
  63 SIGABRT
  64 amrex::Error::1::Couldn't open file: last_chk/Sink_particles/Level_2/DATA_00004 !!!
  65 SIGABRT
  66 amrex::Error::4::Couldn't open file: last_chk/Sink_particles/Level_2/DATA_00000 !!!
  67 SIGABRT
  68 amrex::Error::6::Couldn't open file: last_chk/Sink_particles/Level_2/DATA_00004 !!!
  69 SIGABRT
  70 amrex::Error::7::Couldn't open file: last_chk/Sink_particles/Level_2/DATA_05223 !!!
  71 SIGABRT
  72 amrex::Error::129::Couldn't open file: last_chk/Sink_particles/Level_2/DATA_00000 !!!
  73 SIGABRT
  74 amrex::Error::130::Couldn't open file: last_chk/Sink_particles/Level_0/DATA_00417 !!!
  75 SIGABRT
  76 amrex::Error::132::Couldn't open file: last_chk/Sink_particles/Level_0/DATA_426099344 !!!
  77 SIGABRT
  78 amrex::Error::133::Couldn't open file: last_chk/Sink_particles/Level_1/DATA_388525280 !!!
  79 SIGABRT
  80 amrex::Error::161::Couldn't open file: last_chk/Sink_particles/Level_1/DATA_70970 !!!

To Reproduce (with a smaller problem)
Steps to reproduce the behavior:

1. Compile this problem 'ParticleRadiation'. Set 'max_level=1' and define refine_grid to refine all grids.
2. Run this problem 'ParticleRadiation' to step 3 with 1 process. Then, restart with 8 processes.
3. See error.

Additional context
Add any other context about the problem here.

[chongchonghe]

@BenWibking Just to confirm, we do not expect any issue when restarting a checkpoint with different number of processes, right?

[dosubot]

Quokka is designed to support checkpoint restarts with a different number of MPI processes, including for simulations with particles. The restart logic explicitly rebuilds grids and redistributes particle data to match the new process layout, and this is covered by automated tests in the codebase checkpoint_restart_refine_test.sh.

However, there is a known caveat: if particle descriptors are not re-registered before regridding after a restart, forced refinement tags may be lost, causing the finest AMR level (and its particles) to disappear. This can lead to missing particle data errors, especially if all particles are at the finest level simulation.hpp issue #1449. If you see this, check that particle registration and refinement tagging happen in the correct order during restart.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[BenWibking]

@BenWibking Just to confirm, we do not expect any issue when restarting a checkpoint with different number of processes, right?

Right. It's supposed to work.

I vaguely remember seeing a similar restart issue in the WarpX repo recently. It might be worthwhile to scan their issues or have Codex or Gemini search their recent commits for fixes for particle restarts.

[BenWibking]

@chongchonghe Is this still an issue? AMReX had several bug fixes related to particle I/O recently.

[BenWibking]

@chongchonghe Is this still an issue?

[chongchonghe]

I'll test it and report.

[BenWibking]

Since this is fixed in AMReX, I am going to close this issue for now. If it reappears, please feel free to re-open it.