setForceFinestLevel() failed on macOS

[chongchonghe]

Describe the bug
setForceFinestLevel(true) does not work properly on macOS, although it does on CUDA or ROCm (since the BinaryOrbitCIC test passed in CI tests).

To Reproduce
Compile the BinaryOrbitCIC problem on macOS and

1. Compile this problem 'BinaryOrbitCIC'
2. Run this problem ninja src/problems/BinaryOrbitCIC/test
3. See error: the BinaryOrbitCICRefactor will fail.

The failure is because the BinaryOrbitCICRefactorInit test produces a snapshot with max_level=0, but it should have max_level=1 since the problem called sim.particleRegister_.getParticleDescriptor(quokka::ParticleType::CIC)->setForceFinestLevel(true); and has amr.max_level=1. This causes the restart from checkpoint to fail. The log message shows that max_level=1 at step 19 but become 0 at step 20. I tried to increase amr.n_error_buf to 8 and change grid_eff, but the bug still exists.

Coarse STEP 19 at t = 154195.1209 (0.3083902417%) starts ...
...[level 0] estimated hydro timestep: 1.158381e+04
...[level 0]    hydro timestep limited at cell [14, 15, 14] with signal speed = 4.046595e+07
...[level 0]    cell density = 1.255582e-22, |v| = 2.746595e+07, cs = nan
...[level 0] estimated particle timestep: 1.797693e+308
...[level 0]    max particle velocity: 0.000000e+00
...[level 0]    particle timestep limited at position [nan, nan, nan]
...[level 0] timestep limited by HYDRO
...[level 1] estimated hydro timestep: 5.154591e+03
...[level 1]    hydro timestep limited at cell [36, 33, 31] with signal speed = 4.546917e+07
...[level 1]    cell density = 4.593971e-22, |v| = 3.246917e+07, cs = nan
...[level 1] estimated particle timestep: 3.266997e+04
...[level 1]    max particle velocity: 1.195670e+07
...[level 1]    particle timestep limited at position [2.570965e+12, 1.545021e+12, -2.221424e+11]
...[level 1] timestep limited by HYDRO
...coarse timestep set by level 1
ParticleContainer::Redistribute() resizing m_particles from 2 to 1
ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 128 (2), Max: 128 (2), Total: 128 (2)]
ParticleContainer::Redistribute() time: 3.8291e-05

[Level 0 step 19] ADVANCE with time = 1.5419512087e+05 dt = 5.1545913309e+03
[Level 0 step 19] Advanced 32768 cells
ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 128 (2), Max: 128 (2), Total: 128 (2)]
ParticleContainer::Redistribute() time: 3.775e-05

Gravity BCs: x:periodic y:periodic z:periodic 
Using MLMG solver with fully periodic boundaries...

MLMG: Initial rhs               = 1.306233436e-09
MLMG: Initial residual (resid0) = 1.306233436e-09
MLMG: Final Iter. 3 resid, resid/bnorm = 1.009461931e-14, 7.728036227e-06
MLMG: Timers: Solve = 0.006944792 Iter = 0.006336208 Bottom = 4.4501e-05
MLMG converged with final residual norm: 1.009461931e-14


Coarse STEP 20 at t = 159349.7122 (0.3186994244%) starts ...
...[level 0] estimated hydro timestep: 1.164947e+04
...[level 0]    hydro timestep limited at cell [14, 15, 14] with signal speed = 4.023789e+07
...[level 0]    cell density = 1.287219e-22, |v| = 2.723789e+07, cs = nan
...[level 0] estimated particle timestep: 6.520405e+04
...[level 0]    max particle velocity: 1.198162e+07
...[level 0]    particle timestep limited at position [2.533906e+12, 1.591854e+12, -2.376092e+11]
...[level 0] timestep limited by HYDRO
...coarse timestep set by level 0
[Level 0 step 20] ADVANCE with time = 1.5934971220e+05 dt = 5.6700504640e+03
[Level 0 step 20] Advanced 32768 cells
ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 128 (2), Max: 128 (2), Total: 128 (2)]
ParticleContainer::Redistribute() time: 3.7375e-05

Additional context
Add any other context about the problem here.

[dosubot]

There’s no platform-specific code in Quokka’s AMR or particle refinement logic, so setForceFinestLevel(true) should behave the same on macOS and CUDA/ROCm. The most likely cause for level 1 disappearing is that, during regridding, the tag array is empty—meaning no cells are tagged for refinement, so AMReX prunes the fine level even though amr.max_level=1 remains set. This can happen if particles aren’t present or not tagged at the right time, or if the tagging logic (ErrorEst → refineGridsAroundParticles) isn’t executed as expected on macOS. After a restart, if the particle descriptor isn’t re-registered before regridding, the forced tags won’t be applied and the refined level won’t persist, which matches your symptoms source.

There’s no evidence of macOS-specific logic, so the issue is likely due to runtime state, initialization order, or subtle differences in how/when particle registration and tagging occur on macOS. To debug, try building with -DCMAKE_BUILD_TYPE=Debug or -DENABLE_ASAN=ON to catch any memory or array access bugs that might affect particle or tag state, and compare logs of particle counts and tag arrays before/after the fine level disappears source.

Focus on confirming that particles are present and registered with forceFinestLevel set before any regridding or checkpoint/restart logic runs. If the problem persists, instrument the tagging and regridding code to verify that tags are being set as expected on macOS.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[BenWibking]

I am guessing there is a subtle floating point optimization problem.