[OpenMP] `AddParticles(pc & other)` Broken in ImpactX

[ax3l]

When using more than one OpenMP thread in ImpactX, the addParticles(pc & other) does not work. This might be due to the particular init logic we use in ImpactX for particles when not using collective effects (i.e., redistribute is never called)?

Here is a reproducer:

• pytest-benchmark for all elements BLAST-ImpactX/impactx#987
• uses ParticleContainer: add_particle(other) & make_alike pyamrex#447

In detail, we draw a particle distribution once for a series of tests (which is time consuming) and then copy in the exact same particle container over and over for following tests (to be reproducible in exact particles that we had drawn for other tests & be faster than drawing again).

# once we do
sim.add_particles(bunch_charge_C, distr, npart)

sim.backup_beam = self.sim.particle_container().make_alike()  # note: not sure if the issue could also be here?
self.sim.backup_beam.add_particles(self.sim.particle_container())

# then repeated for each test we do

# instead of drawing from the distribution again, create a 2nd
# particle container and copy the same initial particles again.
pc = sim.particle_container()
pc.clear_particles()
pc.add_particles(sim.backup_beam)  # note: uses default of local=False (no difference if I use local=True)

assert pc.total_number_of_particles() == npart  # this fails for more than one OMP thread

results in total_number_of_particles() (with default arguemtns) to return Npart / OpenMP threads.

amrex/Src/Particle/AMReX_ParticleContainer.H

Lines 574 to 582 in cef9e0c

	/**
	* \brief Returns # of particles at all levels
	*
	* If "only_valid" is true it only counts valid particles.
	*
	* \param only_valid
	* \param only_local
	*/
	Long TotalNumberOfParticles (bool only_valid=true, bool only_local=false) const;

CC @atmyers @WeiqunZhang

[ax3l]

I think the fix will be similar to the fix in #2695.

Calling Redistribute() before addParticles(other) is a work-around for now. But in ImpactX, we have a central mode where we do never call Redistribute() for performance & perfect load balancing reasons.

[ax3l]

Ok, but the downside of calling Redistribute() as a work-around for this benchmark is that the OpenMP parallelism does not work anymore... So we need to really check what is wrong in addParticles() (and potentially make_alike, too).

[ax3l]

Debugged today with @atmyers. The issue is likely in make_alike, where we need to also set the tile sizes of the new pc to the other ones to be indentical in layout.

Currently, the addParticles in ImpactX leaves out tiles that do not exist on both the new (make_alike) and other pc, because the tile sizes do not match:
https://github.com/AMReX-Codes/amrex/blob/25.06/Src/Particle/AMReX_ParticleContainerI.H#L1170

[atmyers]

We found the issue, the fix was on the ImpactX side (the benchmark in question forgot to set particles.do_tiling): BLAST-ImpactX/impactx#1012