-
Notifications
You must be signed in to change notification settings - Fork 45
Description
Current capability
At present, reproducibility in Celeritas requires:
- keeping the same number of threads across runs,
- sorting "track initializers" from secondaries (daughter particles) so that they are produced in an ordered slot (or queued for later injection), and
- preventing new primaries (source particles) from being added during the transport solve.
Additionally, supporting multiple Geant4 CPU threads and framework-selectable events requires us to additionally call rng_reseed on each per-CPU state vector at each new event.
Goal
We want to be able to share a single state vector across multiple Geant4 CPU threads, interleaving EM/optical primaries offloaded from multiple CPU threads (assume the same ordering of primaries coming from each CPU thread, regardless of how Celeritas interacts with it). After this and all the subtasks on #2014 are complete, then we can truly have an asynchronous optimized transporter.
There is a side benefit to having RNG states that only depend on the parent: a requirement by #1349 that killing a track prematurely for efficiency has minimal impact on the invidual (not statistical!) behavior of the particles in the rest of the simulation. Currently that's not true because killing tracks causes all the subsequently tracked particles to have different RNG states.
Implementation plan
- Implement ranluxpp because it's been deemed "high quality" and can be compared against adept results. (Support RANLUX++ as core RNG #2141)
- Add "spawning" from an RNG state. From what I can tell, the ADePT spawning code takes two successive RNG states in RANLUX space, XORs them, and assumes that the result is well distributed. (With 2^576 RNG states, and ~4 doubles per RNG state, this means we're likely never going to repeat if we're getting a truly well distributed sample. But I feel like we need a mathematician to analyze the correlation between two sequential RNG states and whether xoring them is actually going to be well distributed.) It's unclear if this will work as well for a "shuffling" RNG engine like XORWOW that already has XOR built in to its requirements.
- Add the ability to seed from an EM primary state. (For photons the primary would come from the main stepping loop, but our primary concern is Geant4 primaries.) I think event number + track number + step number is sufficient to make a unique seed, but ADePT also hashes the direction, position, and energy. I don't think we should hash into a 64-bit integer; I think we can seed into a much larger state space or even do something simple like .
- Prototype (hack to pieces) a working implementation of using these two different "seed" methods instead of sorting secondaries
- If the performance is just as good we can just make that the new implementation; or if it's somewhat simple to add an option to keep both for testing/publication we can do that too; or if we really want to be hacky we can add another compile-time option.
- A follow-on issue will tackle how to correctly feed multiple primaries into a single state vector and send hits back.
The hard part
For step 4 above, a few needs spring to mind:
- Add RNG states (the 9-element array, not the integer offset) to the track initializers.
- Add individual seeds to the primaries, rather than using "event seed" plus the primary ID.
- Maybe need to add another track sorting strategy or initializer sorting strategy