Create four problem configurations for Branson

[gshipman]

The following problem configurations need to be made for Branson.
These problems should have scripts / input decks.
Scripts / input decks must be version controlled, either in the llnl/benchmarks or in lanl/branson
We need to determine if we will use a branch on lanl/branson or a tag or ..

In order of priority:

• GPU, decomposed, history, SoA
• GPU, decomposed, event, SoA
• GPU, decomposed, event, AoS
• CPU, decomposed, history, SoA

We need to do a throughput study on the GPUs to understand the GPU saturation.

• Single node, plot the computation rate (segments/second or particles/second) as a function of number of particles, and find the point where the speed saturates. Possibly starting with 2M particles per GPU, and increasing the number of particles until the problem OOMs.

[JDTruj2018]

Currently have the GPU build of Branson built and running the default input deck (3D_holhraum_single_node.xml) on a single node of Tuolumne (1, 2, 4, 8, etc. MPI ranks to start as a sanity check).

Ran both mapping 1 XCD per MPI:
flux run -N 1 -n 24 -g 1 --setopt=mpibind=verbose:1 --exclusive ${BRANSON_BIN} ${BRANSON_INPUT}

And letting flux determine mappings and share XCDs between ranks:
flux run -N 1 -n 48 -g 0 --setopt=mpibind=verbose:1 --exclusive ${BRANSON_BIN} ${BRANSON_INPUT}

Ranks	XCD-per-Rank	FOM
1	1	1.05292e+06
2	1	2.02502e+06
4	1	3.97532e+06
8	1	7.83382e+06
16	1	1.38329e+07
24	1	1.60203e+07
1	–	1.01577e+06
2	–	2.00085e+06
4	–	3.97725e+06
8	–	7.97348e+06
16	–	1.39408e+07
24	–	1.61112e+07
32	–	1.29547e+07
48	–	1.59649e+07

I'll update this script to run in CPX mode with hugepages=512GB and 2 ranks per XCD per Alex's suggestion, and start working on generating throughput data.

Will work on getting each configuration running on a single node of Tuolumne. I've pushed to a fork of lanl/branson for now (https://github.com/JDTruj2018/branson/tree/FCR), and scripts will be on the FCR branch.

[JDTruj2018]

First pass at Single-Node Throughput Results on Tuolumne:

I ran in CPX mode, with hugepages=512GB, and 48 MPI ranks (2 per logical GPU) as Alex suggested. FOM increases rapidly up to about 4,000,000 particles and then slowly increases up to 800,000,000 particles before OOMing at 900,000,000 particles. There is not discernible drop in performance before OOMing.

I’d say the plateau is around ~200,000,000 particles per node (50,000,000 per physical APU, 83,333,333 per logical GPU, or 41,666,667 per MPI rank).

[JDTruj2018]

I am running in to a few challenges on multi-node runs and wanted to check with you all in case I missed something.

0. The default 3D_hohlraum_multi_node.xml problem (this is not one of the 4 configs Galen listed in issue Create four problem configurations for Branson #25): GPU, decomposed, history, AoS
   a. The code runs fine on 2 nodes (CPX mode) for up to 12 tasks per node (2 devices or logical GPUs per MPI rank). Beyond that, attempting 24 or 48 tasks per node appears to cause the code to hang. I haven’t yet tested this on more than 2 nodes.

This runs:
flux run -N 2 --tasks-per-node=12 --setopt=mpibind=verbose:1 --exclusive ~/binaries/branson-test/bin/BRANSON ~/branson-git/inputs/3D_hohlraum_multi_node.xml

This hangs:
flux run -N 2 --tasks-per-node=24 --setopt=mpibind=verbose:1 --exclusive ~/binaries/branson-test/bin/BRANSON ~/branson-git/inputs/3D_hohlraum_multi_node.xml

1. Modified 3D_hohlraum_multi_node.xml problem: GPU, decomposed, history, SoA
   a. Same issue as 0a
   b. For runs with 12 or fewer tasks per node, I am getting a non-fatal error: “Error freeing d_cell_tallies”.

2. GPU, decomposed, event, SoA
   a. Same issue as 0a

3. GPU, decomposed, event, AoS
   a. This runs successfully up to 24 tasks per node, but hangs at 48 tasks per node.

I built the multi-node GPU version of Branson with metis 5.1.0 (metis built with Spack). I am not setting any MPI or other specific env variables at the moment.

[JDTruj2018]

I am seeing similar hangs at 2 nodes, 24 tasks per node both pure CPU builds and CPU + GPU builds with use_gpu_transporter set to false. The behavior appears to be the same and reproducible on RZAdams. After a fair bit of experimentation, the CPU build is fine on a single node up to at least 96 ranks, so the issue is definitely on the fabric side for inter-node communication. I've done some preliminary testing of tweaking various environment variables (FI_CXI_DEFAULT_TX_SIZE, FI_CXI_DEFAULT_CQ_SIZE, FI_CXI_RX_MATCH_MODE) but have not yet gotten around to doing deep debugging to get to the root cause of the hangs.

Multi-node CPU Branson w/ metis complete successfully on Rocinante up to 220 ranks.

I will try out a GH200 builds (once Darwin is up) as well.