⚔️ CS 5220 SP2026: Scaling Parallel Stratego Training

Cornell University - CS 5220 SP 2026
Authors - Abrar Amin & Dalton Luce

The classic strategy game Stratego, in your terminal.

╭────────────────────────╮
│    0 1 2 3 4 5 6 7 8 9 │
│ 0  ? ? ? ? ? ? ? ? ? ? │
│ 1  ? ? ? ? ? ? ? ? ? ? │
│ 2  ? ? ? ? ? ? ? ? ? ? │
│ 3  ? ? ? ? ? ? ? ? ? ? │
│ 4  · · ≋ ≋ · · ≋ ≋ · · │
│ 5  · · ≋ ≋ · · ≋ ≋ · · │
│ 6  B 3 2 2 5 2 6 7 5 B │
│ 7  3 8 2 6 6 B 4 6 4 ★ │
│ 8  8 9 B 2 2 3 2 5 7 ⚑ │
│ 9  3 S B 3 7 5 2 4 B 4 │
╰────────────────────────╯

Overview

This project explores the scalability and stability tradeoffs of parallelized Proximal Policy Optimization (PPO) via self-play in the game of Stratego. By distributing environment simulation and training across multiple workers, we aim to maximize experience generation throughput.

To manage computational complexity, training begins on Stratego Tiny (a reduced state/action space variant) and will progressively scale toward the full board state.

We use the modern Stratego numbering system, where larger numbers indicate higher piece value. We do not use the Spotter piece for simplicity. We provide three game types:

• Classic: 10x10 grid, 40 pieces
• Barrage: 10x10 grid, 8 pieces
• Quick: 8x8 grid, 10 pieces
• Tiny: 6x6 grid, 6 pieces

When playing Classic, the starting setup is sampled from a Constrained Weighted Distribution based on the Dobby/Oewesok marginals (see data). The placement logic follows an iterative masking approach: pieces are sampled sequentially, starting with the Flag and Bombs, with the probability map re-normalized after each tile is occupied. This ensures valid, non-overlapping configurations that mirror human expert heuristics while maintaining high entropy for RL training. In practice, these setups are not ideal, as sampling from a distribution for each piece individually doesn't consider the synergy of piece clusters, such as the critical proximity of the Spy to the Marshal or the mutual protection of Bomb-Flag formations. However, it is much better than complete randomness.

Building

Local (CPU, no GPU)

make configure   # override torch path if needed: make configure TORCH_CMAKE=<path>
make build

macOS (Apple Clang): OpenMP is not bundled with Apple's compiler. Install the runtime via Homebrew and expose it to CMake before configuring:

brew install libomp
OpenMP_ROOT=$(brew --prefix libomp) cmake ..

Perlmutter (NERSC)

module modifies your shell environment so it must be run manually before building:

module load PrgEnv-gnu
module load cudatoolkit

Then configure and build using the Perlmutter profile:

make configure_perlmutter
make build

configure_perlmutter sets the NERSC PyTorch and NCCL paths, links the Cray GTL library (libmpi_gtl_cuda) needed for GPU-aware MPI on A100 nodes, and enables BUILD_ITT_STUB (see perlmutter/ittnotify_stub.cpp for why that's needed).

The GTL flags (PE_MPICH_GTL_DIR_nvidia80 / PE_MPICH_GTL_LIBS_nvidia80) are read from the environment automatically by the loaded cray-mpich module, so no version is hardcoded.

Run Tests

./build/test

Run Terminal Game

./ui/play

Training

./rl/train_stratego <game_type> <setup_type> <num_episodes>

Arena

The arena runs head-to-head matches between two agents. Games are embarrassingly parallel — each is fully independent — so the arena uses OpenMP to simulate many games concurrently.

./build/arena --red <spec> --blue <spec> --games <N> [--threads <T>]

Flag	Default	Description
--red	random	Red agent (random, neural:<path>, ucc:<path>)
--blue	random	Blue agent (same specs)
--games	1	Number of games to simulate
--threads	all cores	Number of OpenMP threads

The binary prints whether OpenMP is active and how many threads are in use at startup, so there is no silent fallback to serial execution. The final summary includes total wall-clock time and throughput (games/s).

Example:

./build/arena --red neural:checkpoints/red.pt --blue random --games 1000 --threads 8

Resources

• CS 5782 SP2026: helpful RL background resources

Training on Perlmutter

Training uses MPI + NCCL for distributed PPO. Each MPI rank owns one GPU and runs independent self-play rollouts; weights are averaged across ranks via NCCL AllReduce after each batch.

Single node (4 GPUs)

sbatch run_perlmutter.sh [variant] [setup] [total_episodes] [batch_size]

Defaults: tiny random 20000 256.

Multi-node

Pass --nodes and --ntasks (nodes × 4) at submission time:

# 2 nodes, 8 GPUs
sbatch --nodes=2 --ntasks=8 run_perlmutter.sh tiny random 20000 256

total_episodes is divided evenly across ranks, so to keep the same per-rank workload when scaling out, multiply episodes by the number of ranks:

# same per-rank work on 8 GPUs as 20000 episodes on 1 GPU
sbatch --nodes=2 --ntasks=8 run_perlmutter.sh tiny random 160000 256

Verifying it works

Once the job starts, tail the log:

tail -f logs/train_<JOBID>.out

You should see:

[NCCL+MPI enabled] MPI world_size=8
Ranks: 8 | Per-Rank Batch: 256 | Effective Batch: 2048

If world_size is 1, ranks are not communicating — check that --mpi=pmi2 is used in the srun call (not cray_shasta) and that the binary was built with the GTL library.

Key implementation notes

• PMI launcher: srun --mpi=pmi2 is required. --mpi=cray_shasta triggers an OFI fabric init failure (fi_getinfo() No data available) with this binary.
• GTL library: libmpi_gtl_cuda must be linked at build time (done via PE_MPICH_GTL_DIR_nvidia80 / PE_MPICH_GTL_LIBS_nvidia80 in configure_perlmutter). Do not set MPICH_GPU_SUPPORT_ENABLED=0 — that disables the GTL and breaks OFI init.
• Multi-node device mapping: each rank calls cudaSetDevice(rank % devices_per_node) so ranks on node 1 map to local GPUs 0–3 rather than invalid device indices 4–7.

Architecture & Tech Stack

This project is built for high-performance execution on the Perlmutter supercomputer. To minimize overhead, the entire pipeline is a pure C++ implementation.

• Algorithm: Proximal Policy Optimization (PPO) (Actor-Critic)
• Environment: Custom C++ Stratego grid simulation
• Distributed Communication: NCCL (NVIDIA Collective Communications Library)
• Target Hardware: Perlmutter (Multi-GPU node scaling)

Core Interfaces

As opposed to directly following established RL frameworks like OpenSpiel, we opted for a minimal set of simple abstractions tailored to our needs. The system is decoupled into three primary C++ interfaces to allow for rapid iteration and testing:

1. Environment: A Gymnasium-style interface handling reset, step, and game logic (imperfect information masking, combat resolution, 500-step truncation).
2. Agent: A dual-headed Actor-Critic model. It outputs actions/log-probs for self-play rollouts and evaluates value estimations for the PPO update.
3. RolloutBuffer: An on-policy data store that collects step trajectories and computes Generalized Advantage Estimation (GAE) before flushing.

Training Pipeline (Self-Play)

1. Parallel Rollouts: Multiple workers independently simulate games against a uniformly sampled pool of past checkpoints.
2. Synchronization: Gradients and model parameters are synchronized across GPUs using NCCL. We are evaluating both synchronous (stable but bottlenecked) and asynchronous (higher throughput but noisier) update strategies.
3. Optimization: The PPO clipped surrogate loss is calculated, and weights are updated via mini-batches.

Random Notes and Observations

• Stratego logic is very conditional (e.g., if–then–else rules, piece rank comparisons), making it very hard to model or run quickly on a GPU
• Encoding board state is important

Board Encoding

We have found that devising a smart encoding of the board state is critical to obtain good AI performance. Our scheme follows the standard approach used in modern deep reinforcement learning systems for board games, where the state is encoded as a stack of spatial feature planes separating piece types, player ownership, and hidden information. More specifically, our board state is represented as a C x H x W tensor, where C is the number of channels (see below), and H x W are the height and width of the game board, respectively. Each channel contains different information:

• For each piece ranking (Scout, Miner, etc.), encode a one hot channel for all piece locations. -For each piece ranking (Flag, Lieutenant, Captain, Major), encode a one-hot channel for the current player's pieces
• Also, for each piece ranking, encode a one-hot channel for the opponent's pieces that have been revealed during combat
• Encode one channel for all opponent pieces whose identities are not yet revealed
• Encode one channel containing the lake (impassable tiles)

Action Encoding

We use a flattened discrete action space where each possible move is represented as a single integer. This avoids structured outputs like (x, y, dx, dy) and allows the policy network to output a single categorical distribution over all actions. This is also standard in modern RL.

Each action encodes:

• a start position (x, y)
• a direction in the set {Up, Down, Left, Right}
• a movement distance in the range [1, max_dist], where max_dist = max(H, W) - 1

We flatten this into a single index:

$$ \text{action} = ((y \cdot W + x) \cdot 4 \cdot D) + (\text{direction} \cdot D) + (\text{distance} - 1) $$

where

• $W$ = board width
• $H$ = board height
• $D$ = max distance
• $\text{dir} \in {0,1,2,3}$

This gives a total action space size:

$$ |A| = H \cdot W \cdot 4 \cdot D $$

This gives us a fixed-size action space directly compatible with softmax policy heads. for a classic game of Stratego (H = 10, W = 10, D = 10), our action space has a size of 4000. This mapping is fully reversible so we can decode the action index back into an actual move on the game board. Illegal actions are handled via masking or penalization during environment stepping.