Skip to content

README.md

Latest commit

 

History

History
248 lines (183 loc) · 10.2 KB

README.md

File metadata and controls

248 lines (183 loc) · 10.2 KB
Raw

Class I* neural training (offline)

Python training pipeline for Warp 12 Class I* — Commander heuristics plus a learned residual scorer.

Architecture

  1. Collect — Commander self-play → JSONL trajectories (Node/jiti)
  2. Train — PyTorch MLP → class1-star-v1.onnx + class1-star-v1.json
  3. Bench — Class I* vs Commander win rate
  4. Ship — artifacts in apps/Warp12/public/models/; app loads via ORT (WebNN → wasm → TS)

Coach/advisor path never uses the model — only scoreWithHeuristics.

Ω neural training (standalone self-play)

Ω ("Omega") is a different animal from Class I*. Class I* is a residual bolted onto Commander and trained to imitate Commander — which is why it converged to a Commander clone (~parity, 98% top-1). Omega throws that out:

  • Standalone policy + value network. No heuristic score is added; the net is the player. Policy head scores each candidate (303-dim state+action); value head estimates the acting seat's outcome from the state alone (195-dim).
  • Self-play from scratch. The net drives every seat, sampling from its own policy for exploration. A zero-init net = uniform-random play, iteratively improved by playing itself.
  • Outcome is the only reward. Actor-critic (REINFORCE with a value baseline); the sole supervision is win/loss (+1 / −1) for the acting seat. There is no Commander in the players, the targets, or the labels — anywhere.
  • Deliberately opaque. Omega cannot explain its moves in heuristic terms. That is the trade for letting it exceed Commander instead of imitating it. The coach/advisor path is untouched and stays on the explainable heuristic model (and advisor-assisted matches are unrated anyway).

Pipeline

# One iteration: self-play → train → bench sweep (2p/3p/4p)
yarn omega:pipeline

# Iterative self-play: warm-start from the current net, collect with it, retrain
yarn omega:iterate      # repeat this to climb

# Steps individually
OMEGA_GAMES=1000 yarn omega:collect
yarn omega:train                     # or omega:train:warm to continue a net
OMEGA_BENCH_GAMES=500 yarn omega:bench

Early iterations will look worse than Commander before they surpass it — that is expected for from-scratch self-play. Gate promotion on omega:bench holding above Commander across all player counts and both objectives (the bench prints implied rating deltas per slice), not on a single 2p mode.

Variable Default Purpose
OMEGA_GAMES 200 Self-play games per collection
OMEGA_PLAYERS 2 Table size for collection
OMEGA_OBJECTIVE points points or go-out
OMEGA_TEMPERATURE 1 Self-play sampling temperature (exploration)
OMEGA_WEIGHTS apps/Warp12/public/models/omega-v1.json Net for collect/bench (missing = zero-init)
OMEGA_POLICY_HIDDEN 256,256 Policy hidden widths
OMEGA_VALUE_HIDDEN 256,128 Value hidden widths
OMEGA_EPOCHS 20 Training epochs
OMEGA_VALUE_COEF 1.0 Value-loss weight
OMEGA_ENTROPY_COEF 0.01 Policy entropy bonus (keeps exploration alive)
OMEGA_BENCH_GAMES 200 Games per bench slice
OMEGA_BENCH_PLAYERS 2,3,4 Player counts to sweep

Artifacts

File Purpose
omega-v1.json TS CPU fallback (OmegaModelWeights: policy + value heads)
omega-policy-v1.onnx Policy head (input features [N,303] → logit [N,1])
omega-value-v1.onnx Value head (input state [N,195] → value [N,1])

Quick start

# 1. Python deps (once)
yarn class1-star:setup

# 2. Deep Q pipeline (recommended — RL regret, points default)
CLASS1_STAR_GAMES=1000 CLASS1_STAR_BENCH_GAMES=500 yarn class1-star:pipeline:deepblue

# Imitation baseline (Commander self-play):
CLASS1_STAR_GAMES=1000 yarn class1-star:pipeline:deep

# Or step-by-step:
CLASS1_STAR_GAMES=1000 yarn class1-star:collect
yarn class1-star:train:deep
CLASS1_STAR_BENCH_GAMES=500 yarn class1-star:bench

Deep Q (RL regret)

When imitation hits parity (~51% win rate, 98% top-1), switch to regret-based RL:

  1. Collect RL — Class I* (current weights) plays Commander; record deviations with commanderPick
  2. Train — rl-combined loss: win → reinforce played move; loss + flipped → target Commander pick
  3. Net — 256×256 MLP, α=3.0, 40 epochs; warm-start from prior JSON when shapes match
CLASS1_STAR_GAMES=1000 CLASS1_STAR_BENCH_GAMES=500 yarn class1-star:pipeline:deepblue

# Or step-by-step:
CLASS1_STAR_GAMES=1000 yarn class1-star:collect:rl
yarn class1-star:train:deepblue
CLASS1_STAR_BENCH_GAMES=500 yarn class1-star:bench

Re-run pipeline:deepblue after each train cycle for iterative self-play (collection uses the latest weights).

Fleet Admiral (deep search)

Determinized search with belief constraints — no new net required:

# Parallel bench on M4 Max (uses all cores minus one)
yarn fleet-admiral:bench:500

# Search + Class I* hybrid
yarn fleet-admiral:bench:hybrid

# Full pass: Deep Q train + 500-game fleet bench
yarn fleet-admiral:pipeline
Env var Default Purpose
FLEET_BENCH_GAMES 100 Head-to-head games
AI_BENCH_PARALLEL 1 Shard games across worker threads
AI_BENCH_WORKERS auto Cap worker count
FLEET_BENCH_CLASS1_STAR 0 Use Class I* net on Fleet Admiral seat
FLEET_BENCH_SEAT a Fleet Admiral seat (a or b for symmetry)
ADVISOR_DEEP_THINK 0 250ms time-boxed search for tactical advisor

Go-out objective (alternate pass)

Go-out is noisier in TEI calibration; use this when comparing against prior go-out runs:

CLASS1_STAR_GAMES=1000 yarn class1-star:pipeline:go-out

Writes trajectories to tools/nn/data/trajectories-go-out.jsonl and benches with CLASS1_STAR_OBJECTIVE=go-out.

Memory note

Collection streams one game at a time to disk — you do not need to hold all rows in RAM. Node.js still caps its own heap (~4 GB by default); class1-star:collect sets --max-old-space-size=8192 because points games produce far more rows per game than go-out. Your system RAM (e.g. 64 GB) is not the limit — the Node heap is.

Environment variables

Variable Default Purpose
CLASS1_STAR_GAMES 200 Self-play games for collection
CLASS1_STAR_PROGRESS_EVERY 50 Log progress every N completed games
CLASS1_STAR_PLAYERS 2 Table size
CLASS1_STAR_OBJECTIVE points points or go-out
CLASS1_STAR_OUT tools/nn/data/trajectories.jsonl Output path (default collect file)
CLASS1_STAR_ALL_CANDIDATES 1 Export every legal move per decision (required for ranking)
CLASS1_STAR_BENCH_GAMES 100 Head-to-head games
CLASS1_STAR_LOSS combined combined, rl-combined, ranking, hinge, mse
CLASS1_STAR_WEIGHT_SCHEME outcome outcome, imitation, win-only, uniform (not used by rl-combined)
CLASS1_STAR_ALPHA 1.0 Residual scale (export + combined loss)
CLASS1_STAR_HIDDEN 128,128 Comma-separated hidden widths (Deep Q: 256,256)
CLASS1_STAR_EPOCHS 20 Training epochs
CLASS1_STAR_HINGE_MARGIN 1.0 Pairwise margin for --loss hinge

Training modes

--loss What it learns Use when
combined (default) Softmax on heuristic + α·residual Imitation baseline
rl-combined Same softmax, regret targets from RL data Deep Q — beat Commander
ranking Residual-only softmax over candidates Ablation / imitation baseline
hinge Pairwise margin: chosen beats each alternative Stronger separation signal
mse Predict +1/−1 game outcome Legacy — weak for move ranking
--weight-scheme Effect
outcome (default) 4× weight on winning-captain decisions, 0.25× on losses
imitation 2× win / 1× loss (Commander imitation)
win-only Train only on decisions from the game winner
uniform Equal weight per decision

Why combined + outcome weighting

Earlier runs used ranking loss on the residual alone and imitation weights (2× win). That taught the net to mimic Commander picks (~74% train top-1) but bench win rate stayed ~50% in 2p go-out.

Combined loss optimizes the same objective as play: argmax(heuristic + α·residual).

Outcome weighting up-weights decisions from winning captains so the net learns from outcomes, not all Commander lines equally.

Alpha sweep

Try several α values after training — export is cheap:

CLASS1_STAR_ALPHA=0.5 yarn class1-star:train:deep
CLASS1_STAR_BENCH_GAMES=500 yarn class1-star:bench

CLASS1_STAR_ALPHA=2.0 yarn class1-star:train:deep
CLASS1_STAR_BENCH_GAMES=500 yarn class1-star:bench

Or hand-edit alpha in class1-star-v1.json and re-bench without retraining.

Trajectory schema

Each JSONL row includes:

Field Purpose
features 303-dim state + action vector
heuristicScore Commander heuristic score (for combined loss)
label +1 / −1 game outcome for acting captain
chosen Whether this row is the played move
decisionId Groups candidates for ranking
commanderPick RL mode: candidate matches Commander heuristic pick
actor RL mode: class1Star for exported decisions
gameIndex Self-play game index

Re-collect after upgrading the collector if heuristicScore is missing (combined loss falls back to 0).

Artifacts

File Purpose
class1-star-v1.onnx ONNX Runtime Web (input features [N,303], output residual [N,1])
class1-star-v1.json TS CPU fallback (Class1StarModelWeights)
Feature dim 303 — CLASS1_STAR_FEATURE_DIM in warp12-engine

App prerequisites

yarn sync:ort-wasm   # copy ORT wasm to apps/Warp12/public/ort/

Until trained weights exist, Class I* uses a zero residual (Commander-equivalent).

Calibration log

After class1-star:bench, paste results into docs/calibration-log.md before shipping Class I* broadly.

Record: games collected, loss/weight/alpha, train top-1, decision flip rate, head-to-head win rate, objective, player count.