Split paper LaTeX + research-ops docs out into private repo#34
Closed
amcheste-ai-agent[bot] wants to merge 45 commits into
Closed
Split paper LaTeX + research-ops docs out into private repo#34amcheste-ai-agent[bot] wants to merge 45 commits into
amcheste-ai-agent[bot] wants to merge 45 commits into
Conversation
* refactor agent into modules * code review comments
* improve training reward system * code review comments
* Create Environment.Rewards unit test
* Create Environment.Rewards unit test
* Add paper infrastructure scaffolding (PR #0)
Sets up the research artifacts and tooling needed for paper-grade
experiments. No behavioral changes to training or environment code.
paper/
- analysis_plan.md: pre-registered hypotheses, primary metric (Brock
win-rate at 100M env-steps), secondary metrics, treatment design,
statistical analysis plan (rliable IQM + 95% bootstrap CI), stopping
rules, exclusions, and full compute budget commitment (~4.2B
env-steps total).
- compute_ledger.md: append-only template for logging significant
training runs with git SHA, seeds, env-steps, wall-clock, hardware,
and result. Required for the paper's compute transparency appendix.
- README.md: explains the workflow and reproducibility checklist.
- figures/, notebooks/: placeholders for paper figures and the Jupyter
notebooks that produce them from W&B exports.
scripts/
- eval.py: locked evaluation harness skeleton. Real implementation
follows in PR #2 once save state loading and event flag reading land.
Already enforces the locked n_episodes=20, seed=42 from the analysis
plan and refuses to deviate without explicit override.
- seed_utils.py: multi-library seeding (Python random, NumPy, PyTorch
CPU/CUDA, PYTHONHASHSEED, sb3 set_random_seed). Optional torch
deterministic mode for camera-ready runs.
- __init__.py: makes scripts/ a package so eval.py is runnable as
`python -m scripts.eval`.
save_states/: placeholder directory for PyBoy save states added in PR #2.
ROM-derived save states are local-only and never committed.
MODEL_CARD.md: Mitchell-et-al-style model card template, populated as
experiments produce checkpoints.
requirements.txt: adds rliable==1.2.0 (statistical analysis), sb3-contrib
==2.7.0 (RecurrentPPO for the LSTM policy), optuna==4.1.0 (HPO),
wandb==0.18.7 (experiment tracking). All inserted alphabetically to
match existing convention.
pyproject.toml: bumped version 0.1.0 -> 0.2.0, added description, license
declaration, requires-python>=3.10, authors, keywords, classifiers, and
project URLs. Added pytest markers for slow/integration/paper test
selection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This is the highest-leverage change in the project: giving the agent goal-directed signal toward the Boulder Badge via game event flags. New module: pokemon_red_ai/game/event_flags.py - 18 pre-registered Boulder Badge path flags (analysis_plan.md §9) - Flag reader: read_event_flag(), read_boulder_path_flags() - Total event bit counter: sum_all_event_flag_bits() - Battle state reader: is_in_battle() (wIsInBattle at 0xD057) - EventFlagTracker: detects 0→1 transitions, prevents double-counting - Per-flag reward weights (200 for EVENT_BEAT_BROCK, 5-75 for others) New class: EventProgressRewardCalculator in rewards.py - Event flag transitions (one-shot rewards on flag 0→1) - Milestone bonuses at 5/10/14/16 flags triggered - Dense exploration signal (per-tile and per-map) - Badge diff uses popcount instead of raw integer comparison - Registered as 'events' strategy in the factory function Also: added battle_type and event_flags_start to MEMORY_ADDRESSES. 63 new tests (39 event_flags + 24 event_rewards), 418 total passing. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Ensures every PR is automatically assigned to whoever opened it, so it shows up in Graphite and GitHub project boards. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary - **Event flags wired in**: `get_comprehensive_state()` now populates `event_flags` and `battle_state` keys, so `EventProgressRewardCalculator` receives live flag data during training - **Save state loading**: new `save_state_path` parameter on the env — `reset()` loads a `.state` file instead of replaying the 30-second intro sequence every episode - **SELECT removed**: action space changed from `Discrete(8)` to `Discrete(7)`, matching `analysis_plan.md` §5.2 ## What this unlocks PR #1 added the event flag reader and reward calculator, but they were disconnected — the environment never passed flag data to the calculator. This PR closes that gap. Combined with save state loading, the training loop is now fast enough for serious experiments. ## Key changes | File | Change | |---|---| | `pokemon_red_ai/game/memory.py` | `get_comprehensive_state()` includes `event_flags` + `battle_state` | | `pokemon_red_ai/game/agent.py` | New `load_save_state(path)` and `save_save_state(path)` methods | | `pokemon_red_ai/environment/gym_env.py` | `save_state_path` param, save-state-aware `reset()`, `Discrete(7)`, event progress in info dict | | `tests/unit/environment/test_gym_env.py` | Updated for 7-action space | ## Test plan - [x] 418 tests passing, 0 regressions - [x] Action space tests updated for SELECT removal - [ ] Integration test with actual ROM + save state (requires a real ROM) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary - Adds `create_recurrent_ppo_model()` in `models.py` using sb3-contrib's `RecurrentPPO` with `MultiInputLstmPolicy`, configurable LSTM layers (default 1) and hidden size (default 256) - Updates `PokemonTrainer.create_model()` and `test()` to support `algorithm='RecurrentPPO'`, including proper LSTM state carry-over during evaluation episodes - Adds RecurrentPPO config to `get_model_config()` with a slightly lower default learning rate (2.5e-4 vs 3e-4) for LSTM stability - 17 new unit tests covering model creation, LSTM parameter passthrough, factory routing, training workflow, and evaluation with state threading ## Why RecurrentPPO? Pokemon Red is partially observable — a single frame can't distinguish between identical-looking menus, mid-battle states, or dialogue screens. The LSTM gives the agent short-term memory so it can track what happened in recent frames and make contextual decisions. This is our primary algorithm for the research paper experiments. ## Test plan - [x] All 109 training unit tests pass (was 88, now 109 with 17 new + 4 updated) - [x] Full unit suite passes (449 tests, 0 failures) - [x] RecurrentPPO model creation uses `MultiInputLstmPolicy` with `PokemonFeaturesExtractor` - [x] LSTM state threading verified in evaluation loop - [x] Backward compatible — existing PPO paths unchanged - [x] `conftest.py` mock action space updated to Discrete(7) to match PR #14 🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- **`scripts/train.py`** — standalone training script with argparse CLI, global seeding, W&B experiment tracking, save-state episode resets, and graceful Ctrl+C handling. Default config: RecurrentPPO + events reward strategy + multi-modal observations + 1M timesteps.
- **`WandbCallback`** in `callbacks.py` — SB3 callback that logs `episode/reward_mean`, `game/maps_visited`, `game/badges_max`, `game/event_flags_max`, and more to W&B on each rollout end. Uploads model checkpoint artifacts at `save_freq` intervals. Does not own the W&B run lifecycle (caller handles `wandb.init`/`finish`).
- **CLI updates** — `pokemon-ai train` now supports `--algorithm RecurrentPPO`, `--reward-strategy events`, `--save-state`, `--wandb-project`, `--no-wandb`, and `--seed`.
- **35 new tests** — WandbCallback (metric logging, best-model tracking, log_freq throttling, artifact upload, edge cases) and train script (arg parsing, seeding determinism).
## How to use
```bash
# Minimal (W&B will prompt for login on first run)
python scripts/train.py --rom path/to/PokemonRed.gb -v
# Full research run
python scripts/train.py \
--rom path/to/PokemonRed.gb \
--save-state states/post_intro.state \
--algorithm RecurrentPPO \
--reward-strategy events \
--total-timesteps 1_000_000 \
--seed 42 \
--wandb-project pokemon-red-ai \
-vv
# Without W&B
python scripts/train.py --rom path/to/PokemonRed.gb --no-wandb
```
## Test plan
- [x] 144 training unit tests pass (was 109, now 144 with 35 new)
- [x] Full unit suite passes (484 tests, 0 failures)
- [x] WandbCallback logs correct metric keys and values
- [x] WandbCallback gracefully handles empty ep_info, partial keys, artifact upload failures
- [x] Train script argument parser validates choices and defaults
- [x] Global seeding produces deterministic numpy/torch outputs
- [x] Backward compatible — existing PPO/exploration paths unchanged
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary - Adds `scripts/create_save_states.py` — automates Pokemon Red's opening sequence and saves PyBoy `.state` files for fast episode resets during RL training - The post-intro save state (`s0_post_intro.state`) eliminates the ~10s intro replay on every episode reset, critical for training throughput - Supports 3 modes: **automatic** (creates post-intro state), **interactive** (play and Ctrl+C to save), and **validate-only** (verify existing states) - Defines validation specs for 4 curriculum checkpoints: post-intro, post-starter, pre-Brock, post-Brock - Adds `*.state` to `.gitignore` (binary files containing ROM data) - 28 new unit tests (mocked PyBoy), 512 total passing ## Test plan - [x] `python scripts/create_save_states.py --rom PokemonRed.gb -vv` — creates and validates `s0_post_intro.state` (167KB, 1.5s) - [x] `--validate-only` mode confirms existing state passes all checks - [x] 28 unit tests cover parser, spec definitions, validation logic (map/badge/party checks), creation flow, error handling, and cleanup - [x] Full test suite: 512 passed, 0 failures 🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary - Replaces the `scripts/eval.py` skeleton with a working evaluation pipeline for paper experiments - Runs N deterministic episodes on a trained checkpoint, collecting per-episode metrics: event flags, maps visited, badges, steps-to-Pewter, Boulder Badge win rate - LSTM state threading for RecurrentPPO evaluation (same pattern as trainer.py) - Auto-detects PPO vs RecurrentPPO from the checkpoint file - Outputs JSON matching `EVAL_METRIC_SCHEMA` for direct consumption by rliable analysis notebooks - Locked defaults (20 episodes, seed 42) enforce pre-registered experimental protocol; `--allow-override` for dev/debugging - Environment cleanup guaranteed via `try/finally` ## Test plan - [x] 29 new unit tests covering: CLI parsing, locked value enforcement (rejects deviations without `--allow-override`), file validation, episode rollout (basic, recurrent LSTM states, max-step truncation, Pewter/Brock detection), metric aggregation (mean return, Brock win rate, steps-to-Pewter averaging), env cleanup on success/error, JSON serialization - [x] Full test suite: 541 passed, 0 failures 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…61) (#19) ## Summary - **Pixel** (Treatment 1): 80×72×1 grayscale screen processed by SB3's NatureCNN. Maps to `CnnPolicy`/`CnnLstmPolicy`. - **Symbolic** (Treatment 2): 29-element `float32` vector — position(3) + stats(6) + 18 event flags + exploration(2), all normalised to [0,1]. Maps to `MlpPolicy`/`MlpLstmPolicy`. - **Hybrid** (Treatment 3): `Dict` space combining grayscale screen + symbolic vector, auto-routed by SB3's `CombinedExtractor`. Maps to `MultiInputPolicy`/`MultiInputLstmPolicy`. - New `get_policy_type_for_observation(obs_type, algorithm)` function for automatic policy string selection across all 6 observation types × 2 algorithms. - CLI choices extended in `commands.py`, `scripts/train.py`, and `scripts/eval.py`. - 37 new tests (598 total, all passing). ## Changed files | File | What | |------|------| | `observations.py` | `create_pixel/symbolic/hybrid_observation_space()`, `process_*()`, `_build_symbolic_vector()`, `_screen_to_grayscale()` | | `gym_env.py` | Observation space creation, `_get_observation()`, `_get_default_observation()` branches | | `models.py` | `get_policy_kwargs_for_observation_type()` extended, new `get_policy_type_for_observation()` | | `__init__.py` (×2) | Exports for new public API | | `commands.py`, `train.py`, `eval.py` | `--observation-type` choices extended | | `test_observations.py` | 27 new tests: space shapes, values, normalisation, event flags, graceful failure | | `test_models.py` | 10 new tests: policy kwargs, exhaustive policy-type mapping | | `conftest.py` | Parameterized `observation_type` fixture updated | ## Test plan - [x] All 598 unit tests pass (`pytest tests/ -v`) - [ ] Smoke test: `python scripts/train.py --rom PokemonRed.gb --observation-type pixel --total-timesteps 1000 --no-wandb` - [ ] Smoke test: `python scripts/train.py --rom PokemonRed.gb --observation-type symbolic --total-timesteps 1000 --no-wandb` - [ ] Smoke test: `python scripts/train.py --rom PokemonRed.gb --observation-type hybrid --total-timesteps 1000 --no-wandb` Closes AMC-59, AMC-60, AMC-61. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…-62) (#20) ## Summary - **Bug fix**: `create_ppo_model()`, `create_recurrent_ppo_model()`, and `create_a2c_model()` were hardcoding `PokemonFeaturesExtractor` and `MultiInputPolicy`, which only works for the legacy `multi_modal` observation space. Now they accept `observation_type` and auto-select the correct SB3 policy via `get_policy_type_for_observation()`. - **100k step validation run completed**: RecurrentPPO + event-flag rewards + hybrid obs, seed 42, 19 min on CPU. Reward climbed 718 → 878 across 7 episodes. Best model auto-saved at 877.50. - **Full pipeline validated**: train → checkpoint save → eval.py load → deterministic eval episodes → JSON output → checkpoint resume. All 6 AMC-62 checklist items pass. ## AMC-62 validation checklist - [x] RecurrentPPO + event-flag rewards + hybrid obs trains for 100k steps without crashes - [x] Metrics logged (tensorboard + stdout: reward, loss, entropy, clip_fraction, etc.) - [x] Model checkpoint saved (`best_model.zip` + `final_model.zip`) - [x] Best model auto-saved on reward improvement (triggered at reward 877.50) - [x] `eval.py` loads checkpoint and runs deterministic evaluation episodes - [x] Training can resume from a checkpoint ## Smoke tests passed for all 3 observation types | Type | Policy | Params | Status | |------|--------|--------|--------| | pixel | CnnLstmPolicy | 2.8M | ✅ | | symbolic | MlpLstmPolicy | 786K | ✅ | | hybrid | MultiInputLstmPolicy | 1.9M | ✅ | ## Test plan - [x] 598 unit tests pass - [x] 100k step training run completes without crashes - [x] eval.py produces valid JSON metrics from trained checkpoint - [x] Checkpoint resume works (load + continue training) - [ ] Try it yourself: `./venv/bin/python3 scripts/train.py --rom PokemonRed.gb --save-state save_states/s0_post_intro.state --observation-type hybrid --total-timesteps 50000 --no-wandb --show-game --seed 42 -v` **This PR completes M1 (First Training Run) and unblocks M2 pilot runs (AMC-63/64/65).** Closes AMC-62. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary - Add `MonitoringCallback` extending `WandbCallback` with four game-specific views: - **Map exploration heatmap** — cumulative map visit counts logged to W&B as both a Table and a matplotlib bar-chart image - **Event flag checklist** — all 18 pre-registered Boulder-Path flags from `pokemon_red_ai/game/event_flags.py`, including trigger count and first-triggered episode - **Periodic screen captures** — game screen logged as a W&B image every N episodes (configurable via `--screen-capture-freq`, default 10) - **Per-episode breakdown** — reward, length, maps_visited, final_map, badges, event_flags_triggered logged as a W&B Table row per episode - Mirror all monitoring state to `<save_dir>/dashboard_state.json` on every rollout so local tools can read without a W&B account. - Add `scripts/monitor.py` — a Streamlit dashboard that: - Auto-discovers runs under `--runs-dir` - Plots reward curves (with moving average) from `monitor.csv` - Shows map exploration grid + event flag checklist from `dashboard_state.json` - Supports side-by-side run comparison (overlay seeds) - Optionally overlays `rollout/ep_rew_mean` from TensorBoard if available - Wire `MonitoringCallback` into `scripts/train.py` (replaces raw `WandbCallback` when W&B is enabled). - Pass `info_keywords=MONITORED_INFO_KEYS` to the SB3 `Monitor` wrapper so custom per-step metrics (maps_visited, event_progress, badges_earned, …) propagate into `ep_info_buffer` — this was an existing gap that made a couple of keys in `WandbCallback` silently no-op at runtime. - Add Streamlit as an optional dep (`pip install -e ".[dashboard]"`). Answers the research question *"is this run worth continuing?"* by putting rising reward, new maps, and ticking event flags on one screen. ## Test plan - [x] `tests/unit/training/test_monitoring_callback.py` — 14 tests covering per-episode recording, screen-capture schedule (including disable-at-0 path), map heatmap, flag progress (all 18 flags), episode breakdown, dashboard JSON snapshot (including write-failure tolerance) - [x] `tests/unit/test_monitor_script.py` — 19 tests covering run discovery, `dashboard_state.json` + `monitor.csv` loaders, all DataFrame builders, run summary, CLI parser - [x] Full suite still green: **631 passed** locally - [ ] Launch `streamlit run scripts/monitor.py -- --runs-dir ./training_output` against a live run once reviewer merges 🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary - Adds `scripts/analyze.py` — publication-quality statistical analysis using the [rliable](https://github.com/google-research/rliable) library (Agarwal et al. 2021) - Implements the full pre-registered analysis protocol from `paper/analysis_plan.md` §6: - **IQM** (Interquartile Mean) — not raw mean — as the primary aggregate metric - **95% stratified bootstrap CIs** with 2,000 resamples (configurable via `--reps`) - **Probability of Improvement** > 0.75 with CI excluding 0.5 as the significance threshold - **Performance profiles** showing fraction of runs above each score threshold - **Sample efficiency curves** with per-checkpoint IQM and bootstrap CIs - Generates 4 publication-quality figures saved to `paper/figures/`: - `aggregate_iqm.pdf` — bar chart with error bars - `performance_profiles.pdf` — CDFs per treatment - `probability_of_improvement.pdf` — pairwise heatmap with significance markers - `sample_efficiency.pdf` — learning curves across training - Supports two input formats: - **JSON directory**: treatment subdirectories with `eval.py` output JSONs - **CSV**: flat file with treatment/seed/metric columns - 49 tests covering data loading, score matrix construction, all rliable computations, plot generation, CLI parsing ### Usage ```bash # From eval JSON directory (output of scripts/eval.py) python scripts/analyze.py --results-dir eval_results/ -v # From CSV python scripts/analyze.py --csv results.csv --output-dir paper/figures/ # Specific plots only, custom bootstrap count python scripts/analyze.py --results-dir eval_results/ --plots iqm poi --reps 500 ``` ## Test plan - [x] All 49 tests pass (`pytest tests/unit/test_analyze.py -v`) - [x] End-to-end smoke test with synthetic 3-treatment × 5-seed data produces all 3 figures - [x] No deprecation warnings - [x] `rliable==1.2.0` already in `requirements.txt` 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…#23) ## Summary - Enhances `MonitoringCallback` with three capabilities the observability stack was missing: - **Reward component breakdowns**: Accumulates `reward_components` on every env step (not just episode end) and logs `reward/exploration_mean`, `reward/badge_mean`, `reward/time_mean`, etc. as W&B scalars. Researchers can now see which reward signals the agent is exploiting over training. - **Game-state level/money/party curves**: Records `player_level`, `pokemon_count`, and `money` at episode boundaries. Logs `game/player_level_mean`, `game/player_level_max`, `game/player_level_latest`, plus pokemon count and money curves. - **W&B `define_metric()` organization**: Groups panels under `episode/*`, `game/*`, and `reward/*` with `global_step` as the x-axis so the W&B dashboard auto-creates structured panel groups. - Extends the episode table with 3 new columns: `player_level`, `pokemon_count`, `money` - Extends `dashboard_state.json` with `level_history`, `money_history`, `pokemon_count_history`, and `reward_component_summary` - Adds `pokemon_count` and `money` to `MONITORED_INFO_KEYS` - Multi-env safe: reward accumulators are keyed by env index and reset per-episode ## W&B Dashboard After This PR When you run training, the W&B dashboard will auto-organize into: - **episode/** — reward_mean, reward_max, length_mean, best_reward - **game/** — maps_visited, badges, event_flags, player_level, pokemon_count, money, screen captures - **reward/** — per-component breakdown (exploration, badge, time, death, event_flags, etc.) ## Test plan - [x] 16 new tests added (30 total in `test_monitoring_callback.py`) - [x] All 696 project tests pass - [x] Verified reward accumulation across steps, accumulator reset between episodes, multi-env isolation - [x] Verified graceful handling of missing fields and old wandb versions 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…24) ## Summary - Add **game progress tabs** (player level, party size, money) as line charts over episodes - Add **reward component breakdown** with summary bar chart + per-episode area chart of top 6 components - Add **auto-refresh** with sidebar toggle and configurable interval (default 30s) - Enhance headline metrics with max level and max party count - Fix padding bug in `level_curve_df` when pokemon/money histories are shorter than level history - Add 19 new tests covering `level_curve_df`, `reward_breakdown_df`, `reward_summary_df`, `--refresh-interval` arg, and enhanced `run_summary` All 715 project tests pass. `streamlit` was already in `requirements.txt`. ## Test plan - [x] All 38 monitor script tests pass (19 existing + 19 new) - [x] Full suite: 715 passed, 0 failed - [ ] Manual: `streamlit run scripts/monitor.py -- --runs-dir ./training_output` during a live run 🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary - New `pokemon_red_ai.training.alerts` module: `TrainingAlertCallback` (SB3 callback) + 4 pluggable channels (`LogChannel`, `DesktopChannel`, `SlackChannel`, `EmailChannel`). - Triggers: **first badge**, **new max badge** (1, 2, 3, ...), **new map discovered** (skipping the spawn map), **new event flag**, **reward plateau** (configurable warmup + threshold), **checkpoint at every N timesteps**, and a **crash** helper (`notify_crash()`) for the training script's `except` block. - Per-trigger-type cooldown to prevent spam. YAML/JSON config loader (`load_alert_config`, `channels_from_config`) so alert rules are sharable across runs. - Wired into `scripts/train.py`: new flags `--enable-alerts`, `--alerts-config`, `--alerts-desktop`, `--alerts-slack-webhook`, `--alerts-plateau-episodes`, `--alerts-checkpoint-freq`. The training `try/except` block now calls `notify_crash()` on unhandled exceptions. - Sample config at `configs/alerts.example.yaml`. - 61 new unit tests covering: every channel (with mocked subprocess/HTTP/SMTP), config loader, every trigger, cooldown semantics, multi-channel dispatch, channel-failure isolation, crash notification, robustness to empty info dicts. All channel sends are best-effort — failures are caught and logged at WARN, so a flaky Slack webhook never crashes a 10M-step training run. Closes the second-to-last issue in the **Training Observability** milestone (AMC-79 remains). ## Test plan - [x] All 61 new alert tests pass - [x] Full project suite: 776 passed, 0 failed - [x] `python scripts/train.py --help` shows new flags correctly - [x] `configs/alerts.example.yaml` parses cleanly via `load_alert_config` - [ ] Manual: kick off a short training run with `--enable-alerts --alerts-desktop` and watch for the first-badge / checkpoint pings 🤖 Generated with [Claude Code](https://claude.com/claude-code)
A self-contained walkthrough covering: merging PR #26, pre-pilot prep (W&B login, alerts config, smoke test, compute decision), running the 9 pilots (commands, monitoring, when to kill early), analyzing results (compare.py + analyze.py), drafting the EWRL paper, and parallel research-ops tasks (Oracle IP, NCSA grant, NC State advisor). Includes a timeline arithmetic section showing ~14 days of slack against the 2026-05-25 EWRL deadline if pilots take 10 days, and a quick-reference command cheat sheet at the end. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Summary - New `pokemon_red_ai.analysis` package with reusable, testable comparison logic — separable from the Streamlit UI so it's importable from any script (paper figure generation, notebooks, etc.). - **Treatment detection** from run names: matches `pixel`, `symbolic`, `hybrid`, `multi_modal`, `screen_only`, `minimal` via two-pass regex (multi-word treatments first via boundary-anchored match, then single-word via token split). Falls back to `unknown`. - **Learning curves with bands**: per-treatment mean ± std across seeds, with optional rolling smoothing and length alignment to the shortest seed in the group. - **IQM + bootstrap CIs**: per-seed best-reward distribution per treatment, with percentile bootstrap CIs (pure-numpy implementation, no rliable runtime dependency at call time but mathematically equivalent). - **Milestone race chart**: per-treatment median (or min/mean) of `first_episode` for each event flag — answers "which treatment hits BEAT_BROCK first?". - **Final-performance bars**: mean ± std of last-N-episode reward per treatment. - **Publication-quality figure export**: matplotlib serif font, 300 DPI, PDF/SVG/PNG via `export_figure()`. Style matches `scripts/analyze.py` so live and paper figures are visually consistent. - New `scripts/compare.py` Streamlit app wires it all together: multi-run sidebar selector, per-run treatment override (for badly-named runs), learning-curve overlay, IQM table, final-performance bars, milestone race table, plus PDF/SVG/PNG download buttons for each figure. - `configs/wandb_report_template.md` — manual W&B Report panel spec mirroring the Streamlit layout. (W&B Reports SDK is unstable across versions; markdown spec is more durable.) - 57 new unit tests covering: 8 treatment-name patterns + 4 unknown fallbacks, grouping with deterministic ordering, learning curve construction (single/multi seed, alignment, smoothing, truncation), IQM math (tail-dropping, edge cases, bootstrap CI brackets), summary table (custom metrics, single-seed std=0), milestone race (median/min/mean aggregators, NaN for never-triggered, whitelist), final-performance bars, and PDF/SVG/PNG export including unsupported-extension errors and parent-dir creation. **Closes the Training Observability milestone** alongside AMC-76 / 77 / 78. Streamlit + rliable + alerting + comparison — the full live-research toolkit, ready for the EWRL pilots. ## Test plan - [x] All 57 new tests pass - [x] Full project suite: 833 passed, 0 failed - [x] `python scripts/compare.py --help` parses correctly - [x] `from pokemon_red_ai.analysis import detect_treatment` works - [ ] Manual: `streamlit run scripts/compare.py -- --runs-dir ./training_output` once 3+ pilot runs exist; verify learning curves render with bands, IQM table populates, PDF download produces a valid file 🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary A self-contained walkthrough at `docs/research_playbook.md` covering everything between right-now and the 2026-05-25 EWRL deadline: - **Step 0:** Merge PR #26 (closes Training Observability) - **Step 1:** Pre-pilot prep — W&B login, alerts config, compute decision (local vs RunPod ~$110 vs Lambda ~$135), 5-min smoke test - **Step 2:** Run the 9 pilots — exact commands, suggested order (pixel → symbolic → hybrid), `tmux` tip, "when to kill a run early" rules, per-run checklist template - **Step 3:** Analyze with `scripts/compare.py` + `scripts/analyze.py` - **Step 4:** Draft the EWRL paper (Claude does most; me on review/polish) - **Step 5:** Parallel research ops — Oracle IP review (must precede arXiv preprint), NCSA ACCESS grant, NC State advisor outreach Includes a timeline arithmetic section showing ~14 days of slack against the deadline if pilots take 10 days, plus a quick-reference command cheat sheet at the end. ## Test plan - [x] `cat docs/research_playbook.md` renders cleanly - [ ] Walking through Step 1 commands once — verify smoke test launches and Streamlit dashboard updates 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Drafts the four sections that don't depend on pilot results
(Introduction, Related Work, Environment, Methods) and provides
results/discussion skeletons with explicit todopilot{} markers
flagging every spot a pilot number, figure, or interpretation needs
to land.
Builds cleanly to a 12-page PDF on TeX Live 2026 Basic with
plain pdflatex + bibtex (no latexmk dependency). Citations cover
PPO, RecurrentPPO/DRQN, rliable, ALE/Atari, Procgen/Craftax/MiniHack,
Pokemon-RL prior art, CKA, SB3, and Gymnasium.
Section content is template-agnostic so the documentclass can be
swapped to the EWRL 2026 venue template once published.
paper/SCAFFOLDING.md documents how to grep for todopilot{} markers
after pilot data lands and how to slot the analyze.py-generated
figures into the Results section.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wraps scripts/train.py with consistent save-dir / W&B-run-name paths, caffeinate on macOS so the system doesn't sleep mid-run, per-run log redirection, optional batched parallelism (--parallel N), and skip-completed detection so re-runs after a crash don't redo finished work. Default invocation runs the canonical 9-pilot grid (3 treatments × 3 seeds × 10M steps each). All defaults are overridable via flags; extra train.py args can be forwarded after `--`. Provides --dry-run for previewing commands and --help with worked examples. Validated end-to-end via dry-run with skip-completed, subset selection (--seeds, --treatments), extra-arg forwarding, and error paths (missing/invalid ROM). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
A bash launcher that runs the canonical 9-pilot grid (3 treatments × 3 seeds × 10M steps) with one command instead of nine. Wraps `scripts/train.py` with consistent paths, `caffeinate -i` on macOS, per-run log redirection, batched parallelism, and skip-completed detection.
## Key features
- **`--rom PATH`** required, everything else has sensible defaults
- **`--parallel N`** runs N pilots concurrently (waits for each batch before launching the next)
- **`--dry-run`** prints every command without executing — used to verify the grid before committing compute
- **Skip-completed by default** — pilots with `final_model.zip` are skipped so re-runs after a crash only retry failures
- **`--seeds`, `--treatments`** for subsets (e.g., `--seeds 42` for a single-seed sweep)
- **`-- ...extra args...`** forwards anything after `--` to every train.py invocation (e.g., `-- --save-freq 50000`)
- **macOS-aware:** auto-wraps in `caffeinate -i` so the laptop doesn't sleep, opt-out via `--no-caffeinate`
- **Validated configuration banner** at the top so you see exactly what's about to launch
- **Run summary at the end** with completed/failed/skipped counts and next-step commands (compare.py, analyze.py)
## Default invocation
```bash
scripts/run_pilots.sh --rom path/to/PokemonRed.gb
```
Produces 9 sequential runs in `./training_output/{pixel,symbolic,hybrid}-seed{42,123,456}/` with logs in `./logs/`.
## Parallel mode (recommended for the M3 Max plan)
```bash
scripts/run_pilots.sh --rom path/to/PokemonRed.gb --parallel 3
```
Runs 3 at a time. On a 12-P-core M3 Max this should cut total wall clock from ~13h → ~5h.
## Test plan
- [x] `--help` renders cleanly
- [x] `--dry-run` produces correct 9-command output with `caffeinate -i` prefix
- [x] `--parallel 2 --dry-run` doesn't change command output (parallelism affects execution, not generation)
- [x] `--seeds 42 --treatments pixel,symbolic --dry-run` produces 2 commands with `--save-freq 50000` forwarded after `--`
- [x] Skip-completed correctly skips a directory containing `final_model.zip` and lists it in the summary
- [x] Missing `--rom` errors with a clear message
- [x] Nonexistent ROM path errors with a clear message
- [ ] Real pilot run on M3 Max — to be exercised by Alan when starting the EWRL pilots
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Two professional bash scripts that automate the lifecycle of the EWRL 2026 paper's Overleaf mirror: bin/setup-overleaf-project.sh — one-shot project provisioning that wires a new alias into overleaf-mcp's config + keychain + local clone in a single command. Token is read from $OVERLEAF_TOKEN so it never appears on the command line (no leak via ps). Performs a version preflight (requires overleaf-mcp >= 0.1.2 for non-interactive flags), runs `overleaf-mcp doctor` at the end to verify, and is idempotent: re-running with --force overwrites the alias, an existing clone is left alone. scripts/mirror_paper_to_overleaf.sh — keeps git canonical and Overleaf as a render/edit mirror. Pulls Overleaf first (so manual web-UI edits aren't clobbered), rsyncs paper/ → clone with --delete, commits, pushes. Excludes LaTeX build artefacts (*.aux, *.bbl, etc.) so only source goes to Overleaf. Skips the commit step entirely if nothing changed; supports --dry-run for previewing. Both scripts validated manually against missing-arg, missing-env, missing-clone, and missing-source-dir error paths. Real end-to-end runs are gated on the user creating an Overleaf project and the upstream overleaf-mcp PR (amcheste/overleaf-mcp#13) merging. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
LaTeX skeleton + drafted prose for the EWRL 2026 submission. Sections 1-4 are full first drafts that don't depend on pilot results; sections 5-6 are skeletons with `\todopilot{...}` markers everywhere a number, figure, or interpretation needs to land once the pilots finish.
## What's drafted (no pilot data needed)
- **`paper/main.tex`** — Title, abstract (with one TODO), section glue, `\todopilot{}` macro that renders as `[TODO(pilots): ...]` in red so they're impossible to miss in the PDF.
- **`paper/references.bib`** — 16 citations: PPO, RecurrentPPO/DRQN, rliable, ALE/Atari, Procgen, Craftax, MiniHack, Pokemon-RL prior art, CKA, SB3, Gymnasium, PyBoy.
- **`paper/sections/01_introduction.tex`** — full first draft with contributions list (one TODO for the headline result bullet).
- **`paper/sections/02_related_work.tex`** — full first draft covering long-horizon RL benchmarks, Pokemon-RL prior art, observation representations in RL, recurrent policies, and statistical reporting in deep RL.
- **`paper/sections/03_environment.tex`** — full first draft of the PokeGym section: action space, three observation treatments with exact dimensions, event-flag reward, save-state curriculum, episode truncation, reproducibility note.
- **`paper/sections/04_methods.tex`** — algorithm/architecture, hyperparameter table (one TODO to confirm against `models.py:get_model_config()`), training protocol, evaluation metric, statistical analysis (rliable IQM + 95% bootstrap CIs + PoI), explicit list of deviations from the pre-registered plan, compute placeholder.
## What's skeletoned (waiting on pilot data)
- **`paper/sections/05_results.tex`** — structural skeleton with subsections (Setup recap, Learning Curves, Aggregate Metrics, Probability of Improvement, Milestone Progression, Final-Window Performance, Compute) and `\todopilot{}` markers at every figure caption, table cell, and interpretation paragraph.
- **`paper/sections/06_discussion.tex`** — Discussion structure (What the pilot can/cannot conclude, H1 revisited, Why pixel might underperform, Why hybrid might not strictly dominate, Threats to validity, Future work, Open release). Most prose is drafted — only the results-aware connective tissue uses `\todopilot{}`.
## Build verification
Builds cleanly to a **12-page PDF** on TeX Live 2026 Basic (the minimum macOS install via `brew install --cask basictex`):
```bash
cd paper
make pdf # full build: pdflatex → bibtex → pdflatex × 2
make quick # single pass for prose iteration
make clean # remove build artefacts
```
No `latexmk` or `cleveref` dependency (TeX Live Basic ships neither).
`paper/SCAFFOLDING.md` documents:
- How to grep for `\todopilot{` markers after pilots finish
- How to slot the `scripts/analyze.py`-generated figures into Results
- Style notes for polishing (avoid LLM-isms, drop redundant qualifiers, past/present tense rules)
- How to swap to the EWRL 2026 venue template once it's published
`paper/.gitignore` excludes `*.aux`, `*.bbl`, `*.blg`, `*.log`, `main.pdf`, etc., so build artefacts don't pollute the repo.
## Test plan
- [x] `make pdf` produces a 12-page PDF without errors
- [x] All citations resolve (no `??` in the PDF)
- [x] All `\todopilot{}` markers render as red `[TODO(pilots): ...]` text
- [x] `\Cref{sec:...}` cross-references resolve correctly
- [ ] Manual: spot-check the PDF for typesetting weirdness, run `aspell` on the prose, sanity-check citations against the bib file
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Overleaf returns 403 for any clone URL whose user component isn't literally "git" (matching their documented format https://git:<TOKEN>@git.overleaf.com/<PROJECT_ID>). My initial draft used "x:<TOKEN>" by analogy with GitHub's pattern, which 403s. Confirmed against the live API by setup of the EWRL paper project. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Summary Two professional bash scripts that automate the lifecycle of the EWRL 2026 paper's Overleaf mirror. Companion to <amcheste/overleaf-mcp#13>, which adds the non-interactive CLI flags these scripts depend on. ### `bin/setup-overleaf-project.sh` — one-shot project provisioning ```bash export OVERLEAF_TOKEN=olp_xxxxxx # mint at Overleaf → Account → Git Integration bin/setup-overleaf-project.sh \ --alias pokemon-rl-ewrl-2026 \ --project-id 6620abc123 \ --display-name "Pokemon RL — EWRL 2026" ``` Wires the new alias into the MCP config + keychain + local clone in one command: 1. `overleaf-mcp init --alias ... --project-id ...` (registers alias) 2. `printf '%s' "$OVERLEAF_TOKEN" | overleaf-mcp auth add --project ... --token-stdin` (stores token) 3. `git clone https://x:${OVERLEAF_TOKEN}@git.overleaf.com/<id> ...` (clones with token redacted from output) 4. `overleaf-mcp doctor` (verifies green) Token never appears on the command line (no leak via `ps`). Version preflight requires `overleaf-mcp >= 0.1.2` and prints clear upgrade instructions otherwise. Idempotent: re-running with `--force` overwrites the alias, an existing clone is left alone. ### `scripts/mirror_paper_to_overleaf.sh` — git → Overleaf sync ```bash scripts/mirror_paper_to_overleaf.sh # default alias scripts/mirror_paper_to_overleaf.sh --alias my-paper # different alias scripts/mirror_paper_to_overleaf.sh --dry-run # preview ``` Treats git as canonical, Overleaf as the render/edit mirror: 1. **Pulls from Overleaf first** (`git pull --ff-only`) so manual web-UI edits aren't silently clobbered. If Overleaf has diverged the script aborts with explicit resolution instructions. 2. **Rsync `paper/` → clone** with `--delete` so removed files are pruned on Overleaf. 3. **Excludes LaTeX build artefacts** (`*.aux`, `*.bbl`, `*.log`, etc.) — only source goes to Overleaf. 4. **Skips commit if nothing changed** — re-running on a clean state is a no-op. 5. **Auto-generated commit message** includes the source git SHA + branch for traceability across the two histories. ## How they fit together ``` You write the paper in git → mirror_paper_to_overleaf.sh → Overleaf web UI renders / co-authors edit ↓ (their edits) You pull back via overleaf-mcp sync, then mirror again ``` The setup script is run once per project. The mirror script is run after every paper edit, or wired into a git post-commit hook. ## Test plan - [x] `--help` renders cleanly on both scripts - [x] Missing required args fail with clear error messages and `exit 2` - [x] Mirror script aborts cleanly when source dir or clone is missing - [x] Setup script's version preflight rejects older overleaf-mcp installs - [ ] End-to-end run blocked on (a) Alan creating the Overleaf project and (b) overleaf-mcp PR #13 landing in develop 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The previous README described the project as a hobby RL toolkit focused on "improved exploration training" — copy that pre-dated the current research codebase by several months. It mismatched reality in concrete ways (Python 3.8 vs 3.13, "yourusername" placeholder GitHub URLs, year 2025, default 500k timesteps when scripts/train.py uses 1M and pilots target 10M, "What's Improved" framing relative to a ghost previous version), and made no mention of the work that actually defines the project today: the 3-treatment observation study, pre-registered analysis plan, rliable bootstrap CIs, W&B observability, Streamlit comparison dashboards, alerting system, paper LaTeX, or the 3-paper EWRL/NeurIPS/TMLR cascade. The new README opens with the research question and venue cascade, then orients via a one-table repository tour, a five-step quick-start that takes a reviewer from clone to paper figures, and concrete sections on observation treatments, reward function, statistical methodology, and live monitoring. The "use as a library" path is preserved further down for power users. All emojis dropped — they read as AI-generated to academic reviewers. All factual claims re-verified against the actual code. Also updates paper/README.md to align with the real workflow: figures come from scripts/analyze.py rather than the ghost-empty notebooks/ directory, and the new LaTeX scaffolding (main.tex, sections/, Makefile) is documented alongside the existing analysis_plan.md and compute_ledger.md. Mirror-to-Overleaf flow now linked from the appropriate place. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Six adds + one edit to bring the repo's project files up to the standard expected for a research codebase that external researchers might fork or cite. Nothing in this PR affects code paths. CITATION.cff Schema-compliant citation metadata. Drives GitHub's "Cite this repository" sidebar widget and APA / BibTeX / RIS exports. CONTRIBUTING.md Scope policy (in / out of scope), branch and commit conventions, test-suite expectations, code style, reproducibility expectations (seed_utils, eval.py, compute_ledger), reporting bug template pointers. Codifies the workflow we've been using on PRs #21–#31. .github/PULL_REQUEST_TEMPLATE.md Forces every PR to include Summary, Linked issues, Test plan, Reproducibility considerations, Out of scope. Reproducibility section calls out research-relevant code paths explicitly. .github/ISSUE_TEMPLATE/{bug_report,feature_request}.md + config.yml Two opinionated templates plus a config that disables blank issues and routes general questions to Discussions. Bug template prompts for env / dep versions and W&B run URLs; feature template includes a scope check against the pre-registered analysis plan. .editorconfig Whitespace and indent conventions per file type. Standard for cross-editor consistency; reduces churn from contributors using different IDEs. DEVELOPER_GUIDE.md Refresh Project Overview opening to frame this as the research codebase backing the 3-paper cascade rather than a generic toolkit. Update directory structure to include analysis/, alerts.py, event_flags.py, scripts/, paper/, bin/, configs/, docs/. Add a Research Apparatus section after CLI architecture covering the analysis layer, monitoring callbacks, alerting system, Streamlit dashboards, pilot grid launcher, and Overleaf integration. No changes to existing accurate sections. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Summary Rewrite `README.md` to match the project as it actually exists today — the research codebase producing the EWRL → NeurIPS → TMLR cascade — instead of the hobby RL toolkit described by the previous copy. ## What was wrong with the old README Concrete factual errors and stale framing identified during a doc audit: | Issue | Old | Reality | |-------|-----|---------| | Python version | Badge says 3.13, prerequisites say 3.8+ | 3.13 | | GitHub URL | `yourusername/pokemon-red-ai` placeholder | `amcheste/pokemon-red-ai` | | Citation year | 2025 | 2026 | | Default timesteps | 500k (described as "5x improvement!") | `train.py` uses 1M; pilots target 10M | | Episode length | "3x longer (15k steps)" — relative to *what*? | 15k is the current value, no useful baseline | | Roadmap | Curriculum learning / intrinsic motivation / multi-agent | EWRL May 25 → NeurIPS workshop → TMLR | | Observation types | Mentions multi_modal/screen_only/minimal | Misses the actual research treatments: pixel/symbolic/hybrid | | Statistical methodology | Not mentioned | rliable IQM + bootstrap CIs is core to the paper | | Pre-registered plan | Not mentioned | `paper/analysis_plan.md` is the project's strongest credibility signal | | Live monitoring | TensorBoard | W&B + Streamlit dashboards (`monitor.py`, `compare.py`) | | Alerting | Not mentioned | `pokemon_red_ai.training.alerts` ships with desktop / Slack / email | | Test count | Not mentioned | 833 passing | | Paper artifacts | Not mentioned | `paper/main.tex` + sections + Makefile, mirrored to Overleaf | | Tone | "🚀 Now with improved exploration-focused training!" / "5x increase!" / "📈 IMPROVED!" | Marketing copy reads as AI-generated to academic reviewers | | Emoji density | Every section header | None | ## What the new README does - Opens with the research question (observation representations in long-horizon sparse-reward RL) and the 3-paper venue cascade with a status table - One-table repo tour orienting a new reader to every directory - Five-step quick-start: clone → save states → smoke test → full pilot grid → paper figures - Concrete sections on observation treatments, reward function (linked to `paper/analysis_plan.md` §9), statistical methodology, and live monitoring - "Use as a library" preserved but moved below the research framing - Citation block updated with the real working paper title and 2026 year - Acknowledgments link to PyBoy, SB3, sb3-contrib, Gymnasium, rliable, pret/pokered Net change: 656 → 218 lines. Every claim re-verified against the actual code. ## Also: paper/README.md alignment Updated to remove the stale `notebooks/` reference (no notebooks were ever written; figures come from `scripts/analyze.py`). Added the LaTeX scaffolding files (`main.tex`, `sections/`, `Makefile`), the mirror-to-Overleaf workflow, and a concrete build command. Reproducibility checklist preserved. Net change: 25 → 66 lines. ## Test plan - [x] All cited paths exist on `main` (`paper/main.tex`, `scripts/analyze.py`, `pokemon_red_ai/analysis/`, `docs/research_playbook.md`, `configs/alerts.example.yaml`, etc.) - [x] Test count claim (833) matches `pytest --co -q` - [x] Quick-start commands match what's documented in the playbook and `scripts/run_pilots.sh` - [ ] Manual: render the GitHub view of the README to confirm tables render cleanly 🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary Six adds + one targeted edit to bring the repo's project files up to the standard expected for a research codebase that external researchers might fork or cite. **Zero code changes — pure docs / metadata.** ## What's added | File | Purpose | |------|---------| | `CITATION.cff` | Schema-compliant citation metadata; powers GitHub's "Cite this repository" sidebar widget + APA / BibTeX / RIS exports. | | `CONTRIBUTING.md` | Scope policy (in / out), branch + commit conventions, test expectations, code style, reproducibility expectations. Codifies the workflow we've been using across PRs #21-#31. | | `.github/PULL_REQUEST_TEMPLATE.md` | Forces Summary / Linked issues / Test plan / **Reproducibility considerations** / Out of scope sections. The reproducibility section explicitly calls out research-relevant code paths so we don't accidentally invalidate the pre-registered protocol. | | `.github/ISSUE_TEMPLATE/bug_report.md` | Prompts for env + dep versions and W&B run URLs. No more "it doesn't work" issues. | | `.github/ISSUE_TEMPLATE/feature_request.md` | Includes a scope check against the pre-registered analysis plan and event-flag set. | | `.github/ISSUE_TEMPLATE/config.yml` | Disables blank issues; routes general questions to Discussions. | | `.editorconfig` | Whitespace and indent conventions per file type. | ## What's edited `DEVELOPER_GUIDE.md` — surgical refresh, not a rewrite. The architectural content (mermaid diagrams, layer descriptions) is still accurate and untouched. Three changes: 1. **Project Overview opening** — reframed from "comprehensive RL toolkit" to "research codebase backing the 3-paper cascade" with the toolkit role preserved as secondary. 2. **Directory Structure** — updated to include `analysis/`, `alerts.py`, `event_flags.py`, plus the previously-undocumented `scripts/`, `paper/`, `bin/`, `configs/`, `docs/` directories. 3. **New "Research Apparatus" section** — covers `pokemon_red_ai.analysis`, the monitoring callback hierarchy, `pokemon_red_ai.training.alerts`, the two Streamlit dashboards, `scripts/run_pilots.sh`, and the Overleaf integration. All things we've built in the past few weeks that weren't documented for new contributors. ## Test plan - [x] `git status` clean before commit; `git diff --cached --stat` matches what was intended - [x] `CITATION.cff` parses (basic YAML + cff-version 1.2.0 schema) - [x] No code paths touched, so no test re-run needed - [ ] Manual: render the new docs in GitHub's UI (templates show in the dropdown when opening a new issue / PR) ## Conflict notes This PR is intentionally additive — it does not touch `README.md` or `paper/README.md`, both of which are being rewritten on PR #31. These two PRs can merge in either order. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The smoke test on 2026-04-30 measured ~91 fps with a single PyBoy env on M3 Max, which would put each 10M-step pilot at ~30 hours and threaten the EWRL deadline. PyBoy is single-threaded CPU-bound, so SubprocVecEnv across multiple cores is the right unlock. scripts/train.py — new --n-envs N flag (default 1). When N==1 the single-env DummyVecEnv path is preserved bit-for-bit. When N>1, build N PokemonRedGymEnv factories and pass them to SubprocVecEnv. Each env writes its Monitor CSV to a rank-suffixed path so parallel envs don't clobber each other's logs. W&B integration, alerts, and callbacks are unchanged — they consume SB3's ep_info_buffer regardless of the vectorisation strategy. scripts/run_pilots.sh — adds --n-envs (default 4) and forwards it to every train.py invocation. Also fixes the default --save-state path from the playbook's incorrect "states/post_intro.state" to the actual filename "states/s0_post_intro.state". At --parallel 3 × --n-envs 4 the pilot grid uses 12 P-cores on the M3 Max — the realistic upper bound that completes 9 pilots in ~33 hours wall-clock. pokemon_red_ai/environment/observations.py — fix a real bug surfaced by the smoke test: validate_observation() assumed a Dict observation space, crashing with "'Box' object has no attribute 'spaces'" on the pixel / symbolic / minimal / screen_only treatments (all single-Box spaces). The error is non-fatal but spammed the log every step, which both polluted training logs and degraded throughput. Now handles Dict and Box correctly; emits a clear error for any other type. paper/compute_plan.md — new operational document projecting compute needs across the 3-paper cascade. Confirms M3 Max gets us through EWRL (M2), maps when cloud becomes uncomfortable (M4 NeurIPS) vs mandatory (M5 TMLR), and lists action triggers for the NCSA ACCESS application (AMC-69) and other grants. Sits alongside compute_ledger (actuals) and analysis_plan (locked protocol). Tests: 8 new tests covering the --n-envs flag, the env factory, and the dispatcher's choice between DummyVecEnv and SubprocVecEnv. All 841 tests pass (was 833). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 4-env CPU smoke test on M3 Max measured ~105 fps steady-state,
~1.15× over single-env baseline. PyBoy stepping was scaling but the
PPO gradient update on CPU was the new bottleneck.
Adding --device mps to delegate the gradient update to the M3 Max's
40-core integrated GPU yields ~180 fps in a follow-up smoke test —
~1.75× over 4-env CPU and ~2× over the original baseline. Wall-clock
for 50k smoke run dropped from 9:13 (CPU) to 4:43 (MPS).
For pilots: per-pilot 10M-step run drops from ~27h to ~15h, putting
the 9-pilot grid at ~45 hours wall-clock with --parallel 3 (down from
~3 days), comfortably inside the 25-day EWRL window.
scripts/train.py: new --device flag (default None → SB3 'auto', which
picks cuda or cpu but does NOT pick mps). Plumbed into create_model
via the existing model_overrides dict. Choices restricted to
{auto,cpu,cuda,mps} via click; non-Apple-Silicon users can pass
--device auto or --device cpu explicitly.
scripts/run_pilots.sh: forwards --device to every train.py invocation,
defaulting to mps since the EWRL pilots target the M3 Max. Override
with --device auto on Linux/CUDA hardware.
Tests: existing 841 still pass; the --device flag is a string passthrough
to SB3 so no new behavioural test is required (covered by existing
TestArgumentParser path coverage on similar overrides).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
3 tasks
amcheste
added a commit
that referenced
this pull request
Jun 8, 2026
…of 4) (#47) ## Summary Layer 3 of the pre-pilot QA audit. All changes are documentation / metadata; no behaviour changes. Independent of #45 and #46 — can be reviewed and merged in any order. ### What changed 1. **Save-state path canonicalised** — docs and `scripts/train.py` docstring referenced `states/post_intro.state`; actual path used by `scripts/run_pilots.sh` and `scripts/eval.py` is `save_states/s0_post_intro.state` (what `scripts/create_save_states.py` writes to). Standardised on the canonical path. 2. **\`paper/\` directory links re-pointed** — `MODEL_CARD.md`, `CONTRIBUTING.md`, and `DEVELOPER_GUIDE.md` linked to `paper/analysis_plan.md`, `paper/compute_ledger.md`, `paper/figures/`, `paper/main.tex`. None of these exist in this repo — they're in the private `pokemon-rl-paper` sibling repo (split out via #34). All links now point there with appropriate context. 3. **Day-job IP language genericised** in `docs/research_playbook.md` — the previous wording named a specific current employer and a specific potential affiliation throughout AMC-70 / AMC-68. Rewrote in employer-agnostic terms so a public scrape of the playbook doesn't leak day-job signal into a CAM Labs LLC publication. 4. **Python version unified to 3.10+** — was inconsistent across `pyproject.toml` (>=3.10), README badge (3.13), DEVELOPER_GUIDE (3.8+), CONTRIBUTING (3.10+). All public docs now say 3.10+ (the version range CI actually tests in #46). 5. **DEVELOPER_GUIDE.md refresh** — observation-types section now lists the paper treatments as primary, legacy listed as such; test directory layout reflects the current `tests/` tree; scripts listing includes `seed_utils.py`, `verify_event_flag_ids.py`, and the new CI workflow; GitHub URL fixed (was `yourusername` placeholder); event_flags listed as 15 / pret/pokered-verified. 6. **Test-count "833" claim removed everywhere** — repeated in 5 places (README badge, README repo-at-a-glance table, CONTRIBUTING, two spots in DEVELOPER_GUIDE). Count drifts on every PR; replaced with "passing" (badge) and a pointer to \`pytest tests/\` for the live number. 7. **README "18 critical-path milestones" → 15** with pret/pokered link. 8. **LICENSE copyright** changed to "Copyright (c) 2025-2026 CAM Labs LLC" to match the affiliation declared everywhere else. 9. **CITATION.cff abstract** updated to reflect the 15-flag set. ### Test plan - [x] `pytest tests/` — 841 passed (unchanged from origin/main; this PR is docs-only) - [x] grep -r "Oracle\\|NCSU\\|states/post_intro.state\\|states/s0_post_intro.state\\|833" — clean - [ ] Hover-check the changed Markdown links in GitHub's PR diff view ### Deferred Deletion of `agent.py` and `pokemon_rl_trainer.py` at the repo root. Both are pre-refactor legacy. This PR removed the only doc reference to them (`DEVELOPER_GUIDE.md`); the actual `git rm` was blocked by the safety guard for top-level file removal. Say "ok delete the orphans" and I'll land it as a one-line follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Move all paper-related artifacts (LaTeX source AND the operational documents that describe the research) into a private repository at
amcheste/pokemon-rl-paperto support double-blind review of the EWRL 2026 submission. The public repository becomes purely an open-source environment / training pipeline / analysis tooling codebase.Files moved (now in private repo)
paper/main.tex+paper/sections/*.tex(6 section files)paper/references.bibpaper/Makefile,paper/SCAFFOLDING.md,paper/.gitignorepaper/figures/.gitkeep+paper/notebooks/.gitkeeppaper/analysis_plan.md(pre-registered hypotheses + protocol)paper/compute_ledger.md(append-only run log)paper/README.md(no longer needed in public)paper/ewrl_2025.sty(EWRL template added in private repo's first commit)The entire
paper/directory is removed from the public repo. The public repo is now:pokemon_red_ai/+scripts/+tests/+docs/+bin/+configs/.Pre-registration credibility preserved
analysis_plan.mdwas originally committed publicly on 2026-04-22 in commitdde071c. That commit is preserved in this public repo's git log; reviewers can verify the timestamp and original content. Post-2026-04-30 development of the plan happens in the private repo.The README has been updated to cite
dde071cas the formal pre-registration timestamp.Other documentation updates
README.md— replaced the "paper LaTeX lives in paper/" paragraph with a clear note about the private repo split + the public pre-registration timestamp. Dropped thepaper/row from the Repository-at-a-glance table. Updated the §9-of-analysis-plan reference to cite commitdde071c. Updated Quick Start--output-dirto../pokemon-rl-paper/figures.scripts/mirror_paper_to_overleaf.sh— default--source-dirchanged frompaper(which now doesn't exist) to../pokemon-rl-paper(sibling private repo per~/Repos/amcheste/{...}convention).Sequencing note
PR #33 currently adds
paper/compute_plan.mdto main. After PR #33 lands, this branch will need a 1-line follow-up commit removing that file (or a rebase on top of post-merge main with that deletion added). The follow-up is trivial — flagged here for visibility.Test plan
paper/directoryamcheste/pokemon-rl-papercontains all moved files plus an explanatory README🤖 Generated with Claude Code