Add experiment comparison views (AMC-79)#26
Merged
Conversation
Add a new pokemon_red_ai.analysis package with treatment-comparison logic — detection from run names, learning curves with mean ± std bands, IQM with bootstrap CIs, milestone-first-episode race chart, final-window performance bars, and matplotlib publication-style figure export to PDF/SVG/PNG. A new Streamlit app at scripts/compare.py wires it all together: multi-run sidebar selector, per-run treatment override, learning-curve overlay, IQM table, final-performance bars, milestone race table, and figure download buttons. A parallel W&B report layout is documented in configs/wandb_report_template.md so reviewers can see the same panels without leaving the W&B UI. Closes the Training Observability milestone alongside AMC-76/77/78. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2 tasks
amcheste
added a commit
that referenced
this pull request
Apr 26, 2026
## Summary A self-contained walkthrough at `docs/research_playbook.md` covering everything between right-now and the 2026-05-25 EWRL deadline: - **Step 0:** Merge PR #26 (closes Training Observability) - **Step 1:** Pre-pilot prep — W&B login, alerts config, compute decision (local vs RunPod ~$110 vs Lambda ~$135), 5-min smoke test - **Step 2:** Run the 9 pilots — exact commands, suggested order (pixel → symbolic → hybrid), `tmux` tip, "when to kill a run early" rules, per-run checklist template - **Step 3:** Analyze with `scripts/compare.py` + `scripts/analyze.py` - **Step 4:** Draft the EWRL paper (Claude does most; me on review/polish) - **Step 5:** Parallel research ops — Oracle IP review (must precede arXiv preprint), NCSA ACCESS grant, NC State advisor outreach Includes a timeline arithmetic section showing ~14 days of slack against the deadline if pilots take 10 days, plus a quick-reference command cheat sheet at the end. ## Test plan - [x] `cat docs/research_playbook.md` renders cleanly - [ ] Walking through Step 1 commands once — verify smoke test launches and Streamlit dashboard updates 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 tasks
amcheste
added a commit
to amcheste/mac-dev-setup
that referenced
this pull request
Apr 26, 2026
## Summary Adds Step 3 to `setup-repo`: disable squash merging at the repo level, set rebase as the default merge style, keep merge commits enabled as a fallback for ceremonial merges (e.g. CLI `--no-ff` `develop → main` releases). ## Why Squash-merging PRs silently: 1. Replaces the bot's primary authorship with the merger 2. Drops the `Co-Authored-By: amcheste` trailers entirely That happened today on [pokemon-red-ai#25](amcheste/pokemon-red-ai#25) and [#26](amcheste/pokemon-red-ai#26): the squash commits ended up authored by `amcheste-ai-agent[bot]` (the App identity, not the bot user account), with no co-author trailers, and `@amcheste` got zero contribution-graph credit. Rebase merge preserves per-commit authorship and trailers, so each bot commit lands on the default branch with `@amcheste` as a structurally-recognized co-author (verified via GraphQL `Commit.authors`). The skill update closes the gap on **new** repos. Existing repos (15 `amcheste/*` + 3 `amcheste-ai-agent/*`) were already updated out-of-band via direct PATCH calls. ## Changes - New Step 3: *Set merge policy: disable squash, default to rebase* - Old Steps 4/5/6 renumbered to 5/6/7 - Summary block updated to surface the merge-policy line ## Test plan - [x] SKILL.md renders correctly - [ ] Run `/setup-repo` against a new test repo and confirm the merge settings are applied ## Related - engineering-handbook `docs/philosophies/merge-strategy.md` — the philosophy doc that motivates this rule (separate PR coming)
amcheste
pushed a commit
that referenced
this pull request
Apr 30, 2026
## Summary - New `pokemon_red_ai.analysis` package with reusable, testable comparison logic — separable from the Streamlit UI so it's importable from any script (paper figure generation, notebooks, etc.). - **Treatment detection** from run names: matches `pixel`, `symbolic`, `hybrid`, `multi_modal`, `screen_only`, `minimal` via two-pass regex (multi-word treatments first via boundary-anchored match, then single-word via token split). Falls back to `unknown`. - **Learning curves with bands**: per-treatment mean ± std across seeds, with optional rolling smoothing and length alignment to the shortest seed in the group. - **IQM + bootstrap CIs**: per-seed best-reward distribution per treatment, with percentile bootstrap CIs (pure-numpy implementation, no rliable runtime dependency at call time but mathematically equivalent). - **Milestone race chart**: per-treatment median (or min/mean) of `first_episode` for each event flag — answers "which treatment hits BEAT_BROCK first?". - **Final-performance bars**: mean ± std of last-N-episode reward per treatment. - **Publication-quality figure export**: matplotlib serif font, 300 DPI, PDF/SVG/PNG via `export_figure()`. Style matches `scripts/analyze.py` so live and paper figures are visually consistent. - New `scripts/compare.py` Streamlit app wires it all together: multi-run sidebar selector, per-run treatment override (for badly-named runs), learning-curve overlay, IQM table, final-performance bars, milestone race table, plus PDF/SVG/PNG download buttons for each figure. - `configs/wandb_report_template.md` — manual W&B Report panel spec mirroring the Streamlit layout. (W&B Reports SDK is unstable across versions; markdown spec is more durable.) - 57 new unit tests covering: 8 treatment-name patterns + 4 unknown fallbacks, grouping with deterministic ordering, learning curve construction (single/multi seed, alignment, smoothing, truncation), IQM math (tail-dropping, edge cases, bootstrap CI brackets), summary table (custom metrics, single-seed std=0), milestone race (median/min/mean aggregators, NaN for never-triggered, whitelist), final-performance bars, and PDF/SVG/PNG export including unsupported-extension errors and parent-dir creation. **Closes the Training Observability milestone** alongside AMC-76 / 77 / 78. Streamlit + rliable + alerting + comparison — the full live-research toolkit, ready for the EWRL pilots. ## Test plan - [x] All 57 new tests pass - [x] Full project suite: 833 passed, 0 failed - [x] `python scripts/compare.py --help` parses correctly - [x] `from pokemon_red_ai.analysis import detect_treatment` works - [ ] Manual: `streamlit run scripts/compare.py -- --runs-dir ./training_output` once 3+ pilot runs exist; verify learning curves render with bands, IQM table populates, PDF download produces a valid file 🤖 Generated with [Claude Code](https://claude.com/claude-code)
amcheste
pushed a commit
that referenced
this pull request
Apr 30, 2026
A self-contained walkthrough covering: merging PR #26, pre-pilot prep (W&B login, alerts config, smoke test, compute decision), running the 9 pilots (commands, monitoring, when to kill early), analyzing results (compare.py + analyze.py), drafting the EWRL paper, and parallel research-ops tasks (Oracle IP, NCSA grant, NC State advisor). Includes a timeline arithmetic section showing ~14 days of slack against the 2026-05-25 EWRL deadline if pilots take 10 days, and a quick-reference command cheat sheet at the end. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
amcheste
added a commit
that referenced
this pull request
Apr 30, 2026
## Summary A self-contained walkthrough at `docs/research_playbook.md` covering everything between right-now and the 2026-05-25 EWRL deadline: - **Step 0:** Merge PR #26 (closes Training Observability) - **Step 1:** Pre-pilot prep — W&B login, alerts config, compute decision (local vs RunPod ~$110 vs Lambda ~$135), 5-min smoke test - **Step 2:** Run the 9 pilots — exact commands, suggested order (pixel → symbolic → hybrid), `tmux` tip, "when to kill a run early" rules, per-run checklist template - **Step 3:** Analyze with `scripts/compare.py` + `scripts/analyze.py` - **Step 4:** Draft the EWRL paper (Claude does most; me on review/polish) - **Step 5:** Parallel research ops — Oracle IP review (must precede arXiv preprint), NCSA ACCESS grant, NC State advisor outreach Includes a timeline arithmetic section showing ~14 days of slack against the deadline if pilots take 10 days, plus a quick-reference command cheat sheet at the end. ## Test plan - [x] `cat docs/research_playbook.md` renders cleanly - [ ] Walking through Step 1 commands once — verify smoke test launches and Streamlit dashboard updates 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New
pokemon_red_ai.analysispackage with reusable, testable comparison logic — separable from the Streamlit UI so it's importable from any script (paper figure generation, notebooks, etc.).Treatment detection from run names: matches
pixel,symbolic,hybrid,multi_modal,screen_only,minimalvia two-pass regex (multi-word treatments first via boundary-anchored match, then single-word via token split). Falls back tounknown.Learning curves with bands: per-treatment mean ± std across seeds, with optional rolling smoothing and length alignment to the shortest seed in the group.
IQM + bootstrap CIs: per-seed best-reward distribution per treatment, with percentile bootstrap CIs (pure-numpy implementation, no rliable runtime dependency at call time but mathematically equivalent).
Milestone race chart: per-treatment median (or min/mean) of
first_episodefor each event flag — answers "which treatment hits BEAT_BROCK first?".Final-performance bars: mean ± std of last-N-episode reward per treatment.
Publication-quality figure export: matplotlib serif font, 300 DPI, PDF/SVG/PNG via
export_figure(). Style matchesscripts/analyze.pyso live and paper figures are visually consistent.New
scripts/compare.pyStreamlit app wires it all together: multi-run sidebar selector, per-run treatment override (for badly-named runs), learning-curve overlay, IQM table, final-performance bars, milestone race table, plus PDF/SVG/PNG download buttons for each figure.configs/wandb_report_template.md— manual W&B Report panel spec mirroring the Streamlit layout. (W&B Reports SDK is unstable across versions; markdown spec is more durable.)57 new unit tests covering: 8 treatment-name patterns + 4 unknown fallbacks, grouping with deterministic ordering, learning curve construction (single/multi seed, alignment, smoothing, truncation), IQM math (tail-dropping, edge cases, bootstrap CI brackets), summary table (custom metrics, single-seed std=0), milestone race (median/min/mean aggregators, NaN for never-triggered, whitelist), final-performance bars, and PDF/SVG/PNG export including unsupported-extension errors and parent-dir creation.
Closes the Training Observability milestone alongside AMC-76 / 77 / 78. Streamlit + rliable + alerting + comparison — the full live-research toolkit, ready for the EWRL pilots.
Test plan
python scripts/compare.py --helpparses correctlyfrom pokemon_red_ai.analysis import detect_treatmentworksstreamlit run scripts/compare.py -- --runs-dir ./training_outputonce 3+ pilot runs exist; verify learning curves render with bands, IQM table populates, PDF download produces a valid file🤖 Generated with Claude Code