Add rliable analysis script (AMC-66)#21
Merged
Merged
Conversation
Implements the pre-registered analysis protocol from analysis_plan.md: IQM point estimates, 95% stratified bootstrap CIs (2000 resamples), pairwise probability of improvement, and performance profiles using the rliable library (Agarwal et al. 2021). Generates four publication-quality figures: aggregate IQM bar chart, performance profiles, probability-of-improvement heatmap, and sample efficiency curves. Supports both eval JSON directory and CSV inputs. Includes 49 tests covering data loading, score matrix construction, all rliable computations, plot generation, and CLI parsing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4 tasks
amcheste
added a commit
that referenced
this pull request
Apr 26, 2026
## Summary Six adds + one targeted edit to bring the repo's project files up to the standard expected for a research codebase that external researchers might fork or cite. **Zero code changes — pure docs / metadata.** ## What's added | File | Purpose | |------|---------| | `CITATION.cff` | Schema-compliant citation metadata; powers GitHub's "Cite this repository" sidebar widget + APA / BibTeX / RIS exports. | | `CONTRIBUTING.md` | Scope policy (in / out), branch + commit conventions, test expectations, code style, reproducibility expectations. Codifies the workflow we've been using across PRs #21-#31. | | `.github/PULL_REQUEST_TEMPLATE.md` | Forces Summary / Linked issues / Test plan / **Reproducibility considerations** / Out of scope sections. The reproducibility section explicitly calls out research-relevant code paths so we don't accidentally invalidate the pre-registered protocol. | | `.github/ISSUE_TEMPLATE/bug_report.md` | Prompts for env + dep versions and W&B run URLs. No more "it doesn't work" issues. | | `.github/ISSUE_TEMPLATE/feature_request.md` | Includes a scope check against the pre-registered analysis plan and event-flag set. | | `.github/ISSUE_TEMPLATE/config.yml` | Disables blank issues; routes general questions to Discussions. | | `.editorconfig` | Whitespace and indent conventions per file type. | ## What's edited `DEVELOPER_GUIDE.md` — surgical refresh, not a rewrite. The architectural content (mermaid diagrams, layer descriptions) is still accurate and untouched. Three changes: 1. **Project Overview opening** — reframed from "comprehensive RL toolkit" to "research codebase backing the 3-paper cascade" with the toolkit role preserved as secondary. 2. **Directory Structure** — updated to include `analysis/`, `alerts.py`, `event_flags.py`, plus the previously-undocumented `scripts/`, `paper/`, `bin/`, `configs/`, `docs/` directories. 3. **New "Research Apparatus" section** — covers `pokemon_red_ai.analysis`, the monitoring callback hierarchy, `pokemon_red_ai.training.alerts`, the two Streamlit dashboards, `scripts/run_pilots.sh`, and the Overleaf integration. All things we've built in the past few weeks that weren't documented for new contributors. ## Test plan - [x] `git status` clean before commit; `git diff --cached --stat` matches what was intended - [x] `CITATION.cff` parses (basic YAML + cff-version 1.2.0 schema) - [x] No code paths touched, so no test re-run needed - [ ] Manual: render the new docs in GitHub's UI (templates show in the dropdown when opening a new issue / PR) ## Conflict notes This PR is intentionally additive — it does not touch `README.md` or `paper/README.md`, both of which are being rewritten on PR #31. These two PRs can merge in either order. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
amcheste
added a commit
that referenced
this pull request
Apr 30, 2026
## Summary - Adds `scripts/analyze.py` — publication-quality statistical analysis using the [rliable](https://github.com/google-research/rliable) library (Agarwal et al. 2021) - Implements the full pre-registered analysis protocol from `paper/analysis_plan.md` §6: - **IQM** (Interquartile Mean) — not raw mean — as the primary aggregate metric - **95% stratified bootstrap CIs** with 2,000 resamples (configurable via `--reps`) - **Probability of Improvement** > 0.75 with CI excluding 0.5 as the significance threshold - **Performance profiles** showing fraction of runs above each score threshold - **Sample efficiency curves** with per-checkpoint IQM and bootstrap CIs - Generates 4 publication-quality figures saved to `paper/figures/`: - `aggregate_iqm.pdf` — bar chart with error bars - `performance_profiles.pdf` — CDFs per treatment - `probability_of_improvement.pdf` — pairwise heatmap with significance markers - `sample_efficiency.pdf` — learning curves across training - Supports two input formats: - **JSON directory**: treatment subdirectories with `eval.py` output JSONs - **CSV**: flat file with treatment/seed/metric columns - 49 tests covering data loading, score matrix construction, all rliable computations, plot generation, CLI parsing ### Usage ```bash # From eval JSON directory (output of scripts/eval.py) python scripts/analyze.py --results-dir eval_results/ -v # From CSV python scripts/analyze.py --csv results.csv --output-dir paper/figures/ # Specific plots only, custom bootstrap count python scripts/analyze.py --results-dir eval_results/ --plots iqm poi --reps 500 ``` ## Test plan - [x] All 49 tests pass (`pytest tests/unit/test_analyze.py -v`) - [x] End-to-end smoke test with synthetic 3-treatment × 5-seed data produces all 3 figures - [x] No deprecation warnings - [x] `rliable==1.2.0` already in `requirements.txt` 🤖 Generated with [Claude Code](https://claude.com/claude-code)
amcheste
pushed a commit
that referenced
this pull request
Apr 30, 2026
Six adds + one edit to bring the repo's project files up to the standard expected for a research codebase that external researchers might fork or cite. Nothing in this PR affects code paths. CITATION.cff Schema-compliant citation metadata. Drives GitHub's "Cite this repository" sidebar widget and APA / BibTeX / RIS exports. CONTRIBUTING.md Scope policy (in / out of scope), branch and commit conventions, test-suite expectations, code style, reproducibility expectations (seed_utils, eval.py, compute_ledger), reporting bug template pointers. Codifies the workflow we've been using on PRs #21–#31. .github/PULL_REQUEST_TEMPLATE.md Forces every PR to include Summary, Linked issues, Test plan, Reproducibility considerations, Out of scope. Reproducibility section calls out research-relevant code paths explicitly. .github/ISSUE_TEMPLATE/{bug_report,feature_request}.md + config.yml Two opinionated templates plus a config that disables blank issues and routes general questions to Discussions. Bug template prompts for env / dep versions and W&B run URLs; feature template includes a scope check against the pre-registered analysis plan. .editorconfig Whitespace and indent conventions per file type. Standard for cross-editor consistency; reduces churn from contributors using different IDEs. DEVELOPER_GUIDE.md Refresh Project Overview opening to frame this as the research codebase backing the 3-paper cascade rather than a generic toolkit. Update directory structure to include analysis/, alerts.py, event_flags.py, scripts/, paper/, bin/, configs/, docs/. Add a Research Apparatus section after CLI architecture covering the analysis layer, monitoring callbacks, alerting system, Streamlit dashboards, pilot grid launcher, and Overleaf integration. No changes to existing accurate sections. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
amcheste
added a commit
that referenced
this pull request
Apr 30, 2026
## Summary Six adds + one targeted edit to bring the repo's project files up to the standard expected for a research codebase that external researchers might fork or cite. **Zero code changes — pure docs / metadata.** ## What's added | File | Purpose | |------|---------| | `CITATION.cff` | Schema-compliant citation metadata; powers GitHub's "Cite this repository" sidebar widget + APA / BibTeX / RIS exports. | | `CONTRIBUTING.md` | Scope policy (in / out), branch + commit conventions, test expectations, code style, reproducibility expectations. Codifies the workflow we've been using across PRs #21-#31. | | `.github/PULL_REQUEST_TEMPLATE.md` | Forces Summary / Linked issues / Test plan / **Reproducibility considerations** / Out of scope sections. The reproducibility section explicitly calls out research-relevant code paths so we don't accidentally invalidate the pre-registered protocol. | | `.github/ISSUE_TEMPLATE/bug_report.md` | Prompts for env + dep versions and W&B run URLs. No more "it doesn't work" issues. | | `.github/ISSUE_TEMPLATE/feature_request.md` | Includes a scope check against the pre-registered analysis plan and event-flag set. | | `.github/ISSUE_TEMPLATE/config.yml` | Disables blank issues; routes general questions to Discussions. | | `.editorconfig` | Whitespace and indent conventions per file type. | ## What's edited `DEVELOPER_GUIDE.md` — surgical refresh, not a rewrite. The architectural content (mermaid diagrams, layer descriptions) is still accurate and untouched. Three changes: 1. **Project Overview opening** — reframed from "comprehensive RL toolkit" to "research codebase backing the 3-paper cascade" with the toolkit role preserved as secondary. 2. **Directory Structure** — updated to include `analysis/`, `alerts.py`, `event_flags.py`, plus the previously-undocumented `scripts/`, `paper/`, `bin/`, `configs/`, `docs/` directories. 3. **New "Research Apparatus" section** — covers `pokemon_red_ai.analysis`, the monitoring callback hierarchy, `pokemon_red_ai.training.alerts`, the two Streamlit dashboards, `scripts/run_pilots.sh`, and the Overleaf integration. All things we've built in the past few weeks that weren't documented for new contributors. ## Test plan - [x] `git status` clean before commit; `git diff --cached --stat` matches what was intended - [x] `CITATION.cff` parses (basic YAML + cff-version 1.2.0 schema) - [x] No code paths touched, so no test re-run needed - [ ] Manual: render the new docs in GitHub's UI (templates show in the dropdown when opening a new issue / PR) ## Conflict notes This PR is intentionally additive — it does not touch `README.md` or `paper/README.md`, both of which are being rewritten on PR #31. These two PRs can merge in either order. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/analyze.py— publication-quality statistical analysis using the rliable library (Agarwal et al. 2021)paper/analysis_plan.md§6:--reps)paper/figures/:aggregate_iqm.pdf— bar chart with error barsperformance_profiles.pdf— CDFs per treatmentprobability_of_improvement.pdf— pairwise heatmap with significance markerssample_efficiency.pdf— learning curves across trainingeval.pyoutput JSONsUsage
Test plan
pytest tests/unit/test_analyze.py -v)rliable==1.2.0already inrequirements.txt🤖 Generated with Claude Code