Skip to content

Commit bf63c4f

Browse files
authored
Add rliable analysis script (AMC-66) (#21)
## Summary - Adds `scripts/analyze.py` — publication-quality statistical analysis using the [rliable](https://github.com/google-research/rliable) library (Agarwal et al. 2021) - Implements the full pre-registered analysis protocol from `paper/analysis_plan.md` §6: - **IQM** (Interquartile Mean) — not raw mean — as the primary aggregate metric - **95% stratified bootstrap CIs** with 2,000 resamples (configurable via `--reps`) - **Probability of Improvement** > 0.75 with CI excluding 0.5 as the significance threshold - **Performance profiles** showing fraction of runs above each score threshold - **Sample efficiency curves** with per-checkpoint IQM and bootstrap CIs - Generates 4 publication-quality figures saved to `paper/figures/`: - `aggregate_iqm.pdf` — bar chart with error bars - `performance_profiles.pdf` — CDFs per treatment - `probability_of_improvement.pdf` — pairwise heatmap with significance markers - `sample_efficiency.pdf` — learning curves across training - Supports two input formats: - **JSON directory**: treatment subdirectories with `eval.py` output JSONs - **CSV**: flat file with treatment/seed/metric columns - 49 tests covering data loading, score matrix construction, all rliable computations, plot generation, CLI parsing ### Usage ```bash # From eval JSON directory (output of scripts/eval.py) python scripts/analyze.py --results-dir eval_results/ -v # From CSV python scripts/analyze.py --csv results.csv --output-dir paper/figures/ # Specific plots only, custom bootstrap count python scripts/analyze.py --results-dir eval_results/ --plots iqm poi --reps 500 ``` ## Test plan - [x] All 49 tests pass (`pytest tests/unit/test_analyze.py -v`) - [x] End-to-end smoke test with synthetic 3-treatment × 5-seed data produces all 3 figures - [x] No deprecation warnings - [x] `rliable==1.2.0` already in `requirements.txt` 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent fd84311 commit bf63c4f

2 files changed

Lines changed: 1707 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)