Commit bf63c4f
authored
Add rliable analysis script (AMC-66) (#21)
## Summary
- Adds `scripts/analyze.py` — publication-quality statistical analysis using the [rliable](https://github.com/google-research/rliable) library (Agarwal et al. 2021)
- Implements the full pre-registered analysis protocol from `paper/analysis_plan.md` §6:
- **IQM** (Interquartile Mean) — not raw mean — as the primary aggregate metric
- **95% stratified bootstrap CIs** with 2,000 resamples (configurable via `--reps`)
- **Probability of Improvement** > 0.75 with CI excluding 0.5 as the significance threshold
- **Performance profiles** showing fraction of runs above each score threshold
- **Sample efficiency curves** with per-checkpoint IQM and bootstrap CIs
- Generates 4 publication-quality figures saved to `paper/figures/`:
- `aggregate_iqm.pdf` — bar chart with error bars
- `performance_profiles.pdf` — CDFs per treatment
- `probability_of_improvement.pdf` — pairwise heatmap with significance markers
- `sample_efficiency.pdf` — learning curves across training
- Supports two input formats:
- **JSON directory**: treatment subdirectories with `eval.py` output JSONs
- **CSV**: flat file with treatment/seed/metric columns
- 49 tests covering data loading, score matrix construction, all rliable computations, plot generation, CLI parsing
### Usage
```bash
# From eval JSON directory (output of scripts/eval.py)
python scripts/analyze.py --results-dir eval_results/ -v
# From CSV
python scripts/analyze.py --csv results.csv --output-dir paper/figures/
# Specific plots only, custom bootstrap count
python scripts/analyze.py --results-dir eval_results/ --plots iqm poi --reps 500
```
## Test plan
- [x] All 49 tests pass (`pytest tests/unit/test_analyze.py -v`)
- [x] End-to-end smoke test with synthetic 3-treatment × 5-seed data produces all 3 figures
- [x] No deprecation warnings
- [x] `rliable==1.2.0` already in `requirements.txt`
🤖 Generated with [Claude Code](https://claude.com/claude-code)1 parent fd84311 commit bf63c4f
2 files changed
Lines changed: 1707 additions & 0 deletions
0 commit comments