Skip to content

feat(eval): statistical tests #3055

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mpatel31415 wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

feat(eval): statistical tests #3055

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fab3547
feat(eval): statistical tests - initial commit
mpatel31415 Sep 3, 2026
cfbdf89
simplified code
mpatel31415 Sep 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 55 additions & 1 deletion fern/versions/latest/pages/reference/cli-commands.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ gym eval health-check # verify rollout artifact quality for an existing
gym eval profile # compute a reward profile from rollouts
gym eval reverify # recompute rewards from existing rollouts without re-running inference
gym eval compare # compare a baseline eval run against a candidate run
gym eval stat-test # paired statistical significance test of a baseline run vs a candidate run

# Contributor helpers
gym dev test # run NeMo Gym's unit tests
Expand Down Expand Up @@ -777,7 +778,9 @@ See [Repeat-Level Metrics](/evaluation/aggregate-metrics#repeat-level-metrics) f

Compare a baseline eval run against a candidate run and write a report.

The report contains a metric table (the change, plus each side's value and confidence interval), split into key metrics and all other metrics, and a per-task sample-flips table. Confidence intervals are read from what each run recorded; this command does not yet perform statistical tests or issue a pass/fail verdict.
The report contains a metric table (the change, plus each side's value and confidence interval), split into key metrics and all other metrics, and a per-task sample-flips table. Confidence intervals are read from what each run recorded.

By default, `gym eval compare` also runs a paired statistical significance test on every key metric with per-task pairing data (the same test [`gym eval stat-test`](#gym-eval-stat-test) runs standalone) and writes it to a separate `statistical_tests/` subdirectory next to the report — `compare_report.md`/`compare_report.json` themselves are unaffected by this step; it's a side effect, not a schema change.

| Option | Description |
| --- | --- |
Expand All @@ -790,6 +793,11 @@ The report contains a metric table (the change, plus each side's value and confi
| `--candidate-agents NAME[,NAME...]` | Agent to read from each candidate, in `--candidates` order. Takes precedence over `--agent`. |
| `--output-dir DIR`, `-o` | Where to write the report. Defaults to the directory containing the candidate rollouts JSONL. |
| `--report-format` | `md`, `json`, or `both` (default). |
| `--metric METRIC[,METRIC...]` | Restrict the statistics step to specific metric(s). Defaults to every key metric with per-task pairing data. |
| `--margin DELTA` | Non-inferiority margin for the statistics step (e.g. `0.01` for 1pp). See [`gym eval stat-test`](#gym-eval-stat-test) for the two framings this switches between. |
| `--alpha` | Significance level for the statistics step (default `0.05`). |
| `--stats-output-dir DIR` | Where to write the statistics artifacts. Independent of `--output-dir`, which only controls `compare_report.*`. Defaults to `statistical_tests/` inside the candidate run's own directory. |
| `--no-stats` | Skip the statistics step entirely — no `statistical_tests/` directory is written. |

```bash
gym eval compare \
Expand All @@ -803,9 +811,55 @@ Writes into the output directory:
| --- | --- |
| `compare_report.md` | Human-readable report. |
| `compare_report.json` | Machine-readable result (`schema_version` `"1"`). |
| `statistical_tests/*` | The default statistics step's own artifacts (see `--no-stats` above). |

If a run was collected with `--disable-aggregation` it has no `*_aggregate_metrics.json`; run [`gym eval aggregate`](#gym-eval-aggregate) first, or point at the file directly with `--baseline-agg-metrics` / `--candidates-agg-metrics`.

### `gym eval stat-test`

Paired statistical significance test of a baseline eval run against a candidate run — the same test `gym eval compare` runs by default, but standalone: `gym eval stat-test` never depends on `compare` having been run, and never reads or writes `compare_report.*`.

The method is a single paired test on per-task mean scores (Evan Miller, "Adding Error Bars to Evals", arXiv:2411.00640): each task's repeats are collapsed to one mean, baseline and candidate are paired per task, and the Central Limit Theorem licenses a t-test on the average of the paired differences. Two framings, selected by whether `--margin` is given:

- **No `--margin`**: two-sided test of "did anything change at all" — for repeatability checks (same benchmark/model run twice).
- **`--margin DELTA`**: one-sided non-inferiority test of "is the candidate not meaningfully worse than `DELTA`" — for regression checks with a tolerance (e.g. FP4 quantization: is accuracy within 1pp of the baseline).

Reports only the raw statistical output (n, mean difference, standard error, p-value, significant, minimum detectable effect at 80% power) — no PASS/WARN/FAIL verdict.

| Option | Description |
| --- | --- |
| `--baseline PATH` | Baseline run's rollouts JSONL. Its `*_aggregate_metrics.json` sibling is what gets read. |
| `--candidates PATH[,PATH...]` | Candidate run's rollouts JSONL. One candidate is supported today. |
| `--baseline-agg-metrics PATH` | Baseline's aggregate-metrics JSON, when it is not the sibling of `--baseline`. |
| `--candidates-agg-metrics PATH[,PATH...]` | Candidate's aggregate-metrics JSON, when not the sibling of `--candidates`. |
| `--agent NAME`, `--baseline-agent NAME`, `--candidate-agents NAME[,NAME...]` | Same agent-selection semantics as `gym eval compare`. |
| `--test NAME` | Which statistical test to run. `paired` (the default) is the only one today; `--metric` and `--margin` below belong to it. |
| `--metric METRIC[,METRIC...]` | Metric(s) to test (bare field name, e.g. `reward`, not `mean/reward`). Defaults to every key metric with per-task pairing data. |
| `--margin DELTA` | Non-inferiority margin. See above. |
| `--alpha` | Significance level (default `0.05`). |
| `--output-dir DIR`, `-o` | Given explicitly, used literally. Left unset, defaults to `statistical_tests/` inside the candidate run's own directory (auto-created). |
| `--report-format` | `md`, `json`, or `both` (default). |

```bash
# Repeatability: is the difference between two runs of the same config noise, or real?
gym eval stat-test \
--baseline runs/run_a/rollouts.jsonl \
--candidates runs/run_b/rollouts.jsonl

# FP4 quantization: is the candidate not meaningfully worse than a 1pp margin?
gym eval stat-test \
--baseline runs/bf16/rollouts.jsonl \
--candidates runs/nvfp4/rollouts.jsonl \
--metric reward --margin 0.01
```

Writes into `statistical_tests/` (or `--output-dir` directly, with no nesting, when given explicitly), with the filename indexed by this invocation's parameters — a rerun with a different `--metric`/`--margin`/`--alpha` against the same location writes an additional file rather than overwriting the previous one:

```
statistical_tests/paired__two-sided__alpha-0.05.{md,json}
statistical_tests/paired__metric-reward__margin-0.01__alpha-0.05.{md,json}
```

### `gym eval reverify`

Recompute rewards from existing rollouts by replaying them through a resources server's `/verify` endpoint — without re-running model inference. Starts the resources server automatically from the provided config. Requires the `*_materialized_inputs.jsonl` and `rollouts.jsonl` artifacts produced by `gym eval run`.
Expand Down
55 changes: 53 additions & 2 deletions nemo_gym/cli/eval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
from copy import deepcopy
from multiprocessing import Pool
from pathlib import Path
from typing import Any, Dict, List, Tuple
from typing import Any, Dict, List, Optional, Tuple

from omegaconf import DictConfig, OmegaConf, open_dict
from pydantic import Field
Expand Down Expand Up @@ -602,16 +602,67 @@ def reward_profile(): # pragma: no cover
Repeat-level metrics: {repeat_level_metrics_fpath}""")


def _run_stats_step_for_compare(config: Any, overrides: Dict[str, Any]) -> None:
"""Build the `stat-test` config for `gym eval compare`'s default stats step and run it.

`overrides` carries the raw stats-flag values off `eval compare`'s own global config dict
(`metric`/`margin`/`alpha`/`stats_output_dirpath`) -- `stats_output_dirpath` is remapped to
`output_dirpath` here since `--output-dir` on `compare` must keep controlling only
`compare_report.*`, not this step's own output location.
"""
from nemo_gym.statistical_tests.registry import resolve_stat_test
from nemo_gym.statistical_tests.schema import DEFAULT_STAT_TEST

stats_config_dict = config.model_dump(exclude={"output_dirpath"})
stats_config_dict.update({k: v for k, v in overrides.items() if k != "stats_output_dirpath"})
if overrides.get("stats_output_dirpath"):
stats_config_dict["output_dirpath"] = overrides["stats_output_dirpath"]
test = resolve_stat_test(stats_config_dict.get("test") or DEFAULT_STAT_TEST)
stat_test(test.config_type.model_validate(stats_config_dict))


@exit_cleanly_on_config_error
def compare() -> None: # pragma: no cover
from nemo_gym.comparison.report import render_key_metrics_tables, summary_lines
from nemo_gym.comparison.runner import invoked_command, run_comparison
from nemo_gym.comparison.schema import ComparisonConfig

config = ComparisonConfig.model_validate(get_global_config_dict())
global_config_dict = get_global_config_dict()
config = ComparisonConfig.model_validate(global_config_dict)

result, written = run_comparison(config, invoked_command())

for table in render_key_metrics_tables(result):
print_rich_table(table)
print("\n".join(summary_lines(result, written)))

if not global_config_dict.get("no_stats", False):
_run_stats_step_for_compare(
config,
{
"metric": global_config_dict.get("metric"),
"margin": global_config_dict.get("margin"),
"alpha": global_config_dict.get("alpha", 0.05),
"stats_output_dirpath": global_config_dict.get("stats_output_dirpath"),
},
)


@exit_cleanly_on_config_error
def stat_test(config: Optional[Any] = None) -> None: # pragma: no cover
from nemo_gym.statistical_tests.common import invoked_command
from nemo_gym.statistical_tests.registry import resolve_stat_test, run_stat_test
from nemo_gym.statistical_tests.schema import DEFAULT_STAT_TEST

standalone = config is None
if standalone:
global_config_dict = get_global_config_dict()
test = resolve_stat_test(global_config_dict.get("test") or DEFAULT_STAT_TEST)
config = test.config_type.model_validate(global_config_dict)
else:
test = resolve_stat_test(config.test)

# Record whichever command actually ran: sys.argv holds *its* overrides, not stat-test's.
report, written = run_stat_test(test, config, invoked_command("stat-test" if standalone else "compare"))

print("\n".join(test.summary(report, written)))
111 changes: 85 additions & 26 deletions nemo_gym/cli/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -631,6 +631,61 @@ def _dataset_download(args: argparse.Namespace, overrides: list[str]) -> None:
dispatch(targets[args.storage], overrides)


# Run-selection flags shared by `eval compare` and `eval stat-test` -- both read the same
# `<stem>_aggregate_metrics.json` pair and pick an agent the same way.
_EVAL_RUN_SELECTION_FLAGS = (
_value_flag(
"baseline",
"baseline_rollouts_jsonl_fpath",
"Baseline run's rollouts JSONL (its *_aggregate_metrics.json sibling is what gets read today).",
quote=True,
),
_comma_list_flag(
"candidates",
"candidate_rollouts_jsonl_fpaths",
"Candidate run's rollouts JSONL. Comma-separated list; one candidate is supported today.",
metavar="PATH[,PATH...]",
),
_value_flag(
"baseline-agg-metrics",
"baseline_aggregate_metrics_fpath",
"Baseline's aggregate-metrics JSON, when it is not the sibling of --baseline.",
quote=True,
),
_comma_list_flag(
"candidates-agg-metrics",
"candidate_aggregate_metrics_fpaths",
"Candidates' aggregate-metrics JSON, in --candidates order, when not siblings of --candidates.",
metavar="PATH[,PATH...]",
),
_value_flag("agent", "agent_name", "Agent to compare on both sides (default: all shared agents)."),
_value_flag("baseline-agent", "baseline_agent_name", "Agent to read from the baseline's metrics."),
_comma_list_flag(
"candidate-agents",
"candidate_agent_names",
"Agent to read from each candidate's metrics, in --candidates order.",
metavar="NAME[,NAME...]",
),
)

# Statistical-test flags shared by `eval compare` (its default stats step) and `eval stat-test`.
_STATISTICAL_TEST_FLAGS = (
_comma_list_flag(
"metric",
"metric",
"Metric(s) to test, e.g. `reward` (comma-separated). Default: every key metric with per-task pairing data.",
metavar="METRIC[,METRIC...]",
),
_value_flag(
"margin",
"margin",
"Non-inferiority margin delta (e.g. 0.01 for 1pp). Given: one-sided test of 'candidate is not "
"meaningfully worse than delta'. Omitted: two-sided test of 'did anything change at all'.",
),
_value_flag("alpha", "alpha", "Significance level (default: 0.05)."),
)


# One-line help for each command group, shown in `gym --help`.
GROUPS = {
"list": "List available components (benchmarks, environments, agents, models, resources-servers).",
Expand Down Expand Up @@ -1074,42 +1129,41 @@ def _dataset_download(args: argparse.Namespace, overrides: list[str]) -> None:
target="nemo_gym.cli.eval:compare",
summary="Compare a baseline eval run against candidate runs.",
flags=(
*_EVAL_RUN_SELECTION_FLAGS,
_value_flag(
"baseline",
"baseline_rollouts_jsonl_fpath",
"Baseline run's rollouts JSONL (its *_aggregate_metrics.json sibling is what gets read today).",
"output-dir",
"output_dirpath",
"Where to write the report (default: the candidate run's own directory).",
aliases=("-o",),
quote=True,
),
_comma_list_flag(
"candidates",
"candidate_rollouts_jsonl_fpaths",
"Candidate run's rollouts JSONL. Comma-separated list; one candidate is supported today.",
metavar="PATH[,PATH...]",
_value_flag(
"report-format",
"report_format",
"Report artifacts to write (default: both).",
choices=("md", "json", "both"),
),
*_STATISTICAL_TEST_FLAGS,
_value_flag(
"baseline-agg-metrics",
"baseline_aggregate_metrics_fpath",
"Baseline's aggregate-metrics JSON, when it is not the sibling of --baseline.",
"stats-output-dir",
"stats_output_dirpath",
"Where to write the statistical-test step's own report (default: "
"`<candidate run's directory>/statistical_tests/`). Independent of --output-dir, which "
"controls only compare_report.*.",
quote=True,
),
_comma_list_flag(
"candidates-agg-metrics",
"candidate_aggregate_metrics_fpaths",
"Candidates' aggregate-metrics JSON, in --candidates order, when not siblings of --candidates.",
metavar="PATH[,PATH...]",
),
_value_flag("agent", "agent_name", "Agent to compare on both sides (default: all shared agents)."),
_value_flag("baseline-agent", "baseline_agent_name", "Agent to read from the baseline's metrics."),
_comma_list_flag(
"candidate-agents",
"candidate_agent_names",
"Agent to read from each candidate's metrics, in --candidates order.",
metavar="NAME[,NAME...]",
),
_bool_flag("no-stats", "no_stats", "Skip the default statistical-test step."),
),
),
"eval stat-test": Command(
target="nemo_gym.cli.eval:stat_test",
summary="Statistical significance test between a baseline and a candidate run (default: paired).",
flags=(
*_EVAL_RUN_SELECTION_FLAGS,
_value_flag(
"output-dir",
"output_dirpath",
"Where to write the report (default: the candidate run's own directory).",
"Where to write the report (default: `<candidate run's directory>/statistical_tests/`).",
aliases=("-o",),
quote=True,
),
Expand All @@ -1119,6 +1173,11 @@ def _dataset_download(args: argparse.Namespace, overrides: list[str]) -> None:
"Report artifacts to write (default: both).",
choices=("md", "json", "both"),
),
# Choices are spelled out rather than derived from `statistical_tests.registry`:
# importing it here would put pydantic + the whole stats package on the path of every
# `gym` invocation. `test_cli_test_flag_choices_match_the_registry` pins them together.
_value_flag("test", "test", "Statistical test to run (default: paired).", choices=("paired",)),
*_STATISTICAL_TEST_FLAGS,
),
),
"dev test": Command(target="nemo_gym.cli.dev:dev_test", summary="Run NeMo Gym's unit tests."),
Expand Down
Loading
Loading