Summary
The eval CLI passes n_repeats=8 to AIME and GPQA whenever --debug is false, even when the user supplies --examples.
Both constructors intentionally require n_repeats == 1 when a positive subset size is requested:
assert n_repeats == 1, "n_repeats only supported for num_examples = None"
So a documented command such as python -m gpt_oss.evals --examples 2 --eval aime25 ... reaches the constructor with num_examples=2, n_repeats=8 and fails before evaluation starts. GPQA has the same path.
Proposed resolution
When an explicit sample count is supplied, set n_repeats=1 for AIME and GPQA regardless of debug mode. Preserve 8 repeats only for full non-debug evaluations. Add lightweight regression coverage for explicit, debug, and full-run repeat selection.
Summary
The eval CLI passes
n_repeats=8to AIME and GPQA whenever--debugis false, even when the user supplies--examples.Both constructors intentionally require
n_repeats == 1when a positive subset size is requested:So a documented command such as
python -m gpt_oss.evals --examples 2 --eval aime25 ...reaches the constructor withnum_examples=2, n_repeats=8and fails before evaluation starts. GPQA has the same path.Proposed resolution
When an explicit sample count is supplied, set
n_repeats=1for AIME and GPQA regardless of debug mode. Preserve 8 repeats only for full non-debug evaluations. Add lightweight regression coverage for explicit, debug, and full-run repeat selection.