Summary
The eval CLI resolves an explicit --examples value and passes it to GPQAEval, but also passes debug=True whenever --debug is set. Inside GPQAEval, the debug branch selects one fixed Espresso example and never consults num_examples.
So python -m gpt_oss.evals --debug --examples 2 --eval gpqa ... still evaluates the single hard-coded debug example instead of the requested two-example subset.
Proposed resolution
Treat the fixed GPQA debug example as the default only when no explicit sample count is supplied. When --examples is present, disable the fixed-example branch and use the normal deterministic subset sampler with one repeat. Add lightweight CLI-resolution coverage.
Summary
The eval CLI resolves an explicit
--examplesvalue and passes it toGPQAEval, but also passesdebug=Truewhenever--debugis set. InsideGPQAEval, the debug branch selects one fixed Espresso example and never consultsnum_examples.So
python -m gpt_oss.evals --debug --examples 2 --eval gpqa ...still evaluates the single hard-coded debug example instead of the requested two-example subset.Proposed resolution
Treat the fixed GPQA debug example as the default only when no explicit sample count is supplied. When
--examplesis present, disable the fixed-example branch and use the normal deterministic subset sampler with one repeat. Add lightweight CLI-resolution coverage.