Summary
The eval CLI documents --examples as overriding the default sample count, but all three HealthBench cases hard-code num_examples=10 if debug_mode else num_examples.
As a result, python -m gpt_oss.evals --debug --examples 2 --eval healthbench ... still evaluates 10 examples. GPQA and AIME honor the same explicit override.
Impact
Debug runs can unexpectedly perform five times the requested model/grader work, which is particularly costly for HealthBench because each sample is graded across multiple rubric items.
Proposed resolution
Use the explicit --examples value whenever provided. Only fall back to HealthBench's debug default of 10 when debug mode is active and the user did not specify a count. Add focused coverage for explicit, debug-default, and full-eval cases.
Summary
The eval CLI documents
--examplesas overriding the default sample count, but all three HealthBench cases hard-codenum_examples=10 if debug_mode else num_examples.As a result,
python -m gpt_oss.evals --debug --examples 2 --eval healthbench ...still evaluates 10 examples. GPQA and AIME honor the same explicit override.Impact
Debug runs can unexpectedly perform five times the requested model/grader work, which is particularly costly for HealthBench because each sample is graded across multiple rubric items.
Proposed resolution
Use the explicit
--examplesvalue whenever provided. Only fall back to HealthBench's debug default of 10 when debug mode is active and the user did not specify a count. Add focused coverage for explicit, debug-default, and full-eval cases.