Summary
The AIME and non-debug GPQA constructors guard subset selection with if num_examples:. Because zero is falsey, num_examples=0 silently skips sampling and evaluates the full dataset instead of zero examples.
This is inconsistent with the parameter contract (int | None, where None represents no restriction) and with HealthBench, which can represent an empty sample set.
Impact
A zero-example smoke/debug configuration can unexpectedly launch a full evaluation, causing substantial unnecessary inference work instead of returning an empty result.
Proposed resolution
Distinguish None from zero explicitly. Validate that sample counts are non-negative, use an empty example list for zero, and retain the existing subset sampling behavior for positive counts. Add regression coverage for zero and negative counts.
Summary
The AIME and non-debug GPQA constructors guard subset selection with
if num_examples:. Because zero is falsey,num_examples=0silently skips sampling and evaluates the full dataset instead of zero examples.This is inconsistent with the parameter contract (
int | None, whereNonerepresents no restriction) and with HealthBench, which can represent an empty sample set.Impact
A zero-example smoke/debug configuration can unexpectedly launch a full evaluation, causing substantial unnecessary inference work instead of returning an empty result.
Proposed resolution
Distinguish
Nonefrom zero explicitly. Validate that sample counts are non-negative, use an empty example list for zero, and retain the existing subset sampling behavior for positive counts. Add regression coverage for zero and negative counts.