Honor explicit eval sample counts and repeat selection

[sylvesterkaczmarek]

Summary

Make the eval CLI's documented --examples override apply consistently to sample counts, GPQA debug selection, and repeat counts.

Three CLI paths were inconsistent:

• HealthBench debug runs hard-coded 10 examples even when the user supplied another positive count;
• non-debug AIME and GPQA kept n_repeats=8 when an explicit subset was requested, which violates those constructors' n_repeats == 1 subset requirement;
• GPQA debug mode always selected one fixed Espresso example and ignored an explicit positive sample count.

Fixes #292.
Fixes #298.
Fixes #299.

Fix

• a positive explicit sample count wins over the eval-specific debug default;
• --examples 0 means full/unlimited evaluation outside debug mode and uses the eval-specific debug default in debug mode;
• HealthBench therefore retains its debug default of 10 when no count or zero is supplied;
• positive explicit AIME/GPQA subsets use one repeat;
• non-debug --examples 0 retains eight repeats;
• debug AIME/GPQA runs use one repeat;
• the fixed GPQA debug example is used only when no explicit sample count is supplied;
• full non-debug AIME/GPQA evaluations retain eight repeats;
• keep dependency-free resolution helpers in lightweight cli_utils.py.

Regression coverage

Adds lightweight tests for explicit, debug-default, full-run, and zero-example sample/repeat selection, plus GPQA fixed-debug versus explicit-subset selection.

Datasets, scoring, sampler configuration, and full-eval defaults are otherwise unchanged.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f86f32943

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7c02feaae8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e7fb5b0dd9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Note

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[sylvesterkaczmarek]

Addressed the remaining zero-example debug case: --examples 0 now maps to the eval-specific debug default in debug mode and to full/unlimited evaluation outside debug mode, with regression coverage for both paths. @codex review

[chatgpt-codex-connector]

Note

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.