feat(cli): Add pass@k and pass^k metrics to nao test KPIs

[justin212407]

Description

Closes: #1248

Adds pass@k and pass^k metrics to nao test, alongside the existing per-run pass/fail accuracy score. nao test can now run each test case k times (nao test --k 5), and reports pass@1 (default), pass@k, and pass^k both per test case and aggregated across the suite, in the CLI output and in the structured JSON results file.

Type of change

• Feature

Changes made

• Added TestConfig.k (default 1, ge=1) to nao_config.yaml's test: block, overridable with k / -k, following the same pattern as the existing threads option.
• Each (test case, model) pair now runs k times instead of once. Added TestRunResult.attempt (1-indexed) so a run is self-describing even after the threaded executor reorders results.
• Added pass_metrics_for_group() / summarize_pass_metrics() in summary.py, computing metrics empirically per (test, model) group since -k N runs exactly N attempts, not an oversampled pool:
  • pass@1 - mean of passed across the group's attempts
  • pass@k - 1.0 if any attempt passed, else 0.0
  • pass^k - 1.0 if all attempts passed, else 0.0
• Suite/model aggregates average per test case, not per raw attempt, so a test case run with k=5 counts once toward the aggregate.
• Result JSON gains a new top-level pass_metrics key ({k, aggregate, by_test}). Existing summary / by_model keys are
  unchanged in shape; ModelSummary gained additive k / pass_at_1 / pass_at_k / pass_hat_k fields.
• CLI table output shows pass@1 always; pass@k / pass^k columns only appear when k > 1, so default (k=1) output is unchanged.
• with_model_summaries() still loads older result files without k/pass_metrics, so nao test server keeps working on old data.

Test plan

• make lint passes (ty check, ruff check, import sort, ruff format --check)
• uv run pytest -q passes - 999 passed, 225 skipped (skips are a pre-existing test_mssql.py environment gap, unrelated to this change)
• Added unit tests for pass_metrics_for_group / summarize_pass_metrics (mixed pass/fail, all-pass, all-fail, k=1 equivalence, aggregation weighted by test case not attempt)
• Added tests for the runner's k-based fan-out, attempt population, and CLI/config override + rejection of invalid k
• Smoke-tested nao test --k 3 against example/: confirmed config loading and correct run-count calculation (1 test × 1 model × 3 attempts = 3 runs)
• Full live agent round-trip (nao test --k 3 against a running backend with a real LLM key) - not validated end to end in this environment; plumbing up to the network boundary is verified by the above

Manual testing steps:

cd cli
make lint
uv run pytest -q --ignore=tests/nao_core/commands/sync/integration/test_mssql.py

cd ../example
uv run --project ../cli nao test           # confirm k=1 default output unchanged
uv run --project ../cli nao test --k 3     # confirm pass@1 / pass@3 / pass^3 columns

Review guidance

Changed files: cli/nao_core/commands/test/runner.py, cli/nao_core/commands/test/summary.py, cli/nao_core/config/test/__init__.py, cli/README.md, example/nao_config.yaml, plus matching test files under
cli/tests/. No new dependencies, no changes outside cli/ (backend/frontend untouched).

The metric-definition choice is worth a close look: since --k N runs exactly N attempts (not an oversampled pool), pass@k/pass^k are computed as direct any/all over the group rather than the combinatorial unbiased-estimator formula from the Codex/HumanEval pass@k paper - that estimator only applies when n > k. summarize_pass_metrics() and pass_metrics_for_group() in summary.py are the two functions to check this logic in.

Checklist

• No changes to CSS/frontend (backend-agnostic CLI-only change).
• No print/debug statements left in.
• Existing tests updated where behavior changed; new tests added for new behavior.

Screenshots:

[github-actions]

This PR was auto-closed. Only contributors approved with lgtm can open PRs. Open an issue first.

Maintainers review auto-closed issues daily. Issues that do not meet the quality bar in CONTRIBUTING.md will not be reopened or receive a reply.

If a maintainer replies lgtmi, your future issues will stay open. If a maintainer replies lgtm, your future issues and PRs will stay open.

See CONTRIBUTING.md.

[justin212407]

@Bl3f Can you take a look and review this? Please lemme know if the changes follow the desired implementation. If not happy to follow up accordingly.

[socallmebertille]

Hi, thanks for your implementation !

Sorry for the lack of specification on this issue. Let me give you a preview of how we'd like the new metrics to be rendered in the terminal.

• First table (the one printed by nao test, one row per test) :
  • The green check in the Status column should only appear when all runs of that test passed (100% success)
  • Add a new column next to it showing the % of runs that passed for that test
• Second table (the summary) :
  • Instead of listing every run with its success %, show only a single totals row — i.e. the aggregate across all tests (like the "total" line of the first table)
  • Add an "Always Pass %" column to this totals row: the share of tests that passed on every run (100% success across all attempts)

Something similar to :

Let me know if anything is unclear!

[cubic-dev-ai]

1 issue found across 7 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="cli/nao_core/commands/test/runner.py">

<violation number="1" location="cli/nao_core/commands/test/runner.py:323">
P2: The PR states the CLI table shows pass@k and pass^k columns when k>1, but the code never renders pass@k (any attempt passed). The run table exposes only pass@1 ("Success %") and pass^k ("Status"/"Always Pass %"), so pass@k is available only in the saved JSON. If pass@k should be visible in the table output as described, add a column driven by metrics["pass_at_k"] (and pass_hat_k) rather than only pass_at_1.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The PR states the CLI table shows pass@k and pass^k columns when k>1, but the code never renders pass@k (any attempt passed). The run table exposes only pass@1 ("Success %") and pass^k ("Status"/"Always Pass %"), so pass@k is available only in the saved JSON. If pass@k should be visible in the table output as described, add a column driven by metrics["pass_at_k"] (and pass_hat_k) rather than only pass_at_1.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cli/nao_core/commands/test/runner.py, line 323:

<comment>The PR states the CLI table shows pass@k and pass^k columns when k>1, but the code never renders pass@k (any attempt passed). The run table exposes only pass@1 ("Success %") and pass^k ("Status"/"Always Pass %"), so pass@k is available only in the saved JSON. If pass@k should be visible in the table output as described, add a column driven by metrics["pass_at_k"] (and pass_hat_k) rather than only pass_at_1.</comment>

<file context>
@@ -295,59 +302,62 @@ def save_results(results: list[TestRunResult], output_dir: Path) -> Path:
+                "Test": name,
+                "Model": model,
+                "Status": status_icon(float(metrics["pass_hat_k"]) == 1.0),
+                "Success %": format_pass_fraction(float(metrics["pass_at_1"])),
+                "Tokens": sum(run.get("tokens") or 0 for run in group),
+                "Cost": sum(run.get("cost") or 0 for run in group),
</file context>

[justin212407]

@socallmebertille I have updated the pr with the intended layout we wanted for this issue in the recent commit, here is the mockup of how the outputs look like as of now: