feat:nao test: assert the agent performed a specific action

[justin212407]

Description

Today, nao test can only verify final agent output by comparing the generated SQL's result against a reference sql dataframe. There's no way to assert on intermediate agent behavior, so a test case can't check things like "did the agent ask a clarifying question before running SQL" or "did the agent call a specific tool."

This adds an extensible assertions mechanism to nao test that checks the agent's recorded tool-call trace independently of (and combinable with) the existing dataframe-based SQL verification.

Changes made

• New Assertion framework (cli/nao_core/commands/test/assertions.py):
  introduces a ToolCallAssertion type that checks whether a named tool was invoked, with optional args subset-matching (recursive for nested dicts) and min_count. parse_assertions() validates the assertions list from a test YAML and raises AssertionConfigError on malformed entries (missing type, unknown fields, invalid min_count, etc.). evaluate_assertions() runs all assertions against the tool-call trace and returns a combined pass/fail with a human-readable message.
• TestCase (cli/nao_core/commands/test/case.py) now carries an assertions: list[Assertion] field, parsed from the test YAML alongside the existing sql field, which is now optional (sql: str | None = None) so assertion-only tests don't need reference SQL.
• runner.py: assertions are evaluated alongside SQL verification when both are present (passed = assertions_passed and data_passed), and a new code path handles assertion-only test cases (no sql) by reporting pass/fail on the assertions alone.
• Backend support: apps/backend/src/routes/test.ts makes the sql field in the test request body optional (defaults to ''), and apps/backend/src/agents/tools/index.ts keeps the clarification tool available in testMode (previously stripped) so nao test can assert on clarifying follow-up questions and other intermediate actions in the recorded tool-call trace.
• Docs: cli/README.md documents the new assertions YAML syntax with a tool_call example (asserting a clarification follow-up), and explains that tool_call assertions support tool, args, and min_count, and that SQL verification and assertions can be combined.

Screenshots

Example test case

name: ambiguous_revenue_period
prompt: What was the revenue?
assertions:
  - type: tool_call
    tool: clarification

Validation

• Added unit tests: tests/nao_core/commands/test_assertions.py, test_case.py, test_runner.py covering assertion parsing/validation, TestCase YAML loading with assertions, and runner behavior for combined SQL + assertion checks and assertion-only test cases.
• apps/backend/src/routes/test.ts — verified optional sql field doesn't break existing SQL-based test payloads.
• Manual check: an assertion-only test YAML (no sql) runs end-to-end through nao test and reports pass/fail based solely on the tool-call trace.

[github-actions]

This PR was auto-closed. Only contributors approved with lgtm can open PRs. Open an issue first.

Maintainers review auto-closed issues daily. Issues that do not meet the quality bar in CONTRIBUTING.md will not be reopened or receive a reply.

If a maintainer replies lgtmi, your future issues will stay open. If a maintainer replies lgtm, your future issues and PRs will stay open.

See CONTRIBUTING.md.

[justin212407]

@Bl3f ptal and lmk if there are any changes that you might want me to make

[socallmebertille]

Hi, thanks for your contribution!

You should remove the !testMode condition into apps/backend/src/components/ai/system-prompt.tsx (l. 109) if you want to make available the clarification tool in test mode.

[socallmebertille]

Maybe we should avoid cluttering this file with this comment, this is basically information you've already included in the PR description, there is no need to repeat it here again

Suggested change

	// Keep clarification available in testMode so `nao test` can assert follow-up
	// questions and other intermediate actions on the recorded tool-call trace.

[socallmebertille]

You can delete this part as long as you remove the three call-site props in the same edit.

Suggested change

	/**
	* @deprecated No longer strips tools. Kept so existing callers that pass
	* `testMode` continue to typecheck. Clarification stays available so
	* `nao test` can assert intermediate actions (e.g. follow-up questions).
	*/
	testMode?: boolean;

[socallmebertille]

I think this file is unnecessary

[justin212407]

I have removed the !testMode condition so clarification guidance is available during tests, removed the obsolete testMode option and its call sites, the redundant comments, and deleted the standalone test_assertions.py file.n Also updated test-mode stopping so runs terminate when clarification is called.

[cubic-dev-ai]

All reported issues were addressed across 14 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 6 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="cli/nao_core/config/databases/duckdb.py">

<violation number="1" location="cli/nao_core/config/databases/duckdb.py:24">
P2: When a DuckDB schema or table name contains an apostrophe, this SQL becomes invalid, so `description()` and `_fetch_column_descriptions()` silently omit comments. Escape these values as SQL string literals or bind them before constructing both metadata queries.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When a DuckDB schema or table name contains an apostrophe, this SQL becomes invalid, so description() and _fetch_column_descriptions() silently omit comments. Escape these values as SQL string literals or bind them before constructing both metadata queries.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cli/nao_core/config/databases/duckdb.py, line 24:

<comment>When a DuckDB schema or table name contains an apostrophe, this SQL becomes invalid, so `description()` and `_fetch_column_descriptions()` silently omit comments. Escape these values as SQL string literals or bind them before constructing both metadata queries.</comment>

<file context>
@@ -15,6 +15,42 @@
+        try:
+            query = (
+                "SELECT comment FROM duckdb_tables() "
+                f"WHERE schema_name = '{self._schema}' AND table_name = '{self._table_name}'"
+            )
+            row = self._fetchone(self._conn.raw_sql(query))  # type: ignore[union-attr]
</file context>