feat(qa): publish actionable eval diagnostics

[cbullinger]

Summary

• publish sanitized per-case QA evaluation diagnostics instead of verdict-only artifacts
• isolate each skill run to its exact Inspect log and validate reports before upload
• add representative ASP trajectory checks and allow required Atlas project discovery

Security

Raw Inspect logs are not uploaded. Reports pass through the private harness public-field allowlist and secret scrubber before reaching this public repository.

[Copilot]

Pull request overview

This PR upgrades the QA eval GitHub Actions workflow to publish sanitized, per-sample diagnostic reports (instead of verdict-only artifacts) while ensuring each skill run is isolated to a single, exact Inspect log that is then gated and exported safely.

Changes:

• Update qa-eval.yml to run each skill into its own log directory, gate the single resulting .eval, export a sanitized per-skill diagnostics report, and validate report shape before upload.
• Add a Node test that asserts key safety/behavior invariants in the workflow (log isolation, exact log gating, report export/validation, and jq argument formatting).
• Add representative trajectory_checks to ASP eval cases and relax wording in ASP metadata to allow optional Atlas project discovery (atlas-list-projects) before the first ASP tool call.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
testing/qa-eval-workflow.test.mjs	Adds regression tests to enforce workflow invariants (log isolation, exact gating, report validation, jq arg style).
testing/package.json	Includes the new workflow test in the test suite.
testing/mongodb-atlas-stream-processing/evals/metadata/14-validate-connections-before-deploy.json	Updates assertion description to allow optional atlas-list-projects discovery before ASP tool calls.
testing/mongodb-atlas-stream-processing/evals/metadata/07-debug-failed-processor.json	Updates assertion descriptions to allow optional atlas-list-projects discovery and clarifies ordering wording.
testing/mongodb-atlas-stream-processing/evals/evals.json	Adds representative trajectory_checks for several ASP eval cases.
.github/workflows/qa-eval.yml	Implements per-skill log isolation, exact .eval gating, sanitized report export + strict schema/allowlist validation, and uploads only qa-eval-report-*.json.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

testing/qa-eval-workflow.test.mjs:23

• The regexes intended to assert the jq filters include a literal | currently use an unescaped |, which acts as alternation in JS regex and can pass even if the workflow no longer contains the | type == ... portion. This weakens the test and could let regressions slip through.

  assert.match(workflow, /uv run python public_report\.py/);
  assert.match(workflow, /\(\.samples \| type == "array"\)/);
  assert.match(workflow, /\(\.validTotal \| type == "number"\)/);

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/workflows/qa-eval.yml:350

• jq -c '. + {...}' "$DETAIL_REPORT" > "$REPORT_TMP" (and the subsequent jq -e validation) will cause the entire workflow step to exit under set -e if public_report.py produces malformed JSON or a non-object (e.g., truncated output). That undermines the intended behavior of treating report/export issues as INCONCLUSIVE and still finalizing per-skill output.

            jq -c \

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

testing/qa-eval-workflow.test.mjs:23

• The regexes for the jq validation checks use an unescaped |, which in JavaScript regex means alternation. This makes the assertions much weaker than intended (they can pass even if the | pipe is missing). Escape the pipe and allow flexible whitespace so the test actually verifies the jq filter structure.

  assert.match(workflow, /\(\.samples \| type == "array"\)/);
  assert.match(workflow, /\(\.validTotal \| type == "number"\)/);

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/workflows/qa-eval.yml:293

• The loop over SKILLS uses command substitution and word-splitting (for SKILL in $(...)), which will break if a skill name ever contains whitespace/newlines (e.g., it would treat one skill as multiple entries and produce incorrect log/report paths). Since SKILLS is JSON, iterate with jq into an array and loop over the array elements instead.

          for SKILL in $(echo "$SKILLS" | jq -r '.[]'); do
            echo "== QA eval: $SKILL (model $PANEL_MODEL, k=1) =="
            LOG_DIR="$RUNNER_TEMP/inspect-logs/$SKILL"
            DETAIL_REPORT="$RUNNER_TEMP/reports/.qa-eval-detail-$SKILL.json"
            REPORT="$RUNNER_TEMP/reports/qa-eval-report-$SKILL.json"