refactor(test-classifier): own the skill in-repo + hide parse markers on local runs (#60)

* refactor(test-classifier): own the skill in-repo; hide parse markers locally

Two local-ergonomics + provenance fixes.

1. Drop the agent-skills fetch — the bundle owns its skill.
   The in-repo .skills/test-classifier/SKILL.md is already the canonical, and is
   in fact NEWER than agent-skills@v0.1.0 (the fetch was overwriting the better
   in-repo copy with a staler one each run). Remove scripts/fetch-skills.sh, the
   .skills-vendor/ indirection, the dispatcher's vendor-preference branch, the
   "Vendor SKILL.md" steps in both workflows + the Jenkinsfile/README, and the
   .gitignore entry. SKILL_PATH_CANONICAL now always points at the in-repo copy.

2. Strip machine-only markers from terminal output.
   On a local run the raw <!-- AI_CLASSIFIER_JSON_* --> block and the
   <<<AI_REVIEW_RESULT:…>>> sentinel leaked into the terminal — meaningful only
   to the parser / PR-comment poster, noise to a human. strip_machine_markers()
   removes them from what's PRINTED; parse_result and extract_classifier_json
   still read the untouched classifier_output, so CLASSIFIED/NO_ACTION detection
   and --post-comment/--json-only are unchanged.

Verified: strip leaves the human report intact (markers gone, blank lines
collapsed); both scripts pass bash -n; no fetch-skills/agent-skills/skills-vendor
references remain anywhere in testing/ or .github/.

* feat(test-classifier): actionable terminal summary after the report

The markdown report is formatted for the PR comment; locally a developer had to
read the whole wall to find the verdict and what to do about it. Add a compact,
colored terminal summary printed after the report on a human run: one line per
failing test — VERDICT, file:line, confidence — plus the one-line action
derived from the verdict (→ Fix the TEST / Fix the CODE / Re-run + deflake /
Fix the ENV), mirroring SKILL.md's taxonomy table.

The skill is diagnostic-only by design (it names the side to fix, never writes
a patch), so the summary points; it doesn't auto-fix. Rendered from the JSON the
dispatcher already extracts, printed to STDERR (never touches the parsed stdout),
gated to non---json-only runs, and best-effort (skips silently without python3
or a parseable block). Verified end-to-end against a real classified diff.

Author: fg-nava

.github/workflows/test-classifier.yml

@@ -196,11 +196,6 @@ jobs:
           cp -R .classifier-bundle/testing "$GITHUB_WORKSPACE/testing"
           chmod +x testing/classifier/.skills/_lib/ai-classifier-dispatch.sh
           chmod +x testing/classifier/.skills/test-classifier/scripts/test-classifier-dispatcher.sh
-          chmod +x testing/classifier/scripts/fetch-skills.sh
-
-      - name: Vendor SKILL.md from agent-skills
-        working-directory: testing/classifier
-        run: scripts/fetch-skills.sh
 
       # ── Bedrock auth (provider=bedrock only) ────────────────────────────────
       # Exchange this run's GitHub OIDC token for short-lived STS credentials by

testing/classifier/.github/workflows/ai-test-classifier.yml

@@ -115,11 +115,6 @@ jobs:
         run: |
           chmod +x .skills/_lib/ai-classifier-dispatch.sh
           chmod +x .skills/test-classifier/scripts/test-classifier-dispatcher.sh
-          chmod +x scripts/fetch-skills.sh
-
-      - name: Vendor canonical skill from agent-skills
-        working-directory: testing/classifier
-        run: scripts/fetch-skills.sh
 
       # Bedrock auth (AI_PROVIDER=bedrock only): OIDC → short-lived STS creds for
       # the caller-owned IAM role. Claude Code picks them up via the AWS SDK

testing/classifier/.gitignore

@@ -13,9 +13,6 @@
 .codex/skills/
 .github/copilot/skills/
 
-# Skills vendored from agent-skills by scripts/fetch-skills.sh.
-.skills-vendor/
-
 # ── Classifier output ──────────────────────────────────────────────────────
 # The workflow writes the per-run classification (the CI artifact / audit trail)
 # here before uploading it as a CI artifact. Not source-of-truth — don't commit.

testing/classifier/.skills/test-classifier/scripts/test-classifier-dispatcher.sh

@@ -61,14 +61,10 @@ fi
 # ── Skill identity ──────────────────────────────────────────────────────────
 SKILL_NAME="test-classifier"
 SKILL_HUMAN_NAME="AI Test Classifier (application-bug / test-bug / flaky / environment)"
-# Skill text: prefer the copy vendored from agent-skills by fetch-skills.sh;
-# fall back to the in-repo .skills/ copy when the vendor dir is absent.
+# Skill text lives in-repo and is the canonical source of truth (this bundle owns
+# its skill — no external fetch).
 BUNDLE_ROOT="$(cd "${SKILLS_ROOT}/.." && pwd)"   # .skills → classifier (bundle root)
-if [[ -f "${BUNDLE_ROOT}/.skills-vendor/test-classifier/SKILL.md" ]]; then
-  SKILL_PATH_CANONICAL=".skills-vendor/test-classifier/SKILL.md"
-else
-  SKILL_PATH_CANONICAL=".skills/test-classifier/SKILL.md"
-fi
+SKILL_PATH_CANONICAL=".skills/test-classifier/SKILL.md"
 
 # ── Classifier-specific arg parsing ────────────────────────────────────────
 # We intercept our own flags first, then pass the remainder to the shared
@@ -254,9 +250,7 @@ in this repository at:
 
   ${SKILL_PATH_CANONICAL}
 
-(The canonical skill text is published in navapbc/agent-skills and vendored here
-by scripts/fetch-skills.sh; when the vendored copy is absent the in-repo
-.skills/ copy is used. Either way, follow the file at the path above.)
+Follow the file at the path above.
 
 Classify the failing tests for the change under test. The exact git range is in
 the AI_REVIEW_DIFF_RANGE env var (base→HEAD normally; base→index, i.e. committed
@@ -333,6 +327,78 @@ extract_classifier_json() {
   '
 }
 
+# Strip the machine-only markers from the AI output for HUMAN display: the
+# JSON block (BEGIN→END inclusive) and the <<<AI_REVIEW_RESULT:…>>> line. These
+# exist for parsing / PR-comment posting and are noise in a local terminal run.
+# Parsing always uses the untouched ${classifier_output}; this only affects what
+# is printed. Trailing blank lines left by the removal are collapsed.
+strip_machine_markers() {
+  local input="$1"
+  printf '%s\n' "${input}" | awk '
+    /<!-- AI_CLASSIFIER_JSON_BEGIN -->/ { skip=1; next }
+    /<!-- AI_CLASSIFIER_JSON_END -->/   { skip=0; next }
+    skip { next }
+    /^<<<AI_REVIEW_RESULT:[A-Z_]+>>>[[:space:]]*$/ { next }
+    { print }
+  ' | awk 'NF { blanks=0; print; next } { blanks++; if (blanks<=1) print }'
+}
+
+# Compact, terminal-native summary of the verdicts — the actionable punchline a
+# local developer needs, lifted out of the long markdown report. One line per
+# failing test: VERDICT, file:line, confidence, and the one-line "what to do"
+# action derived from the verdict (mirrors SKILL.md's taxonomy table). The skill
+# is diagnostic-only, so this points at the side to fix; it never proposes a
+# patch. Printed to STDERR so it never touches the parsed stdout. Best-effort:
+# if python3 or the JSON is unavailable, we silently skip it (the full report
+# above still stands).
+render_terminal_summary() {
+  local classifier_json="$1"
+  command -v python3 &>/dev/null || return 0
+  [[ -n "${classifier_json}" ]] || return 0
+
+  AI_C_RED="${AI_C_RED}" AI_C_YELLOW="${AI_C_YELLOW}" AI_C_GREEN="${AI_C_GREEN}" \
+  AI_C_BLUE="${AI_C_BLUE}" AI_C_BOLD="${AI_C_BOLD}" AI_C_RESET="${AI_C_RESET}" \
+  python3 - "${classifier_json}" >&2 <<'PY'
+import json, os, sys
+
+RED=os.environ.get("AI_C_RED",""); YEL=os.environ.get("AI_C_YELLOW","")
+GRN=os.environ.get("AI_C_GREEN",""); BLU=os.environ.get("AI_C_BLUE","")
+BOLD=os.environ.get("AI_C_BOLD",""); RST=os.environ.get("AI_C_RESET","")
+
+try:
+    data = json.loads(sys.argv[1])
+except Exception:
+    sys.exit(0)
+
+cls = data.get("classifications") or []
+if not cls:
+    sys.exit(0)
+
+# verdict → (color, one-line action). Mirrors SKILL.md's "What it means / action".
+ACTION = {
+    "APPLICATION_BUG":  (RED, "Fix the CODE — the app regressed; the test caught a real defect."),
+    "TEST_BUG":         (YEL, "Fix the TEST — the app is correct; the assertion is stale."),
+    "FLAKY_FAILURE":    (BLU, "Re-run to confirm, then deflake — not a code/test-logic patch."),
+    "ENVIRONMENT_ISSUE":(BLU, "Fix the ENV / re-run — neither the app nor the test is at fault."),
+}
+
+mode = data.get("mode","")
+print(f"\n{BOLD}─── Test Classifier — {len(cls)} classified" + (f" ({mode})" if mode else "") + f" ───{RST}")
+for c in cls:
+    verdict = c.get("verdict","") or "UNKNOWN"
+    color, action = ACTION.get(verdict, (RST, ""))
+    loc = c.get("path","")
+    line = c.get("line")
+    if loc and line not in (None, ""):
+        loc = f"{loc}:{line}"
+    conf = c.get("confidence","")
+    conf_s = f"  ({conf})" if conf else ""
+    print(f"  {color}{verdict:<17}{RST} {loc}{conf_s}")
+    if action:
+        print(f"    {color}→{RST} {action}")
+PY
+}
+
 # Render the ONE PR comment body from the classifier JSON, including the
 # mandatory 👍/👎 ask. We use python3 because pure-bash JSON handling is brittle.
 render_pr_comment_body() {
@@ -554,7 +620,9 @@ test_classifier::run() {
   if (( JSON_ONLY == 1 )); then
     extract_classifier_json "${classifier_output}"
   else
-    printf '%s\n' "${classifier_output}"
+    # Human-facing terminal output: drop the machine-only markers (JSON block +
+    # result sentinel). Parsing below still uses the untouched classifier_output.
+    strip_machine_markers "${classifier_output}"
     ai_review::log "────────────────────────────────────────────────────────────"
   fi
 
@@ -569,6 +637,11 @@ test_classifier::run() {
   case "${result}" in
     CLASSIFIED)
       ai_review::info "Classifier result: CLASSIFIED"
+      # Terminal-native actionable summary (verdict + file:line + what-to-do),
+      # for a human run only. Skipped under --json-only (machine consumption).
+      if (( JSON_ONLY != 1 )); then
+        render_terminal_summary "$(extract_classifier_json "${classifier_output}")"
+      fi
       ;;
     NO_ACTION)
       ai_review::ok "Classifier result: NO_ACTION (nothing failed)."

testing/classifier/README.md

@@ -33,8 +33,8 @@ reuses its conventions exactly:
   `copilot`. See [`docs/BEDROCK.md`](./docs/BEDROCK.md).
 - The result-marker contract: the AI ends with one `<<<AI_REVIEW_RESULT:...>>>`
   marker the dispatcher parses.
-- The skill text vendored from `navapbc/agent-skills` by
-  `scripts/fetch-skills.sh`, with the in-repo `.skills/` copy as a fallback.
+- The skill text at `.skills/test-classifier/SKILL.md` — canonical and in-repo
+  (this bundle owns its skill; no external fetch).
 - A GitHub Actions workflow that ships **disabled**, **advisory/non-blocking**
   by default, with a `--gate` flag to opt into build-failing later.
 - Same bash style: `set -euo pipefail`, `ai_review::`-namespaced helpers,

testing/classifier/docs/LOCAL_TEST_CLASSIFIER.md

@@ -118,6 +118,12 @@ AI_RUN_SUITE=1 test-classifier        # run the suite locally and triage REAL fa
   and never posts a PR comment. Posting (with the mandatory 👍/👎 ask that feeds
   the metrics loop) happens on the PR run — Path B, or `--post-comment` against a
   real PR. See [`SETUP.md`](./SETUP.md).
+- **The actionable summary.** After the full report, the dispatcher prints a
+  compact, colored summary — one line per failing test with its verdict,
+  `file:line`, confidence, and the one-line *what to do* (`→ Fix the TEST`,
+  `→ Fix the CODE`, `→ Re-run / deflake`, `→ Fix the ENV`). That's the punchline:
+  the classifier is **diagnostic** — it tells you which side to fix, never writes
+  the patch. Read the summary, then make the change yourself.
 - **INFERRED vs OBSERVED.** Without `AI_RUN_SUITE=1` the agent predicts failures
   from the diff (INFERRED) and never touches your toolchain. With it, the agent
   runs your suite (OBSERVED) — the only mode in which `FLAKY_FAILURE` /

testing/classifier/docs/PLAYBOOK.md

@@ -54,8 +54,8 @@ It mirrors the security workstream exactly in its plumbing:
   `set -euo pipefail`, and color helpers suppressed in CI.
 - The **result-marker contract**: the AI ends its output with exactly one
   `<<<AI_REVIEW_RESULT:...>>>` marker that the dispatcher parses.
-- The **skill text** vendored from `navapbc/agent-skills` by
-  `scripts/fetch-skills.sh`, with the in-repo `.skills/` copy as a fallback.
+- The **skill text** at `.skills/test-classifier/SKILL.md` — canonical and
+  in-repo (this bundle owns its skill; no external fetch).
 - A GitHub Actions workflow that ships **disabled** (`on: workflow_dispatch`
   only), is **non-blocking/advisory by default**, and flips to build-failing
   only behind an explicit `--gate` flag.

testing/classifier/jenkins/Jenkinsfile

@@ -92,11 +92,6 @@ pipeline {
           esac
         '''
 
-        // Vendor the skill from agent-skills before invoking the dispatcher.
-        dir('testing/classifier') {
-          sh 'chmod +x scripts/fetch-skills.sh && scripts/fetch-skills.sh'
-        }
-
         // Ensure the scripts are executable (a fresh checkout may drop the bit),
         // then hand off to the CI adapter — it normalizes CHANGE_ID/CHANGE_TARGET
         // and execs the dispatcher with --pr / --against / --post-comment.

testing/classifier/jenkins/README.md

@@ -103,9 +103,6 @@ stage('AI test classifier') {
       sh '''
         set -euo pipefail
         npm install -g @anthropic-ai/claude-code     # or @openai/codex
-        # Run fetch-skills from the classifier dir in a subshell so the cd
-        # doesn't leak — don't rely on $WORKSPACE being stable across steps.
-        ( cd testing/classifier && chmod +x scripts/fetch-skills.sh && scripts/fetch-skills.sh )
         chmod +x testing/classifier/.skills/_lib/ai-classifier-dispatch.sh \\
                  testing/classifier/.skills/test-classifier/scripts/test-classifier-dispatcher.sh \\
                  testing/classifier/jenkins/ci-adapter.sh