feat(test-classifier): make OBSERVED the default; add --no-run-suite opt-out (#77)

Running the suite when one exists is the high-signal path, so it is now the
default for every run (local, CI, Jenkins) rather than CI-only. Flip the
AI_RUN_SUITE default 0 -> 1 in the lib and in the dispatcher's prompt branch.
The skill still falls back to INFERRED on its own when there is no suite, the
toolchain is missing, or it would apply real infra with no teardown.

OBSERVED executes branch code, so add an easy opt-out for untrusted diffs:
the new --no-run-suite flag (exports AI_RUN_SUITE=0 before the lib loads) and
the existing AI_RUN_SUITE=0 env var. Reframe SKILL.md, SETUP.md, and the local
docs around 'OBSERVED by default, opt out for untrusted code'. CI/Jenkins keep
their explicit AI_RUN_SUITE=1 (now redundant but self-documenting).

Author: fg-nava

testing/classifier/.github/workflows/ai-test-classifier.yml

@@ -169,7 +169,8 @@ jobs:
           AI_REVIEW_TOOL: ${{ vars.AI_REVIEW_TOOL }}
           CI: 'true'
           # Grant the agent execution so it runs the repo's suite for OBSERVED
-          # verdicts (safe on the ephemeral runner; read-only by default locally).
+          # verdicts (safe on the ephemeral runner). OBSERVED is the default;
+          # set explicitly here so CI's behavior is pinned and self-documenting.
           AI_RUN_SUITE: '1'
           # Pass through whichever API key matches the chosen tool. Each secret
           # is set only if the team uses that tool — unused ones stay blank.

testing/classifier/.skills/_lib/ai-classifier-dispatch.sh

@@ -53,13 +53,18 @@ fi
 _AI_CLASSIFIER_DISPATCH_LOADED=1
 
 # ── Suite-running mode ──────────────────────────────────────────────────────
-# By default the classifier is read-only: it triages whatever failing-test
-# signal already exists and never executes the repo's suite. When AI_RUN_SUITE=1
-# (the CI workflows set it) the agent is granted execution so it can locate,
-# install, and RUN the repo's tests, then classify the OBSERVED failures. This
-# gate keeps a local Path-B run safe — a developer's machine won't auto-install
-# deps or run tests unless they opt in with AI_RUN_SUITE=1.
-AI_RUN_SUITE="${AI_RUN_SUITE:-0}"
+# By default the classifier is OBSERVED: the agent locates, installs, and RUNS
+# the repo's suite, then classifies the failures it actually observes. Running
+# the suite when one exists is the high-signal path, so it is the default for
+# every run (local, CI, Jenkins). The skill still falls back to INFERRED on its
+# own when there is no suite to run, the toolchain is missing, or it would apply
+# real infrastructure with no guaranteed teardown.
+#
+# OBSERVED executes the change's code on the current machine. That is fine on a
+# branch you trust (your own work, CI's checkout). To triage an UNTRUSTED diff
+# without executing it, opt out with AI_RUN_SUITE=0 (or --no-run-suite) for a
+# read-only, INFERRED-only pass.
+AI_RUN_SUITE="${AI_RUN_SUITE:-1}"
 
 # Bounds for the agent loop when it is running the suite, so a runaway
 # install/test cycle can't hang the job. The timeout wraps the whole CLI call;

testing/classifier/.skills/test-classifier/SKILL.md

@@ -110,11 +110,12 @@ out of current scope.
 The classifier needs to know *what failed and why it says it failed*. There are
 two ways to get that signal, and which one you use determines the result `mode`:
 
-### OBSERVED — run the suite yourself (when `AI_RUN_SUITE=1`)
+### OBSERVED — run the suite yourself (the default, unless `AI_RUN_SUITE=0`)
 
-When the environment variable `AI_RUN_SUITE=1` is set (the CI workflows set it,
-and you have been granted shell execution), **locate and run the repo's test
-suite yourself**, then classify the failures you actually observe:
+This is the default mode. Unless `AI_RUN_SUITE=0` is set (the `--no-run-suite`
+opt-out, for triaging an untrusted diff without executing it), you have been
+granted shell execution: **locate and run the repo's test suite yourself**, then
+classify the failures you actually observe:
 
 1. **Locate the test command.** Inspect the checked-out repo: `package.json`
    `scripts.test`, a `Makefile` `test:` target, `pytest.ini`/`tox.ini`/
@@ -165,17 +166,18 @@ suite yourself**, then classify the failures you actually observe:
 
 ### INFERRED — predict from the diff (fallback)
 
-If `AI_RUN_SUITE` is not set, **or** you cannot locate / install / run the suite
-(no test command found, missing toolchain, the suite needs services like a
-database, it times out, **or it would apply real infrastructure with no
-guaranteed teardown** — see the IaC ladder above), do **not** fabricate a run.
+If `AI_RUN_SUITE=0` is set (the explicit `--no-run-suite` opt-out), **or** you
+cannot locate / install / run the suite (no test command found, missing
+toolchain, the suite needs services like a database, it times out, **or it would
+apply real infrastructure with no guaranteed teardown** — see the IaC ladder
+above), do **not** fabricate a run.
 Instead reason
 statically over the diff (Step 2) and PREDICT which tests the change would cause
 to fail and why. Then:
 
 - Set the result `mode` to `"INFERRED"`.
 - State the reason you fell back in the `summary` (e.g. "no test script found",
-  "deps failed to install", "suite timed out", "AI_RUN_SUITE not set") so the
+  "deps failed to install", "suite timed out", "AI_RUN_SUITE=0 set") so the
   reviewer sees why this is a prediction, not an observation.
 - Prefer lower confidence, and remember `FLAKY_FAILURE`/`ENVIRONMENT_ISSUE` are
   generally NOT determinable from a diff alone — only assert them with explicit

testing/classifier/.skills/test-classifier/scripts/test-classifier-dispatcher.sh

@@ -97,6 +97,10 @@ SKILL_PATH_CANONICAL="${SKILLS_ROOT}/test-classifier/SKILL.md"
 #                         and skips the prompt + row.
 #   --gate                Exit 1 if the result is CLASSIFIED (CI-blocking mode)
 #   --json-only           Print only the JSON block (machine consumption)
+#   --no-run-suite        Read-only INFERRED pass: do NOT run the repo's suite
+#                         (predict from the diff). The default is OBSERVED (run
+#                         the suite); use this to triage an UNTRUSTED diff
+#                         without executing its code. Same as AI_RUN_SUITE=0.
 #
 # All other flags (--dry-run, --no-block, --against, --unpushed) fall through to
 # the lib. --unpushed classifies the local committed+staged diff with NO PR —
@@ -142,6 +146,12 @@ while [[ $# -gt 0 ]]; do
       POST_COMMENT=1
       shift
       ;;
+    --no-run-suite)
+      # Opt out of the OBSERVED default: read-only, predict from the diff
+      # (INFERRED). Exported so the sourced lib and the agent see it.
+      export AI_RUN_SUITE=0
+      shift
+      ;;
     --gate)
       GATE_MODE=1
       shift
@@ -370,10 +380,12 @@ ai_review::discover_pr_context() {
 # We instruct the AI to emit BOTH a human-readable report AND a fenced JSON
 # block. The dispatcher extracts the JSON block to render the PR comment.
 #
-# Step 1 (the failing-test signal) is mode-dependent: with AI_RUN_SUITE=1 the
-# agent has shell execution and must locate + run the suite (OBSERVED); otherwise
-# it predicts from the diff (INFERRED). See SKILL.md Step 1 for the full procedure.
-if [[ "${AI_RUN_SUITE:-0}" == "1" ]]; then
+# Step 1 (the failing-test signal) is mode-dependent: with AI_RUN_SUITE=1 (the
+# default) the agent has shell execution and must locate + run the suite
+# (OBSERVED); with AI_RUN_SUITE=0 (--no-run-suite) it predicts from the diff
+# (INFERRED). See SKILL.md Step 1 for the full procedure. This default must match
+# the lib's (ai-classifier-dispatch.sh) — both default to 1 (OBSERVED).
+if [[ "${AI_RUN_SUITE:-1}" == "1" ]]; then
   read -r -d '' SIGNAL_STEP <<'SIGNAL' || true
   1. Collect the failing-test signal in OBSERVED mode (AI_RUN_SUITE=1 — you have
      shell execution): LOCATE this repo's test command (package.json scripts,
@@ -497,14 +509,18 @@ test-classifier options (in addition to the shared options above):
                        "Was this helpful?" and appends one Testing Events row.
   --gate               Exit 1 when the result is CLASSIFIED (CI-blocking mode).
   --json-only          Print only the machine-readable JSON block.
+  --no-run-suite       Read-only INFERRED pass: predict from the diff instead of
+                       running the suite (the default). Use it on an UNTRUSTED
+                       diff you don't want to execute. Same as AI_RUN_SUITE=0.
 
 Defaults when no diff-range/PR selector is given:
   (no arguments)       → --unpushed (committed + staged; local, report-only).
   a bare ref           → --against <ref>, e.g.  test-classifier origin/main
 
 Environment:
-  AI_RUN_SUITE=1       Run the suite (OBSERVED) instead of inferring from the
-                       diff. Prefix it:  AI_RUN_SUITE=1 test-classifier --pr 42
+  AI_RUN_SUITE         Run the suite (OBSERVED, default) vs. infer from the diff.
+                       OBSERVED is the default; set AI_RUN_SUITE=0 (or pass
+                       --no-run-suite) for a read-only INFERRED pass.
   METRICSAI_WEBHOOK_URL / METRICSAI_WEBHOOK_KEY
                        Required only by --submit to append the Testing Events row.
 HELP_ADDENDUM

testing/classifier/docs/LOCAL_TEST_CLASSIFIER.md

@@ -27,23 +27,25 @@ and uses the same `--unpushed` scope rule. This is the ergonomics layer over
   fast feedback before pushing; it never blocks anything. By default it's
   **report-only** (prints to your terminal, posts nothing) — but you can opt in
   to posting against a real PR with `--pr N --post-comment` (see Usage).
-- **OBSERVED on demand.** By default a local run is INFERRED (it predicts from
-  the diff — read-only, never touches your machine's toolchain). Opt into
-  OBSERVED with `AI_RUN_SUITE=1` to have the agent actually locate, install, and
-  run your suite and triage the real failures — the same behavior CI uses.
+- **OBSERVED by default.** When your repo has a suite, a local run is OBSERVED:
+  the agent locates, installs, and runs your suite and triages the *real*
+  failures — the same behavior CI uses, and the high-signal path. It falls back
+  to INFERRED on its own when there's no suite to run. To force a read-only
+  INFERRED pass (predict from the diff, never touch your toolchain), pass
+  `--no-run-suite` or set `AI_RUN_SUITE=0`.
 
 > [!WARNING]
-> **OBSERVED (`AI_RUN_SUITE=1`) runs the change's code on your machine — there
-> is no sandbox.** The agent installs dependencies (running each package's
+> **OBSERVED (the default) runs the change's code on your machine — there is no
+> sandbox.** The agent installs dependencies (running each package's
 > install/postinstall scripts) and executes the test suite **directly in your
 > checkout**, with your shell, your environment, and your credentials. It also
 > leaves build artifacts behind (`node_modules/`, downloaded browsers, etc.).
 >
-> Only use `AI_RUN_SUITE=1` on a branch **you trust** — typically your own
-> work-in-progress. Do **not** point it at an untrusted or unreviewed PR
-> (e.g. `--pr N` for someone else's contribution): installing and running that
-> branch's code is arbitrary code execution on your laptop. For untrusted
-> changes, use the default **INFERRED** mode (omit `AI_RUN_SUITE`), which only
+> This is fine on a branch **you trust** — typically your own work-in-progress.
+> Do **not** run it against an untrusted or unreviewed PR (e.g. `--pr N` for
+> someone else's contribution): installing and running that branch's code is
+> arbitrary code execution on your laptop. For untrusted changes, opt out with
+> `--no-run-suite` (or `AI_RUN_SUITE=0`) for the **INFERRED** mode, which only
 > reasons over the diff and never executes anything, or let CI run OBSERVED in
 > its disposable runner.
 
@@ -102,24 +104,25 @@ It forwards args verbatim, and the dispatcher applies these defaults:
   (`--dry-run`, `--json-only`, …). You can combine a ref with flags too:
   `test-classifier origin/main --post-comment`.
 
-Prefix `AI_RUN_SUITE=1` to run the suite locally (OBSERVED) instead of inferring
-from the diff: `AI_RUN_SUITE=1 test-classifier`.
+By default a local run is OBSERVED — it runs your suite and triages the real
+failures. Pass `--no-run-suite` (or `AI_RUN_SUITE=0`) for a read-only INFERRED
+pass that only reasons over the diff: `test-classifier --no-run-suite`.
 
 ---
 
 ## Usage
 
 ```bash
-test-classifier                       # everything unpushed: committed + staged (INFERRED)
+test-classifier                       # everything unpushed: runs the suite, triages REAL failures (OBSERVED)
 test-classifier origin/main           # the committed range origin/main..HEAD
 test-classifier HEAD~1                # just the last commit
 
-AI_RUN_SUITE=1 test-classifier        # run the suite locally and triage REAL failures (OBSERVED)
+test-classifier --no-run-suite        # read-only: predict from the diff, don't run the suite (INFERRED)
 
 # Post the result as a PR comment (needs an open PR + gh authed):
 test-classifier --post-comment              # auto-discovers the current branch's PR
 test-classifier --pr 42 --post-comment      # explicit PR number
-AI_RUN_SUITE=1 test-classifier --pr 42 --post-comment   # OBSERVED + post
+test-classifier --pr 42 --no-run-suite --post-comment   # INFERRED + post
 
 # Streamlined: post AND record the metric in one shot (see --submit below):
 test-classifier --pr 42 --submit
@@ -151,8 +154,8 @@ Events** tab — verdict, file context, confidence, and your 👍/👎 (with an
 one-line reason on a 👎). No GitHub round-trip, no separate harvest.
 
 ```bash
-test-classifier --pr 42 --submit                 # classify (INFERRED) → post → ask → record
-AI_RUN_SUITE=1 test-classifier --pr 42 --submit  # OBSERVED → post → ask → record
+test-classifier --pr 42 --submit                 # classify (OBSERVED) → post → ask → record
+test-classifier --pr 42 --no-run-suite --submit  # INFERRED → post → ask → record
 test-classifier --submit                         # auto-discovers the current branch's PR
 ```
 
@@ -200,11 +203,11 @@ The row lands in the **Testing Events** tab by default; override with
   `→ Fix the CODE`, `→ Re-run / deflake`, `→ Fix the ENV`). That's the punchline:
   the classifier is **diagnostic** — it tells you which side to fix, never writes
   the patch. Read the summary, then make the change yourself.
-- **INFERRED vs OBSERVED.** Without `AI_RUN_SUITE=1` the agent predicts failures
-  from the diff (INFERRED) and never touches your toolchain. With it, the agent
-  runs your suite (OBSERVED) — the only mode in which `FLAKY_FAILURE` /
-  `ENVIRONMENT_ISSUE` are reliably reachable, since you can't see a timeout or
-  non-determinism from a diff.
+- **OBSERVED vs INFERRED.** By default the agent runs your suite (OBSERVED) —
+  the only mode in which `FLAKY_FAILURE` / `ENVIRONMENT_ISSUE` are reliably
+  reachable, since you can't see a timeout or non-determinism from a diff. With
+  `--no-run-suite` (`AI_RUN_SUITE=0`) it predicts failures from the diff
+  (INFERRED) and never touches your toolchain.
 - **Live progress.** On a local interactive run the dispatcher streams the
   agent's steps (each `⏺` reasoning line and `⏎` tool call) to your terminal as
   they happen, so it isn't a blinking cursor while it works. The final report is
@@ -240,7 +243,7 @@ run (Path B) is still the recorded, metrics-feeding pass; this is your preview.
 | `--unpushed: couldn't determine what's been pushed` | No upstream and no remote default branch (e.g. brand-new branch, no remote). Pass an explicit base: `test-classifier origin/main`. |
 | `AI_REVIEW_TOOL … not set` | Export `AI_REVIEW_TOOL=claude` (or `codex`/`copilot`); see `README.md`. |
 | Classification seems to miss recent edits | `--unpushed` excludes *unstaged* changes. `git add` or commit them first. |
-| Everything lands in INFERRED | That's the default. Add `AI_RUN_SUITE=1` to run the suite (OBSERVED). Suites needing services may still fall back to INFERRED, by design. |
+| Expected OBSERVED but got INFERRED | OBSERVED is the default, but the agent falls back to INFERRED when it can't run the suite — no test command found, missing toolchain, suite needs services (DB), it times out, or it would apply real infra with no teardown. Check the `summary`, which states the reason. (And confirm you didn't pass `--no-run-suite` / set `AI_RUN_SUITE=0`.) |
 | Posted to / recorded the wrong repo (e.g. the upstream of your fork) | The repo is resolved from your `origin` remote. On a fork whose `origin` isn't the repo the PR lives in, export `AI_REVIEW_REPO=owner/name` to set it explicitly. |
 
 ---

testing/classifier/docs/SETUP.md

@@ -123,16 +123,17 @@ testing/classifier/.skills/test-classifier/scripts/test-classifier-dispatcher.sh
 testing/classifier/.skills/test-classifier/scripts/test-classifier-dispatcher.sh \
   --dry-run
 
-# Opt in to running the suite locally (OBSERVED). OFF by default for the local
-# path so a run never auto-installs deps or executes tests on your machine:
-AI_RUN_SUITE=1 \
+# Read-only INFERRED pass — predict from the diff, don't run the suite. Use on
+# an untrusted diff you don't want to install deps for / execute on your machine:
 testing/classifier/.skills/test-classifier/scripts/test-classifier-dispatcher.sh \
-  --post-comment
+  --no-run-suite --post-comment
 ```
 
-By default a **local** run is read-only and INFERRED (predicts from the diff) —
-it will not install deps or run your suite. Set `AI_RUN_SUITE=1` to let the agent
-run the suite locally (the same OBSERVED behavior CI uses). CI sets this for you.
+By default a run is OBSERVED (the same behavior CI uses): when the repo has a
+suite, the agent installs deps and runs it to triage the real failures. Pass
+`--no-run-suite` (or set `AI_RUN_SUITE=0`) for a read-only INFERRED pass that
+predicts from the diff and never touches your toolchain — use it on a branch you
+don't trust enough to execute.
 
 By default the dispatcher prints the classification report to the terminal
 only. Add `--post-comment` to also post the PR comment.
@@ -256,8 +257,10 @@ The full report is also uploaded as a CI artifact, and the run is non-blocking.
 
 #### OBSERVED vs INFERRED — what the classifier actually sees
 
-In CI the workflow sets `AI_RUN_SUITE=1`, which grants the agent shell execution
-so it **runs your suite itself**: it locates your test command (package.json,
+OBSERVED is the default everywhere; the CI workflow still sets `AI_RUN_SUITE=1`
+explicitly (self-documenting, and pins it regardless of the default). That grants
+the agent shell execution so it **runs your suite itself**: it locates your test
+command (package.json,
 Makefile, pytest/tox, go.mod, Cargo.toml, or your CI's test step), installs deps
 from your lockfile best-effort, runs the tests, and triages the **real**
 failures. That comment is marked **Observed**.