fix(test-classifier): raise turn budget + treat max-turns overflow as soft outcome (#79)

* fix(test-classifier): raise turn budget + treat max-turns overflow as soft outcome

Cold-runner suite bootstrap (Ruby+bundle, Postgres, asset precompile) could
consume the whole agentic turn budget before the agent emitted a verdict,
tripping "Reached max turns" and failing the job red.

- Raise AI_SUITE_MAX_TURNS default 40 -> 80 for bootstrap headroom.
- Grant Grep,Glob,Task,Agent so read-only search batches in parallel and
  large-diff discovery can be delegated to a subagent (one parent turn).
- Enable codex network access in workspace-write so bundle/npm/pip installs
  work in suite mode (read-only triage keeps the no-network default).
- Detect turn-budget overflow (ai_review::is_max_turns) and degrade to a
  non-blocking soft outcome (exit 0) instead of failing the build.
- Document budget discipline in SKILL.md / dispatcher: reuse the repo's own
  bootstrap entry point or CI recipe, batch lookups, run the suite once
  blocking, and emit a best-effort verdict before overflowing.

* docs(test-classifier): trim overlong inline comments and prompt prose

Author: fg-nava

testing/classifier/.skills/_lib/ai-classifier-dispatch.sh

@@ -66,11 +66,13 @@ _AI_CLASSIFIER_DISPATCH_LOADED=1
 # read-only, INFERRED-only pass.
 AI_RUN_SUITE="${AI_RUN_SUITE:-1}"
 
-# Bounds for the agent loop when it is running the suite, so a runaway
-# install/test cycle can't hang the job. The timeout wraps the whole CLI call;
-# --max-turns (claude) caps agentic iterations.
+# Bounds for the agent loop when running the suite. The timeout wraps the whole
+# CLI call; --max-turns caps agentic iterations. The budget must cover both a
+# cold-runner bootstrap (toolchain, DB, assets) and the classification — 40 was
+# too low (bootstrap alone hit the cap). Overflow is a soft outcome now (see
+# ai_review::is_max_turns), not a hard failure.
 AI_SUITE_TIMEOUT_SECS="${AI_SUITE_TIMEOUT_SECS:-1500}"   # 25 min hard ceiling
-AI_SUITE_MAX_TURNS="${AI_SUITE_MAX_TURNS:-40}"
+AI_SUITE_MAX_TURNS="${AI_SUITE_MAX_TURNS:-80}"
 
 # ── Color helpers (suppressed in CI / non-TTY) ──────────────────────────────
 if [[ -t 1 ]] && [[ "${CI:-}" != "true" ]] && [[ "${NO_COLOR:-}" == "" ]]; then
@@ -494,6 +496,10 @@ ai_review::invoke_claude() {
     # this flag set is required for suite-running, not optional. --allowedTools
     # scopes it to exactly what the task needs; --max-turns bounds the loop.
     #
+    # Grep,Glob batch in parallel within one turn (vs a `bash grep` per turn);
+    # Task/Agent (subagent tool, both names) lets a whole discovery pass cost
+    # the parent ONE turn. WebSearch/WebFetch stay ungranted (data minimization).
+    #
     # NOTE: no 2>&1 — keep the agent's stderr diagnostics OUT of the captured
     # stdout so they can't corrupt the JSON/marker parse. stderr still flows to
     # the terminal / CI log.
@@ -503,14 +509,14 @@ ai_review::invoke_claude() {
       # there is no stdin to read.
       $(ai_review::timeout_prefix) claude -p "${SKILL_PROMPT}" \
         --permission-mode bypassPermissions \
-        --allowedTools "Bash,Read,Edit" \
+        --allowedTools "Bash,Read,Edit,Grep,Glob,Task,Agent" \
         --max-turns "${AI_SUITE_MAX_TURNS}" \
         --output-format stream-json --verbose < /dev/null \
         | ai_review::stream_split
     else
       $(ai_review::timeout_prefix) claude -p "${SKILL_PROMPT}" \
         --permission-mode bypassPermissions \
-        --allowedTools "Bash,Read,Edit" \
+        --allowedTools "Bash,Read,Edit,Grep,Glob,Task,Agent" \
         --max-turns "${AI_SUITE_MAX_TURNS}"
     fi
   else
@@ -541,15 +547,24 @@ ai_review::invoke_codex() {
   # folded into the captured stdout that gets parsed.
   local stream="${AI_REVIEW_DO_STREAM:-0}"
   local sandbox="read-only"
-  (( AI_RUN_SUITE == 1 )) && sandbox="workspace-write"
+  local -a codex_cfg=()
+  if (( AI_RUN_SUITE == 1 )); then
+    sandbox="workspace-write"
+    # workspace-write allows file writes but not network by default, so dep
+    # installs (bundle/npm/pip) fail without this. Enable it only for suite
+    # mode; read-only triage keeps the no-network default.
+    codex_cfg=(-c sandbox_workspace_write.network_access=true)
+  fi
 
   if (( stream == 1 )); then
     ai_review::info "Streaming the agent's steps below (set AI_REVIEW_STREAM=0 to silence)…" >&2
     $(ai_review::timeout_prefix) codex exec --json --sandbox "${sandbox}" \
+      "${codex_cfg[@]+"${codex_cfg[@]}"}" \
       --skip-git-repo-check "${SKILL_PROMPT}" \
       | ai_review::codex_stream_split
   else
     $(ai_review::timeout_prefix) codex exec --sandbox "${sandbox}" \
+      "${codex_cfg[@]+"${codex_cfg[@]}"}" \
       --skip-git-repo-check "${SKILL_PROMPT}" 2>&1
   fi
 }
@@ -674,6 +689,17 @@ ai_review::invoke_ai() {
   esac
 }
 
+# ── Turn-budget overflow detection ──────────────────────────────────────────
+# When the agent exhausts --max-turns it exits non-zero with a recognizable
+# banner instead of our marker. That's recoverable, not a real failure — the
+# caller degrades to a non-blocking soft outcome. The match is broad/case-
+# insensitive across CLIs; a false positive only softens a hard failure, which
+# is the safe direction.
+ai_review::is_max_turns() {
+  local output="$1"
+  grep -qiE 'reached max turns|max[ _-]?turns|turn limit' <<< "${output}"
+}
+
 # ── Result marker parsing ───────────────────────────────────────────────────
 # The canonical marker is:  <<<AI_REVIEW_RESULT:CLASSIFIED|NO_ACTION>>>
 # We grep for the structured form; if absent we return UNPARSEABLE so the
@@ -752,6 +778,13 @@ ai_review::run() {
   ai_review::log  "────────────────────────────────────────────────────────────"
 
   if (( invoke_rc != 0 )); then
+    # Turn-budget overflow is a soft outcome: no verdict, but exit 0 rather than
+    # failing the build. Bump AI_SUITE_MAX_TURNS or set AI_RUN_SUITE=0.
+    if ai_review::is_max_turns "${review_output}"; then
+      ai_review::warn "${SKILL_HUMAN_NAME}: agent hit the turn budget (AI_SUITE_MAX_TURNS=${AI_SUITE_MAX_TURNS}) before finishing — no verdict emitted this run."
+      ai_review::warn "  This is non-blocking. Bump AI_SUITE_MAX_TURNS for more headroom, or run with AI_RUN_SUITE=0 (--no-run-suite) for a fast diff-only (INFERRED) pass."
+      exit 0
+    fi
     ai_review::err "AI CLI (${AI_REVIEW_TOOL_RESOLVED}) exited with code ${invoke_rc}."
     if (( AI_REVIEW_NO_BLOCK == 1 )); then
       ai_review::warn "--no-block in effect: not failing despite CLI error."

testing/classifier/.skills/test-classifier/SKILL.md

@@ -129,6 +129,30 @@ classify the failures you actually observe:
    your OBSERVED signal.
 4. Set the result `mode` to `"OBSERVED"` in the JSON.
 
+> **Spend your turn budget on classification, not on rediscovering the
+> environment.** You have a bounded number of agentic turns
+> (`AI_SUITE_MAX_TURNS`), and a turn is one assistant iteration, not one tool
+> call — batch independent calls into one turn. Trial-and-error bootstrap on a
+> cold runner can burn the whole budget before you emit a verdict, so:
+>
+> - **Reuse the repo's own setup.** Prefer its bootstrap entry point
+>   (`bin/setup`, `script/bootstrap`, a Makefile `setup` target); otherwise
+>   reproduce the CI workflow (`.github/workflows/*.yml` — setup steps, service
+>   containers, version pins, test command) rather than guessing. If CI
+>   precompiles assets, do that too (a suite that skips it fails with an
+>   `ENVIRONMENT_ISSUE`, not an app bug).
+> - **Batch.** Chain setup steps with `&&` in one command (shell state doesn't
+>   persist across calls); issue independent read-only lookups together.
+> - **Run the suite once, blocking** — don't background it and poll with
+>   `sleep`/`pgrep`.
+> - **Delegate large-diff discovery to a subagent if you have one** (Claude
+>   Code's `Task`/`Agent`): its whole exploration costs the parent ONE turn.
+>   Give it the paths/questions it needs (it can't see your history). Codex and
+>   Copilot have none — keep discovery serial but batched.
+> - **Degrade before you overflow.** Near the cap, emit your best verdict from
+>   what you've observed (`confidence: low` beats no output); if bootstrap can't
+>   finish, fall back to `INFERRED` and say so.
+
 > **Infrastructure-as-code tests need a teardown guarantee — never strand real
 > resources.** Some repos (Terraform/OpenTofu modules, Pulumi, CloudFormation)
 > have a "test suite" that **applies real cloud resources**. That is fine to run

testing/classifier/.skills/test-classifier/scripts/test-classifier-dispatcher.sh

@@ -400,6 +400,17 @@ if [[ "${AI_RUN_SUITE:-1}" == "1" ]]; then
      cannot locate/install/run it (no suite, missing toolchain, needs services,
      times out), fall back to predicting from the diff, set "mode":"INFERRED",
      and state the reason in "summary".
+
+     BUDGET DISCIPLINE — you have a bounded number of agentic turns; a turn is
+     one assistant iteration, not one tool call, so batch independent lookups
+     and chain setup steps with && into one command. Derive the bootstrap from
+     the repo FIRST (its own setup entry point, else its CI workflow) instead
+     of guessing package managers by trial and error. Run the suite ONCE as a
+     blocking command — don't poll with sleep/pgrep. For a LARGE diff, if you
+     have a subagent tool (Claude Code Task/Agent), delegate discovery to it
+     (one parent turn; give it the paths/questions — it can't see your
+     history); codex/copilot have none, so keep discovery serial but batched.
+     Near the budget, STOP and emit your best verdict rather than overflowing.
 SIGNAL
 else
   read -r -d '' SIGNAL_STEP <<'SIGNAL' || true
@@ -1048,6 +1059,14 @@ test_classifier::run() {
   fi
 
   if (( invoke_rc != 0 )); then
+    # Turn-budget overflow is a soft outcome: no verdict, but exit 0 so the PR
+    # check stays green-advisory rather than failing red. Bump AI_SUITE_MAX_TURNS
+    # or set AI_RUN_SUITE=0 for a diff-only (INFERRED) pass.
+    if ai_review::is_max_turns "${classifier_output}"; then
+      ai_review::warn "AI Test Classifier: agent hit the turn budget (AI_SUITE_MAX_TURNS=${AI_SUITE_MAX_TURNS}) before emitting a verdict — no classification this run (non-blocking)."
+      ai_review::warn "  Bump AI_SUITE_MAX_TURNS for more headroom, or set AI_RUN_SUITE=0 (--no-run-suite) for a fast diff-only (INFERRED) pass."
+      exit 0
+    fi
     ai_review::err "AI CLI (${AI_REVIEW_TOOL_RESOLVED}) exited with code ${invoke_rc}."
     exit 1
   fi