auto: a node denied its tools reports PASS if it says so — the engine never sees the denial

[jitokim]

Raised by the maintainer while reading #213's measurement. It is the other
half of that issue, and the half that survives it.

Why this is specifically an auto problem

auto manufactures the condition. A planned node runs with
--setting-sources "" (coordinator.toolPolicyFor → isolatedSettingSources),
so the user's own pre-authorisations are not loaded and the plan's
allowed_tools is the entire grant. A hand-written run node inherits the
user's settings (internal/schedule builds ToolPolicy{AllowedTools: …} with
SettingSources nil), so it is far less likely to be denied anything.

The default permission_mode is dontAsk (internal/schedule/scheduler.go:47),
and a node has no TTY, so an unauthorised tool call is denied immediately —
it does not prompt and does not hang.

What actually happens on a denial

The tool call is refused, the model is told, and it keeps going. The node
exits 0. Whether the run notices depends entirely on whether the model chooses
to mention it and on what success_check asks.

We got lucky once already. In run 20260819-223604.575080000-1 the verify
node was denied Bash(gofmt …), said so honestly — "I did not route around the
denial … formatting is unverified" — and refused to claim PASS, so
result_matches failed it. A node that had started its reply with PASS and
put the caveat afterwards would have passed while never running the check.

handoff.LintToolGrants already names this exact failure in its warning text:

a tool your settings do not pre-authorise is denied silently, and the node's
own result_matches passes on the prose explaining it

The engine sees none of it — but the signal is structured

grep -rn "has been denied" over the product code returns nothing. There is
no detection at all.

The interesting part: a denial is not only prose. In the session transcript
it is a tool_result block carrying is_error: true and a tool_use_id
(verified on session 13f6c560-8204-4606-807e-183bb316a849). So unlike ADR
0009's session limit — which the CLI offers only as prose — this one has a
machine-readable shape.

It is simply out of band. The runner passes --output-format json and parses
six envelope fields (session_id, result, total_cost_usd, subtype,
is_error, errors — internal/runner/claude_protocol.go:84-89). Per-tool
results are not among them. Nothing in the tree uses stream-json.

Options, each with its real cost

1. Read the session transcript (~/.claude/projects/**.jsonl). The data is
   there today. Cost: a new dependency on Claude's private on-disk format, and
   no Codex equivalent — it would be a Claude-only signal, which ADR 0009 §Scope
   already established is an acceptable but explicit narrowing.
2. Consume stream-json so tool results arrive in-band. Cost: a real change
   to the runner's output handling, which is one of the four exec seams.
3. Prose-match the final result. Cheapest, and the weakest: it depends on the
   model choosing to mention the denial, which is exactly the behaviour that
   cannot be relied on here.
4. Do nothing to detect it; lean on success_check.verify. The house answer
   is already that a verify command is independent evidence. Arguably this issue
   is "encourage verify harder" rather than "detect denials".

Why #213's prescription does not cover this

#213 proposed catching the mismatch statically, before the run. That was
measured and rejected — 110 noise in 114 hits (#217), and the failure is
structural. If the mismatch cannot be caught before the run, the remaining place
to catch it is during it, which is this issue.

Not proposing an option here. The measurement that would decide it: over the
planned corpus, how many nodes were denied at least one tool call, and how many
of those still reported PASS.

[jitokim]

Measured — PR #220. Two corrections to this issue's own framing.

1. The signal is NOT structured. This issue claimed a denial is
machine-readable via is_error: true, unlike ADR 0009's session limit. That is
wrong. Every is_error: true tool_result in the corpus carries the identical
key set {content, is_error, tool_use_id, type} — an ordinary go test exiting
1 is indistinguishable from a denial except by the prose in content, and
the match must be anchored (HasPrefix "Permission to use "), because five
ordinary Bash failures here carry the denial vocabulary inside captured stdout.

That removes option 2 (stream-json): there is no marker to bring in-band.

2. The exposure is real, and bigger than expected — 18 planned runs, 81
nodes, 73 with transcripts on disk:

(a) denied ≥1 tool call	53 / 73
(b) of those, PASS	49
(c) of those, with engine-run verify	5
passed with a denial and no independent evidence	44

Recommendation: option 4 — lean on success_check.verify, denial detection
advisory at most.

The honest reading of (c) is that a detector would be treating a symptom.
Only 5 of 49 nodes that passed after a denial had the engine check anything.
The problem is not that denials go unnoticed; it is that most planned nodes have
no engine-run evidence at all, so nothing about them is checked independently
— denial or no denial. ADR 0030 already moved on this for the run as a whole
(--verify-cmd at the sink); this measurement says the same argument applies
per node.

Caveats stated in the write-up: it is a self-measurement on a live corpus (a
re-run during PR review gave 59/79 and 6/55 as other lanes added runs — ratios
stable at 75% and 11%), and 8 sessions had no transcript and are excluded from
the denominator rather than dropped silently.