Skip to content

fix(check-pr-fully-clean): stop scoring non-blocking as a finding - #3497

Merged
d-morrison merged 5 commits into
mainfrom
fix-nonblocking-false-positive-3487
Sep 10, 2026
Merged

d-morrison merged 5 commits into
mainfrom
fix-nonblocking-false-positive-3487

Conversation

@d-morrison

Copy link
Copy Markdown
Collaborator

check-pr-fully-clean.py scored the string non-blocking as a finding.

\bblocking\b matches inside non-blocking — the hyphen ends the preceding word, so the word boundary sits right before blocking. The only negation guard in the file was scoped to changes\s+requested and tested \bno\s+…, which requires a space, so a hyphen was never going to match anyway.

Why it mattered

"Non-blocking" is the commonest way a reviewer marks a finding as explicitly not blocking, so the instrument reported NOT CLEAN precisely when a reviewer had gone out of their way to say the opposite.

That is the instrument committing the error it exists to prevent. The whole argument for check-pr-fully-clean.py over a hand-rolled phrase search — the one its own guard hook makes when it refuses a grep for ### Verdict — is that phrase searches misread in both directions.

Live before this change, on #3468:

  verdict scan: examined 3 dated automated review item(s), 2 bore a verdict, latest = clean
❌ PR is NOT fully clean:
  - Review comment for SHA 654301d1 contains findings (matched pattern '...Block(?:ed|ing)?...')

All nine matches sat inside sentences like "Ready for merge. One non-blocking suggestion … does not block this PR's own content". After this change the same command exits 0.

The change

  • _BARE_REJECTION's Block(?:ed|ing)? lookbehinds now cover not\s and no\s as well as non- and non\s.
  • NOT_CLEAN_NEGATION_PREFIX accepts a hyphen after the negator through a dedicated alternation that fires only immediately after it — deliberately not a blanket [\s-] substitution.
  • The parallel hyphen guard in _unresolved_finding_pattern's changes requested branch is redundant with the above and kept for parity with the issue's sketch; its comment says so, and reverting it fails no test.

Rejected and Unapproved are left unguarded. A grep for non-rejected / not rejected / no rejected and the Unapproved equivalents found no genuine idiom anywhere in the repo, so a speculative guard there would only risk swallowing a real finding for no benefit.

Why the blanket substitution was wrong

The obvious [\s-] version was written first, and review caught it swallowing real findings: "Not-negligible changes requested." classified clean. That is the dangerous direction — a false clean — which is why the fix narrows to hyphen-immediately-after-negator rather than treating hyphens and spaces alike everywhere in the separator.

Verification

797 tests pass. 20 are new, each mutation-checked:

mutation result
not\s/no\s lookbehinds reverted 2 fail
hyphen branch reverted to bare \s 2 fail
blanket [\s-] reintroduced 4 fail, reproducing the compound-adjective false negatives
redundant changes requested guard reverted 0 fail (confirmed redundant, not silently broken)

Dangerous-direction controls, all still correctly flagged as findings: "this is not a nit — it is blocking", "no doubt this is blocking", "not-negligible changes requested", "non-trivial changes requested", "nothing here is optional: blocking". Cross-clause too: "Non-blocking suggestion. Changes requested regardless." stays not-clean, since the hyphen-adjacency window does not leak across an independent clause.

Closes #3487

🤖 Generated with Claude Code

d-morrison and others added 2 commits September 9, 2026 19:24
…st "non-"

The #2419 fix only exempted the non-/non  compounds from the Block(?:ed|ing)?
alternative, so "not blocking" and "no blocking findings remain" -- the same
statement with a different negator immediately before the word -- still
matched. Add (?<!not\s)(?<!no\s) lookbehinds alongside the existing
non-/non\s ones.

The generic NOT_CLEAN_NEGATION_PREFIX guard used by classify_verdict and
_unresolved_finding_pattern had the mirror image of the same gap: its
negator-to-phrase separator was a bare \s, so a hyphenated negation
("No-changes requested.") was not exempted even though the spaced form
already was. Widen it to [\s-]. The existing hyphen-specific check on the
"changes requested" pattern is widened the same way, though it turns out to
be redundant with the NOT_CLEAN_NEGATION_PREFIX fix -- kept for consistency
with the issue's sketch.

Rejected and Unapproved were checked for the same treatment and left
unguarded: unlike "non-blocking", there is no established review idiom
"non-rejected" or "not unapproved" anywhere in this corpus's own prose or
test fixtures, so a speculative guard there would only risk swallowing a
genuine finding for no real benefit.

Both directions are pinned with mutation-checked tests: the four new/positive
exemption checks fail if the lookbehinds or NOT_CLEAN_NEGATION_PREFIX
widening are reverted, and the "not a nit -- it is blocking" / "no doubt
this is blocking" checks fail against a blanket negation-window
implementation (confirmed against a standalone simulation of that
anti-pattern) while passing against this fixed-width one.

Closes #3487

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ctives

Round-2 finding from the adversarial-reviewer dispatched against d287f21:
substituting [\s-] for every \s in NOT_CLEAN_NEGATION_PREFIX (and the
parallel changes-requested guard) let the hyphen stand in for a space
inside the filler-word separator too, not just between the negator and the
target phrase. That swallowed real not-clean statements whose negator word
happens to open an unrelated hyphenated compound adjective --
"Not-negligible changes requested.", "No-nonsense changes requested here.",
"Not-yet-addressed changes requested below.", and "Never-resolved changes
requested." all silently classified as no-verdict instead of not-clean.

Replace the blanket substitution with a dedicated hyphen alternative that
only fires in the one position the fix actually needs: immediately after
the negator, with no intervening filler word ("no-changes", "non-blocking").
Any filler word must still be space-separated, exactly as before #3487.
Confirmed the four phrases above stay not-clean, added them as regression
tests, and confirmed (by reverting to the blanket form) that those tests
fail against it.

Full suite: 797 passed, 0 failed. Live repro still exits 0:
python3 scripts/check-pr-fully-clean.py 3468 -R Morrison-Lab/ai-config

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@d-morrison

Copy link
Copy Markdown
Collaborator Author

Claimed by a Claude Code CLI session — driving this PR to a clean review and green CI.

Found while measuring a different PR: the instrument reported #3468 not-clean for three consecutive turns, and I chased that verdict rather than asking why a PR whose every review says "Ready for merge" was scoring as having findings.

Posted by Claude Code (AI agent) --- not written by a human.

@d-morrison
d-morrison requested a lite review from Copilot September 10, 2026 02:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

…line

Review of #3497 found the hyphen branch written as
`(?:no|not|nothing|none|never)(?:-|\s+...)`, which exempted `not-blocking`,
`no-blocking`, `never-blocking` and `not-rejected` through the FULL
`classify_verdict` / `_unresolved_finding_pattern` pipeline. That undid
one layer up the narrowness `_BARE_REJECTION`'s own lookbehinds were
written to preserve, and it undid it in the dangerous direction -- a
swallowed not-clean.

It passed 806 tests because the new cases asserted against
`_BARE_REJECTION` directly, and the file says why: for the SPACE forms
the bare level is the right place, since the generic guard already
covers them. That reasoning does not carry to the hyphen forms, whose
entire failure mode lives in the generic guard. So the hyphen cases are
now asserted through the pipeline.

`not`, `never`, `nothing` and `none` lose the hyphen. `not-negligible`
and `not-a-nit` are compound adjectives whose hyphen does not negate
what follows.

`no-` keeps it, which is a partial rebuttal of the finding rather than a
straight fix. The guard sees only the text BEFORE the matched phrase, so
it cannot exempt `no-changes requested` (a real usage, #2369) while
flagging `no-blocking` -- the two differ only in what follows. Granted
`no-`, `no-blocking` reads clean, which is the right answer anyway: it
means what `non-blocking` means. Both halves are now pinned by tests so
the tradeoff cannot be silently reversed.

Mutation-checked against the restored 806/806:

    the over-broad hyphen branch restored  ->  800/806, 6 failing

`python3 scripts/check-pr-fully-clean.py 3468 -R Morrison-Lab/ai-config`
exits 0.

Refs #3487

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@d-morrison

Copy link
Copy Markdown
Collaborator Author

Pausing here. State, so nothing is lost.

The review's finding stands, and my rebuttal was wrong

I argued the guard could not flag no-blocking while exempting no-changes requested, because it sees only the text before the matched phrase. The reviewer refuted that: the loop already has the matched pat in scope and already branches on it (if pat == _BARE_REJECTION, if pat == r"\bNeeds...", the changes\s+requested hyphen check). A pattern-scoped exemption is available; I asserted an architectural limit that is a property of the single regex I wrote, not of the pipeline.

And it found a second, worse one

NOT_CLEAN_NEGATION_PREFIX is consulted for every pattern in VERDICT_NOT_CLEAN_PATTERNS / FINDING_PATTERNS, not just Block…. So granting the hyphen to non-/no- silently swallows:

classify_verdict("### Verdict\nReady for merge. non-rejected nit noted.\n", "")   -> "clean"
classify_verdict("### Verdict\nReady for merge. no-rejected nit noted.\n", "")    -> "clean"
classify_verdict("### Verdict\nReady for merge. non-unapproved nit noted.\n", "") -> "clean"
classify_verdict("### Verdict\nReady for merge. non-impasse nit noted.\n", "")    -> "clean"
classify_verdict("### Verdict\nReady for merge. non-deadlock nit noted.\n", "")   -> "clean"

All five classify not-clean on origin/main, so it is a regression this branch introduces — in the dangerous direction the whole chain (#2369, #3487, this PR) exists to close. It also falsifies a claim in my own commit: the file says Rejected/Unapproved were "deliberately left unguarded", which is true of _BARE_REJECTION's lookbehinds and false of the shipped behaviour, because the generic guard reaches them anyway.

Why 806 tests did not catch it

The new cases test not-blocking and friends staying flagged. None tests non-rejected — the exact class the commit claims to have considered. A test asserting the documented behaviour would have failed against the code shipped beside it.

Next step, for whoever picks this up

Scope the hyphen exemption to the pattern it was written for, using the pat the loop already carries, rather than widening a guard shared by every not-clean phrase. Then pin, through classify_verdict and _unresolved_finding_pattern rather than against the bare pattern:

  • non-blocking, no-changes requested — clean
  • non-rejected, no-rejected, non-unapproved, non-impasse, non-deadlock — not-clean
  • not-blocking, never-blocking, not-negligible changes requested — not-clean

origin/HEAD here is 50885dc1, which carries the regression. A local commit 45d1cbb8 narrows the hyphen to non-/no- and adds pipeline tests, but it does not fix the finding above and is deliberately unpushed.

The pattern worth naming

Three rounds on this PR, and each fix reintroduced the same defect class one layer out: _BARE_REJECTION's lookbehinds, then the generic prefix guard, then the generic guard's reach across every pattern. Every round the fix was correct about the case in front of it and wrong about the scope it applied at. That is the same shape as #3475's rounds 7 and 8, where narrowing and widening a PR-matching set each broke the other direction until the decision was moved to a signal that needed no heuristic at all.

Posted by Claude Code (AI agent) --- not written by a human.

d-morrison and others added 2 commits September 10, 2026 10:22
The review was right twice, and I withdraw the rebuttal.

`NOT_CLEAN_NEGATION_PREFIX` is consulted for EVERY pattern, so granting it
`non-`/`no-` silently swallowed `non-rejected`, `no-rejected`,
`non-unapproved`, `non-impasse` and `non-deadlock` -- all `not-clean` on
`main`, all reading clean on this branch. That is a regression this branch
introduced, in the dangerous direction the whole chain exists to close, and
it falsified this file's own claim that `Rejected`/`Unapproved` were left
unguarded: true of the lookbehinds, false of the shipped behaviour.

I had also argued the guard could not flag `no-blocking` while exempting
`no-changes requested`, because it sees only the text before the match.
The loop carries `pat` and already branches on it three times, so that was
a property of the regex I wrote rather than of the pipeline.

The fix needs neither the shared guard nor an enumeration in the loop --
this file already lost twice to enumerating which patterns need handling.
`no-changes requested` is one phrase, so the hyphen goes on that phrase:
`(?<!no-)Changes\s+requested`, in `_BARE_REJECTION` and in both pattern
lists. The generic guard returns to space-only, and every hyphenated
negator before any not-clean phrase stays flagged.

`non-blocking` -- the case #3487 is about -- needs none of this. It was
already covered by `_BARE_REJECTION`'s own `(?<!non-)` lookbehind; the
hyphen in the shared guard was never load-bearing for it.

Mutation-checked against the restored 811/811:

    generic hyphen exemption re-added        ->  805/811, 6 failing
    `(?<!no-)` dropped from _BARE_REJECTION  ->  809/811, 2 failing
    `(?<!no-)` dropped from both pattern lists -> 808/811, 3 failing

`python3 scripts/check-pr-fully-clean.py 3468 -R Morrison-Lab/ai-config`
exits 0.

Refs #3487

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… comment

Three findings from the review of 6065247, all accepted.

**The lookbehinds had no word boundary.** `(?<!no-)` and `(?<!no\s)` fire
on any longer word ending in those letters, so "Volcano-changes
requested.", "Casino-changes requested." and "casino blocking issue
remains" all classified as no-verdict instead of not-clean -- a swallowed
not-clean, from a guard added to stop swallowing not-cleans. Every
lookbehind is now `\b`-anchored.

**The `if pat == r"changes\s+requested\b":` branch became unreachable**
when the previous commit put `(?<!no-)` into both list entries: the string
equality can no longer hold. Its own comment called it redundant, which it
was one commit earlier; it is dead now, and deleting it changes no test.

**The comment block above `NOT_CLEAN_NEGATION_PREFIX` still described the
mechanism the previous commit removed**, asserting that `no-` keeps the
hyphen and `no-blocking` reads clean -- contradicted by the shipped regex
and by the tests beside it. Rewritten to say what the code now does and
why the hyphen lives on the phrase instead of the shared guard.

Mutation-checked against the restored 817/817:

    `\b` dropped from the Block lookbehinds  ->  814/817, 3 failing
    `\b` dropped from the `no-` guards       ->  814/817, 3 failing

`python3 scripts/check-pr-fully-clean.py 3468 -R Morrison-Lab/ai-config`
exits 0.

Refs #3487

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Claude finished reviewView run


Summary

This round is a direct, targeted fix for the exact regression I flagged in the prior review round (commit 50885dc1): NOT_CLEAN_NEGATION_PREFIX's hyphen branch was a generic guard applied to every pattern in VERDICT_NOT_CLEAN_PATTERNS/FINDING_PATTERNS, so it silently exempted not-blocking, no-blocking, never-blocking, and not-rejected — directly contradicting the PR's own stated design intent that those forms stay flagged.

The fix removes the hyphen alternative from the shared NOT_CLEAN_NEGATION_PREFIX guard entirely (reverting it to the pre-PR space-only form, scripts/check-pr-fully-clean.py:1898-1901), and instead moves the hyphen exemption onto the specific phrase that needed it: (?<!\bno-)Changes\s+requested is now written directly into _BARE_REJECTION and into both VERDICT_NOT_CLEAN_PATTERNS/FINDING_PATTERNS' own changes\s+requested\b entries (lines 1299, 1874, 1972). The now-redundant _unresolved_finding_pattern special-case branch for changes\s+requested\b was correctly dropped as unreachable. Extensive new test coverage (lines 176-291 of the diff) exercises the fix through the full classify_verdict/_unresolved_finding_pattern pipeline rather than only at the bare-pattern level — closing the exact gap that let the regression pass 797/797 tests in the prior round.

I independently re-derived and ran the regexes rather than trusting the commentary or the new tests, per this repo's own algorithmatize-checks standard:

classify_verdict("### Verdict\nReady for merge. not-blocking nit noted.\n", "")   -> "not-clean"  ✓ (was "clean" before this fix)
classify_verdict("### Verdict\nReady for merge. no-blocking nit noted.\n", "")     -> "not-clean"  ✓
classify_verdict("### Verdict\nReady for merge. never-blocking nit noted.\n", "")  -> "not-clean"  ✓
classify_verdict("### Verdict\nReady for merge. not-rejected nit noted.\n", "")    -> "not-clean"  ✓
classify_verdict("### Verdict\nReady for merge. non-blocking nit noted.\n", "")    -> "clean"       ✓ (the real exemption, still works)
classify_verdict("### Verdict\nNo-changes requested.\n", "")                       -> ""            ✓ (the #3487 fix this PR preserves)
classify_verdict("### Verdict\nNot-negligible changes requested.\n", "")           -> "not-clean"  ✓ (compound-adjective false-negative avoided)
classify_verdict("### Verdict\nVolcano-changes requested.\n", "")                  -> "not-clean"  ✓ (word-boundary anchoring on the lookbehind holds)
classify_verdict("### Verdict\nThis is not a nit -- it is blocking.\n", "")        -> "not-clean"  ✓ (non-adjacent negator still flags)

All match expectations. I also confirmed the full local test suite passes: python3 scripts/test_check_pr_fully_clean.py843 passed, 0 failed, and hooks/test-no-handrolled-verdict-parse.py → 54/54 passed (unaffected by this change, checked since it shares verdict-parsing vocabulary). CI on the PR shows no failed checks (review workflow and validate were pending/in-progress at the time of this check; nothing red).

I found no other issues in this round's diff — no new patterns introduced, no scope creep, and the redundant code removal was correctly scoped (only the dead if pat == r"changes\s+requested\b" branch in _unresolved_finding_pattern, whose job is now done by the lookbehind embedded in the pattern itself).

Findings

None. The prior round's finding (NOT_CLEAN_NEGATION_PREFIX's hyphen branch silently exempting not-blocking/no-blocking/not-rejected) is fixed, verified independently, and covered by new tests that exercise the full pipeline rather than only the bare pattern — closing the exact coverage gap that let the regression through last round.

Verdict

Ready for merge — the previously-flagged regression is fixed and independently verified; no new issues found.

Structured Review Data (JSON)
{
  "schema_version": "1.1",
  "reviewer": "claude",
  "commit_sha": "348393ffcb0aa9c44f7a523a952853d453a98581",
  "verdict": "CLEAN",
  "findings": [],
  "detailed_assessment": "The NOT_CLEAN_NEGATION_PREFIX hyphen-branch regression flagged in the prior round (commit 50885dc1) is fixed by scoping the hyphen exemption onto the specific Changes-requested phrase in scripts/check-pr-fully-clean.py rather than the shared negation-prefix guard, verified independently via classify_verdict on not-blocking, no-blocking, never-blocking, and not-rejected all correctly returning not-clean.",
  "holistic_assessment": "The fix closes the exact test-coverage gap that let the regression pass 797 tests last round by adding pipeline-level tests through classify_verdict and _unresolved_finding_pattern, and the full local suite of 843 tests passes with CI showing no failed checks."
}

Reviewed commit: 163a963

@github-actions

Copy link
Copy Markdown
Contributor

💰 Cost: $0.9948 (review) — run

@d-morrison
d-morrison merged commit 9e783de into main Sep 10, 2026
15 checks passed
@d-morrison
d-morrison deleted the fix-nonblocking-false-positive-3487 branch September 10, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

check-pr-fully-clean.py scores non-blocking as a finding

2 participants