feat(gates): a check that never ran is not a check that passed - #2
Merged
Conversation
`gate.sh static` returned 0 with FORMAT_CHECK_CMD, LINT_CMD and TYPECHECK_CMD all unset, and
`gate.sh green` inherited it. Every unconfigured command was announced as SKIPPED and contributed 0
to the verdict, so a project with an empty _STACK.md finished green having verified nothing — by
exit code, indistinguishable from a project where all three ran and passed. That is the same disease
the engine probe exists to cure one layer up ("installed does not mean usable"), and the harness had
it at home.
Adds a third outcome. Exit 3 = DEGRADED: the gate ran no check at all. Not a pass, not a failure —
the run continues and the claim shrinks.
- lib.sh: adw_run_cmd returns 3 for an unconfigured command, and takes the config key so it can quote
the operator's stated reason. `LINT_CMD= # none: no linter for Luau` now reads "declared none"
rather than "not configured". The wording changes; the verdict does not, because no linter is still
no lint coverage.
- gate.sh static: 1 if anything failed · 3 if nothing ran · 0 otherwise, naming what was skipped.
Partial coverage still passes — something did run — but it says so, because the PR has to.
- gate.sh green: 3 only when neither half ran. One half missing prints `PARTIAL` and passes; a real
failure outranks every degradation.
- gate.sh all: a failure wins, a degradation survives, so `all` cannot launder a 3 into a 0.
Callers updated to read three codes instead of two: adw-run (GREEN phase + rules), slice-implement,
verification-before-completion, the AGENTS.md block, the _STACK.md template, install.sh's closing
instructions, and the /adw-install verification step — where exit 3 is the likeliest outcome of a
fresh install and the one that looks like success if you only skim.
Tests: the two `check_xfail` assertions from the previous commit are promoted to `check`, asserting
3 rather than 1 — "must not report success" is satisfied by a code that means neither. 15 further
assertions cover the declared-none wording, both partial cases, failure-outranks-degradation, and
`all`'s aggregation. 29 in that case, 118 in the suite, 0 failures, 1 known gap remaining. Verified
non-vacuous by mutation: restoring the swallowed degradation in gate_static fails 6 assertions,
dropping green's both-degraded branch fails 3.
VERSION 2.0.0 → 2.1.0: the gate contract changed, and `doctor` compares installs against it.
Refs RUS-187
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1 — review that first, or read this diff alone since it only touches
gate.sh,lib.shand one test case.
Closes RUS-187.
The problem
gate.sh staticreturned 0 whenFORMAT_CHECK_CMD,LINT_CMDandTYPECHECK_CMDwere all unset, andgate.sh greeninherited it. Each unconfigured command was announced asSKIPPEDand contributed 0 tothe verdict, so a project with an empty
_STACK.mdfinished green having verified nothing —indistinguishable, by exit code, from one where all three ran and passed.
The harness already cures this one layer up: an engine CLI on PATH is not usable until it answers a
probe, because "a missing review lens looks exactly like a clean one". Same disease, at home.
The change
A third outcome. Exit 3 = DEGRADED: the gate ran no check at all. Not a pass, not a failure — the
run continues and the claim shrinks.
staticgreennever configured: …PARTIALgate.sh allkeeps the ordering: a failure wins, a degradation survives, so it cannot launder a 3 intoa 0.
adw_run_cmdnow takes the config key so it can quote the reason the operator wrote:The wording changes, the verdict does not — a declared absence is honest, and it is still no lint
coverage. That distinction already existed in
adw_cfg_noteand was previously visible only todoctor.Callers updated to read three codes
adw-run(GREEN phase + rules),slice-implement,verification-before-completion, theAGENTS.mdblock, the
_STACK.mdtemplate,install.sh's closing instructions, and the/adw-installverification step — where exit 3 is the likeliest outcome of a fresh install and the one that looks
like success if you only skim.
VERSION2.0.0 → 2.1.0: the gate contract changed, anddoctorcompares installs against it.How to verify
The two
check_xfailassertions added in #1 are promoted tocheck, now asserting 3 rather than 1 —"must not report success" is satisfied by a code meaning neither. 15 further assertions cover the
declared-none wording, both partial cases, failure-outranks-degradation, and
all's aggregation.Non-vacuous by mutation:
Dropping
green's both-degraded branch fails 3.Not verified
project whose
_STACK.mdis genuinely sparse, and exit 3 there is the intended answer, not a bug.gate.sh formatstill returns 0 with noFORMAT_CMD. Formatting writes, it does not verify, so ithas no degraded state to report.
## Not verifiedPR section is still written by the agent from what it saw. This makes thedegradation visible to it; it does not mechanically connect
gate readyto which gates degraded.