Skip to content

feat(gates): a check that never ran is not a check that passed - #2

Merged
ruslan-korneev merged 1 commit into
harness-self-testsfrom
gates-degraded
Jul 27, 2026
Merged

feat(gates): a check that never ran is not a check that passed#2
ruslan-korneev merged 1 commit into
harness-self-testsfrom
gates-degraded

Conversation

@ruslan-korneev

Copy link
Copy Markdown
Owner

Stacked on #1 — review that first, or read this diff alone since it only touches gate.sh, lib.sh
and one test case.

Closes RUS-187.

The problem

gate.sh static returned 0 when FORMAT_CHECK_CMD, LINT_CMD and TYPECHECK_CMD were all unset, and
gate.sh green inherited it. Each unconfigured command was announced as SKIPPED and contributed 0 to
the verdict, so a project with an empty _STACK.md finished green having verified nothing
indistinguishable, by exit code, from one where all three ran and passed.

The harness already cures this one layer up: an engine CLI on PATH is not usable until it answers a
probe, because "a missing review lens looks exactly like a clean one". Same disease, at home.

The change

A third outcome. Exit 3 = DEGRADED: the gate ran no check at all. Not a pass, not a failure — the
run continues and the claim shrinks.

static green
something failed 1 1
nothing ran 3 3 (neither half ran)
partial: some ran, some unconfigured 0 + never configured: … 0 + PARTIAL
everything ran and passed 0 0

gate.sh all keeps the ordering: a failure wins, a degradation survives, so it cannot launder a 3 into
a 0.

adw_run_cmd now takes the config key so it can quote the reason the operator wrote:

LINT_CMD=      # none: no linter for Luau
[adw] lint: SKIPPED — declared none in .tasks/_STACK.md (none: no linter for Luau)

The wording changes, the verdict does not — a declared absence is honest, and it is still no lint
coverage. That distinction already existed in adw_cfg_note and was previously visible only to
doctor.

Callers updated to read three codes

adw-run (GREEN phase + rules), slice-implement, verification-before-completion, the AGENTS.md
block, the _STACK.md template, install.sh's closing instructions, and the /adw-install
verification step — where exit 3 is the likeliest outcome of a fresh install and the one that looks
like success if you only skim.

VERSION 2.0.0 → 2.1.0: the gate contract changed, and doctor compares installs against it.

How to verify

bash tests/run.sh                          # 118 passed, 0 failed, 1 known gap
ADW_TEST_SHELL=/bin/bash bash tests/run.sh # same under macOS bash 3.2

The two check_xfail assertions added in #1 are promoted to check, now asserting 3 rather than 1 —
"must not report success" is satisfied by a code meaning neither. 15 further assertions cover the
declared-none wording, both partial cases, failure-outranks-degradation, and all's aggregation.

Non-vacuous by mutation:

# put the swallowed degradation back
perl -0pi -e 's/(adw_warn "DEGRADED: static.*?\n)    return \$ADW_SKIPPED/$1    return 0/s' scripts/gate.sh
bash tests/run.sh gate-static   # 6 assertions fail
git checkout scripts/gate.sh

Dropping green's both-degraded branch fails 3.

Not verified

  • No real project has been run against this yet — only fixtures. The first install to hit it will be a
    project whose _STACK.md is genuinely sparse, and exit 3 there is the intended answer, not a bug.
  • gate.sh format still returns 0 with no FORMAT_CMD. Formatting writes, it does not verify, so it
    has no degraded state to report.
  • The ## Not verified PR section is still written by the agent from what it saw. This makes the
    degradation visible to it; it does not mechanically connect gate ready to which gates degraded.

`gate.sh static` returned 0 with FORMAT_CHECK_CMD, LINT_CMD and TYPECHECK_CMD all unset, and
`gate.sh green` inherited it. Every unconfigured command was announced as SKIPPED and contributed 0
to the verdict, so a project with an empty _STACK.md finished green having verified nothing — by
exit code, indistinguishable from a project where all three ran and passed. That is the same disease
the engine probe exists to cure one layer up ("installed does not mean usable"), and the harness had
it at home.

Adds a third outcome. Exit 3 = DEGRADED: the gate ran no check at all. Not a pass, not a failure —
the run continues and the claim shrinks.

- lib.sh: adw_run_cmd returns 3 for an unconfigured command, and takes the config key so it can quote
  the operator's stated reason. `LINT_CMD=   # none: no linter for Luau` now reads "declared none"
  rather than "not configured". The wording changes; the verdict does not, because no linter is still
  no lint coverage.
- gate.sh static: 1 if anything failed · 3 if nothing ran · 0 otherwise, naming what was skipped.
  Partial coverage still passes — something did run — but it says so, because the PR has to.
- gate.sh green: 3 only when neither half ran. One half missing prints `PARTIAL` and passes; a real
  failure outranks every degradation.
- gate.sh all: a failure wins, a degradation survives, so `all` cannot launder a 3 into a 0.

Callers updated to read three codes instead of two: adw-run (GREEN phase + rules), slice-implement,
verification-before-completion, the AGENTS.md block, the _STACK.md template, install.sh's closing
instructions, and the /adw-install verification step — where exit 3 is the likeliest outcome of a
fresh install and the one that looks like success if you only skim.

Tests: the two `check_xfail` assertions from the previous commit are promoted to `check`, asserting
3 rather than 1 — "must not report success" is satisfied by a code that means neither. 15 further
assertions cover the declared-none wording, both partial cases, failure-outranks-degradation, and
`all`'s aggregation. 29 in that case, 118 in the suite, 0 failures, 1 known gap remaining. Verified
non-vacuous by mutation: restoring the swallowed degradation in gate_static fails 6 assertions,
dropping green's both-degraded branch fails 3.

VERSION 2.0.0 → 2.1.0: the gate contract changed, and `doctor` compares installs against it.

Refs RUS-187
@ruslan-korneev
ruslan-korneev merged commit 99f8a60 into harness-self-tests Jul 27, 2026
2 checks passed
@ruslan-korneev
ruslan-korneev deleted the gates-degraded branch July 27, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant