Skip to content

integration.sh: stop reading any nonzero exit as a gateway rejection - #4

Open
MattJackson wants to merge 1 commit into
ci/openapi-shape-testsfrom
fix/false-green-sweep
Open

integration.sh: stop reading any nonzero exit as a gateway rejection#4
MattJackson wants to merge 1 commit into
ci/openapi-shape-testsfrom
fix/false-green-sweep

Conversation

@MattJackson

Copy link
Copy Markdown
Contributor

No check goes red as a result of this change. Part of a fleet-wide sweep for checks that report GREEN without having checked anything (21 found, 17 proven).

The finding

The else branch treated any nonzero exit from the gateway call as the expected rejection. Run against a stub binary with no config apply subcommand at all — a plausible state after a CLI rename — and the script printed INTEGRATION OK and exited 0, having proven nothing about the gateway. Only a recognised rejection is tolerated now; anything else fails.

Proven: constructed the stub, captured INTEGRATION OK / exit 0; the fixed script fails against that same stub and passes against a real gateway.

On spec-drift — not fixed here, and it stays on the list

Another agent owns this. Recording what I verified, since it is useful evidence for them:

  • The gate still compares info.version strings, and still exits 0 when the GitHub API is unreachable.
  • The compensating control is sound. I audited the 769-line conformance suite that shipped on this branch: it checks actual response shapes, not version numbers, and it does not have the vacuous-loop or skip-to-green defects this sweep was hunting. The risk is genuinely contained today.
  • It nonetheless stays on the list. A version string is a proxy: both files can claim 1.5.3 identically while having diverged, which is exactly how this was found. The gate is green about a property it does not check, and being compensated elsewhere does not make the gate correct.

The else branch treated ANY nonzero exit from the gateway call as the expected
rejection. Run against a stub binary with no `config apply` subcommand at all
— a plausible state after a CLI rename — and the script printed INTEGRATION OK
and exited 0, having proven nothing about the gateway. Only a real rejection
(recognised by its message) is tolerated now; anything else fails.

Unrelated but worth recording, since another agent owns the fix: the spec-drift
job still compares info.version strings and still exits 0 when the GitHub API
is unreachable. I verified the compensating control — the 769-line conformance
suite that shipped on this branch — and it is sound: it checks shapes, not
version strings, so the risk is contained today. The gate itself is still green
about a property it does not check, so it stays on the list until it compares
shapes rather than a version number that both sides can claim identically while
having diverged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant