Judge the conformance report instead of the exit code

[alistair3149]

The MCPJam conformance suite withholds its verdict whenever a check could not run, and exits with the same code it uses for a protocol violation. Two checks can never run against this server, so the exit code alone turns a clean run red.

Neither check reflects a gap to close. One needs a tool that asks the caller for input, so that it can prove the server rejects a client which never declared that capability, and it gives up before it contacts the server at all. The other waits for the completion result a server sends when it tears a subscription down, which this server does send on shutdown, but the check only aborts its own end of the stream and so can never observe one.

Parse the report and judge it instead: fail on any check that failed, on any check that could not run beyond those two, and on a run too small to establish anything. That last guard matters because the suite reports an empty check list as an incomplete run, which an allowlist alone would wave through as a pass.

A run now also names the checks it tolerated, so a green result is not mistaken for full conformance.

Why this is needed

mcp-checks is red on #528. The visible bump there is @mcpjam/cli 3.16.0 to 3.17.0, but the conformance suite lives in @mcpjam/sdk, which the CLI depends on through a caret range. 3.17.0 only widens that range, so the lockfile re-resolves the SDK 2.1.0 to 2.4.0, and 2.4.0 changed passed from "no check failed" to "the outcome is passed". That run reports 0 failed checks.

This is not specific to #528. package.json declares ^3.16.0 with no overrides, so any lockfile regeneration picks up SDK 2.4.0 and turns the job red with no bump involved. Master is green only because npm ci replays a lock that happens to pin 2.1.0.

Verified

• npm run check:mcp exits 0 under CLI 3.16.0/SDK 2.1.0 (this branch as it stands), and under 3.17.0/2.4.0 and 3.19.0/2.4.0 in throwaway copies. The raw CLI exits 1 and 3 respectively in those last two.
• Emptying the allowlist makes it exit 1 naming both checks, so the allowlist is what makes it green rather than a vacuous parse.
• New unit tests cover the injected-failure, non-allowlisted, empty-report and legacy-report paths. Written first and seen failing.
• npm run preflight green; 1753 tests pass.

CI on this PR does not exercise the new path. The branch still pins SDK 2.1.0, whose report carries no outcome and no skip reasons, so CI takes the legacy branch of the judge. The 2.4.0 path was only verified locally.

What you need to decide

Whether to retarget #528 at 3.19.0 rather than 3.17.0. 3.19.0 is current, and upstream added the distinct incomplete exit code 3 there. Either version goes green once this merges; 3.19.0 alone without this change does not, it only turns exit 1 into exit 3.

_{AI-authored — Claude Code, Opus 5 1M (ultracode); one-line ask from @alistair3149 to investigate the CI failure, fix approach then chosen from offered options, no revisions; diff not yet human-reviewed; tests written TDD and run locally, preflight green, newer-SDK path verified in local copies only.}