Add MCP protocol conformance and tool-surface checks to CI - #525
Merged
Conversation
A new mcp-checks CI job runs three structural checks against the built server, none of which involve an LLM. They are driven by the MCPJam CLI, pinned as a devDependency: - a health check of the shipped artifact over stdio - MCP protocol conformance (pinned to 2026-07-28) against the HTTP transport - a tool-surface diff against the PR base, failing on breaking changes The health check and conformance run are also available locally as `npm run check:mcp`. Both use a checked-in minimal wiki configuration so results do not depend on the developer's config.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cleanup path awaited a child exit event that could already have fired, hanging the promise chain and letting Node exit 0 after a server crash — a crashed HTTP transport passed the check. The exit promise is now armed at spawn so cleanup always resolves, a pre-spawn probe rejects a port already occupied by another process instead of running conformance against it, and the doctor's spawned server is pinned to the stdio transport so an exported MCP_TRANSPORT cannot skew it. Also caps the mcp-checks CI job at 15 minutes and documents the conformance port and its PORT override. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
alistair3149
force-pushed
the
mcpjam-ci-checks
branch
from
July 31, 2026 22:12
26546e3 to
cd99149
Compare
alistair3149
changed the title
alistair3149
marked this pull request as ready for review
July 31, 2026 22:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a second CI job,
mcp-checks, running three structural checks against the built server, driven by the MCPJam CLI (an MCP server testing tool, pinned as a devDependency). None of the checks involve an LLM, so the job costs nothing beyond ~a minute of runner time.dist/artifact actually starts, completes an MCP handshake, and serves its tools and resources. Nothing else in CI boots the built artifact.2026-07-28) against the HTTP transport — an external implementation checking the spec-compliance surface. Currently 20/31 checks pass, 0 fail, 11 skip (legacy-era and SSE-session checks that don't apply to the stateless deployment).--fail-on breaking); description changes appear in the report without failing.The health check and conformance run are also available locally as
npm run check:mcp(build first). Both CI and the local script use the checked-in minimal wiki configuration inscripts/mcp-checks.config.json, so results are deterministic and independent of the developer'sconfig.json. Telemetry is disabled on every invocation.Preflight is deliberately untouched: these checks need a booted server, a port, and (for the diff) a base checkout, none of which belong in the hermetic local release gate.
To review
--fail-on breakingis the right diff policy, or whether description changes should also fail (--fail-on any).PORToverrides it.Verified
npm run check:mcppasses end-to-end locally (health check ready with 27 tools, conformance 20/31 passed, 0 failed). The exact export/diff commands from the workflow were run locally against the built server: identical surfaces exit 0, and a mutated baseline (removed tool) exits 1 under--fail-on breaking. Failure paths verified to actually fail: a crashed server process, an already-occupied port, and a pollutedMCP_TRANSPORTenvironment each produce the intended outcome. Lint, typecheck, fmt, and the full test suite pass.🤖 Generated with Claude Code