Add MCP protocol conformance and tool-surface checks to CI (#525)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: alistair3149

.github/workflows/ci.yml

@@ -21,4 +21,54 @@ jobs:
       run: npm run preflight
 
     - name: Ensure no changes
-      run: git diff --exit-code
+      run: git diff --exit-code
+
+  mcp-checks:
+    runs-on: ubuntu-24.04-arm
+    timeout-minutes: 15
+    steps:
+    - uses: actions/checkout@v7
+
+    - uses: actions/setup-node@v7
+      with:
+        node-version: 24
+        cache: 'npm'
+
+    - run: npm ci
+
+    - run: npm run build
+
+    - name: Doctor and protocol conformance
+      run: npm run check:mcp
+
+    - name: Check out PR base
+      if: github.event_name == 'pull_request'
+      uses: actions/checkout@v7
+      with:
+        ref: ${{ github.event.pull_request.base.sha }}
+        path: base
+
+    - name: Build PR base
+      if: github.event_name == 'pull_request'
+      working-directory: base
+      run: npm ci && npm run build
+
+    - name: Export base tool surface
+      if: github.event_name == 'pull_request'
+      run: >
+        node_modules/.bin/mcpjam server export --stable
+        --transport stdio --command node
+        --args "$GITHUB_WORKSPACE/base/dist/index.js"
+        --cwd "$GITHUB_WORKSPACE/base"
+        --env "CONFIG=$GITHUB_WORKSPACE/scripts/mcp-checks.config.json"
+        --no-telemetry --quiet > base-surface.json
+
+    - name: Diff tool surface against base
+      if: github.event_name == 'pull_request'
+      run: >
+        node_modules/.bin/mcpjam server diff --baseline base-surface.json
+        --transport stdio --command node
+        --args "$GITHUB_WORKSPACE/dist/index.js"
+        --cwd "$GITHUB_WORKSPACE"
+        --env "CONFIG=$GITHUB_WORKSPACE/scripts/mcp-checks.config.json"
+        --fail-on breaking --no-telemetry

AGENTS.md

@@ -34,6 +34,7 @@ A path handed to a runtime `createRequire( import.meta.url )` resolves against t
 - `npm test` — run the vitest suite once.
 - `npm run lint` — oxlint.
 - `npm run typecheck` — `tsc --noEmit` over `src` **and** `tests`, via `tsconfig.lint.json`. `npm run build` only typechecks `src`, so this is the check that covers test code.
+- `npm run check:mcp` — MCPJam doctor + MCP protocol conformance against the built server (build first). CI runs it on every PR and also diffs the tool surface against the PR base, failing on breaking changes.
 - `npm run fmt` / `npm run fmt:check` — oxfmt (write / dry-run).
 - `npm run preflight` — full gate (install, lint, typecheck, fmt check, validate `server.json`, test, build, bundle). Run before a release.
 - Git hooks: `lefthook` auto-installs on `npm install`. Pre-commit runs `oxfmt` (auto-fix on staged files) + `oxlint`. Pre-push runs `npm run typecheck` + the test suite. Bypass with `--no-verify`.

docs/testing.md

@@ -70,6 +70,16 @@ Like the MCP Inspector, but with a built-in MCP client that can drive the server
 npm run mcpjam
 ```
 
+## MCPJam CLI checks
+
+Token-free structural checks against the built server: a stdio doctor sweep plus an MCP protocol conformance run against the HTTP transport. Build first with `npm run build`, then:
+
+```sh
+npm run check:mcp
+```
+
+Both use the minimal wiki configuration in `scripts/mcp-checks.config.json`, so results do not depend on your local `config.json`. The conformance run binds `127.0.0.1:3117`; set `PORT` to use a different port. CI runs the same script on every pull request, and additionally diffs the tool surface against the PR base — a removed or renamed tool fails the job, while description changes only show up in the report.
+
 ## MCP Inspector CLI (integration tests)
 
 The [MCP Inspector CLI](https://github.com/modelcontextprotocol/inspector) exercises tools against a real wiki. Build first with `npm run build`, then: