Merge PR #502: benchmark: one honest snapshot for every perf claim; fix the apparent slowdown

Author: merge-queue-bot

.claude/skills/docs-author/SKILL.md

@@ -210,9 +210,9 @@ adapted to mdsmith.
    feature list. Paste what the tool actually
    prints; never fabricate output.
 4. **One concrete number, not adjectives.** mise:
-   "900+ tools, 1 toml file". mdsmith: "523 files
-   in ~0.2 s, ~10x Node markdownlint", linked to a
-   reproducible benchmark.
+   "900+ tools, 1 toml file". mdsmith: "checks its
+   Markdown in ~0.5 s, ~10x faster than Node
+   markdownlint", linked to a reproducible benchmark.
 5. **Short pitch, then detail.** Three to five
    benefit-named groups, one line each, before any
    long card list. Never open with the full

.github/copilot-instructions.md

@@ -87,7 +87,7 @@ row: "- [{summary}](../{filename})"
 - [`mdsmith lsp` emits diagnostics, quick-fixes, and navigation — definition, references, symbol search, and a call-hierarchy over `<?include?>`, `<?catalog?>`, and cross-file links — consumed by any LSP-aware editor.](../docs/features/live-diagnostics.md)
 - [`mdsmith extract` projects a schema-conformant Markdown file into a JSON, YAML, or msgpack data tree; `mdsmith export` writes a portable, directive-free copy that renders anywhere.](../docs/features/markdown-as-data.md)
 - [Pin a Markdown convention to get a curated rule preset and a target renderer flavor in one switch. `MDS034` flags syntax the flavor will not render; a placeholder vocabulary spares template tokens.](../docs/features/markdown-conventions.md)
-- [A single static Go binary, no runtime to start. The workspace walk runs in parallel, embeds are linted once, and `check` is built for the hot path — roughly 4x faster than Node markdownlint, with a CI gate against regression.](../docs/features/performance.md)
+- [A single static Go binary, no runtime to start. The workspace walk runs in parallel, embeds are linted once, and `check` is built for the hot path — an order of magnitude faster than Node markdownlint, with a CI gate against regression.](../docs/features/performance.md)
 - [CI badge, Go Report Card grade, and Codecov coverage badge report live project health. mdsmith lints its own docs with the rules it ships, and a coverage gate blocks any merge that drops below the line.](../docs/features/quality.md)
 - [`mdsmith list query 'status: "✅"' plan/` selects files by a CUE expression on front matter; `mdsmith metrics rank` ranks files by any shared metric — both ready to pipe into a release script.](../docs/features/release-gating.md)
 - [Rename a heading and every workspace anchor link that points at it is rewritten in one atomic edit. Link-reference labels rename with their uses. A colliding slug fails loudly instead of silently breaking cross-file links.](../docs/features/rename.md)

.github/workflows/benchmark.yml

@@ -5,10 +5,15 @@ name: Cross-tool benchmark
 # pattern): both the refreshed numbers (data + fragments) and
 # the benchmark Markdown pages rendered from main's READMEs with
 # those numbers spliced in. Record-only: it never gates a merge
-# or a release, and it never pushes to protected main — the
-# in-repo committed snapshot stays the `bench-fragments` gate's
-# source of truth, hand-refreshed via run.sh. The website pulls
-# the assets numbers at build time.
+# or a release, and it never pushes to protected main. The
+# in-repo committed snapshot — hand-refreshed via run.sh and
+# reviewed in a PR — stays the `bench-fragments` gate's source of
+# truth AND the numbers the website reads at build time. This
+# per-merge run re-measures on a noisy shared runner, so it is a
+# record-only drift signal (a demo.gif-style artifact on the
+# assets branch), NOT a live source the website or docs read; if
+# it fed the published figures they would swing run-to-run with
+# whatever else shared the runner.
 
 on:
   push:

.github/workflows/pages.yml

@@ -86,10 +86,11 @@ jobs:
     # before publishing.
     #
     # deploy re-runs the same sync and lint after the
-    # version-stamp + site-assets pulls (which mutate docs/
-    # before the second sync), so the deploy's second pass
-    # also covers anything those steps could break — work that
-    # build-content deliberately skips on PR.
+    # version-stamp step (which mutates tracked manifests before
+    # the second sync), so the deploy's second pass also covers
+    # anything that step could break — work that build-content
+    # deliberately skips on PR. The demo-GIF pull lands in
+    # website/static/img/ and never touches docs/.
     #
     # No Pages permissions — this job never touches the Pages
     # API; deploy carries those separately.
@@ -259,25 +260,21 @@ jobs:
         # resolved version, so the deployed site shows it.
         if: env.SITE_VERSION != ''
         run: go run ./cmd/mdsmith-release stamp "$SITE_VERSION"
-      - name: Pull published demo + benchmark numbers
-        # The demo GIF and the cross-tool benchmark numbers are
-        # regenerated post-merge and pushed to the orphan assets
-        # branch (.github/workflows/{demo,benchmark}.yml), never
-        # committed. Pull them in so the deployed site serves the
-        # current GIF as a first-party asset and bakes the live
-        # benchmark table into the comparison page. The committed
-        # fragment snapshot is the fallback while the assets
-        # branch has not published numbers yet; the
-        # `bench-fragments` gate (a separate workflow) still
-        # validates that committed snapshot. The scoped
-        # `mdsmith fix` refreshes only the one page's <?include?>
-        # body from the freshly pulled fragment — the intended,
-        # minimal exception to build-website's --no-fix. See
-        # docs/development/release-tooling.md for why the pull is
-        # a Go subcommand rather than inline shell.
-        run: |
-          go run ./cmd/mdsmith-release pull-site-assets
-          go run ./cmd/mdsmith fix docs/background/markdown-linters.md
+      - name: Pull published demo GIF
+        # The demo GIF is regenerated post-merge and pushed to the
+        # orphan assets branch (.github/workflows/demo.yml), never
+        # committed. Pull it in so the deployed site serves the
+        # current GIF as a first-party asset. The cross-tool
+        # benchmark numbers are NOT pulled: they come from the
+        # committed in-repo snapshot under docs/research/benchmarks/
+        # (refreshed via run.sh, reviewed in a PR), so the noisy
+        # per-merge benchmark.yml re-measurement stays a record-only
+        # drift signal and never moves the published figures. The
+        # `bench-fragments` gate (a separate workflow) validates that
+        # committed snapshot against the committed JSON. See
+        # docs/development/release-tooling.md for why the pull is a
+        # Go subcommand rather than inline shell.
+        run: go run ./cmd/mdsmith-release pull-site-assets
       - name: Build Hugo content tree from docs
         # Snapshot ./docs into ./website/content/docs, dropping
         # proto.md schema templates, renaming index.md to
@@ -298,10 +295,9 @@ jobs:
         # the rationale for each disabled rule. Re-runs here as
         # defense-in-depth: build-content already ran the same
         # check, but deploy's own sync above happens AFTER the
-        # version-stamp and `mdsmith fix
-        # docs/background/markdown-linters.md` steps that mutate
-        # the docs/ tree, so this pass is the only one that
-        # covers the post-mutation result.
+        # version-stamp step that mutates tracked manifests, so
+        # this pass is the only one that covers the post-stamp
+        # result.
         run: |
           go run ./cmd/mdsmith check \
             --config ./website/build-output.mdsmith.yml \

.github/workflows/release.yml

@@ -871,3 +871,131 @@ jobs:
     uses: ./.github/workflows/pages.yml
     with:
       version: ${{ inputs.version }}
+
+  # Re-measure the cross-tool benchmark on every release and open a PR
+  # with the refreshed numbers and the pages that embed them, so the
+  # committed snapshot the README, the benchmark doc, and the website
+  # read never freezes between releases (there is no local run.sh step
+  # in the normal flow). bench-check is the regression gate: it compares
+  # mdsmith's ratio to mado -- machine- and corpus-size-independent, the
+  # "same code, same factor" signal -- and fails this job, visibly, on a
+  # real relative slowdown. It runs AFTER `release` and is intentionally
+  # not in the publish gating chain: the cross-tool fetch (pinned
+  # tarballs, npm, Rust clones) has enough external moving parts that a
+  # flake should not block shipping. The PR lands through the normal
+  # review-and-merge path; the bench-fragments gate re-checks the
+  # fragments against the data when the PR is enqueued in the merge
+  # queue, before it merges. (Like record-secret-rotation.yml, the PR
+  # is opened with the default token, so ci.yml does not auto-run on
+  # PR open — the merge queue is where the gate fires.)
+  benchmark-refresh:
+    needs: [release]
+    if: *release_repo_trigger_ok
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    # contents: write to push the refresh branch; pull-requests: write
+    # for `gh pr create`. The change to main still goes through the
+    # normal review-and-merge path -- this token only opens the PR.
+    permissions:
+      contents: write
+      pull-requests: write
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          # Never persist the token in git config; the push step
+          # authenticates explicitly via a one-shot, masked header.
+          persist-credentials: false
+      - uses: actions/setup-go@7a3fe6cf4cb3a834922a1244abfce67bcef6a0c5 # v6.2.0
+        with:
+          go-version-file: go.mod
+          cache: false
+      - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          # markdownlint-cli2 installs via `npm ci` from the committed
+          # docs/research/benchmarks/npm/ lockfile.
+          node-version: "24"
+      - name: Snapshot the committed baseline
+        # The committed data/ is the previous release's numbers. Keep a
+        # copy so bench-check can compare the fresh ratios against it
+        # after the measurement overwrites data/ in place.
+        run: |
+          # rm first so a re-run on a persistent runner replaces the
+          # baseline rather than nesting it under /tmp/baseline-data/data.
+          rm -rf /tmp/baseline-data
+          cp -r docs/research/benchmarks/data /tmp/baseline-data
+      - name: Re-measure the cross-tool benchmark
+        run: go run ./cmd/mdsmith-release bench /tmp/mdsmith-bench
+      - name: Re-splice the fragment includes
+        run: |
+          go run ./cmd/mdsmith fix \
+            docs/research/benchmarks/results.fragment.md \
+            docs/research/benchmarks/headline.fragment.md
+          go run ./cmd/mdsmith fix \
+            README.md docs/research/benchmarks/README.md
+      - name: Check for regression and open the refresh PR
+        shell: bash
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          set -euo pipefail
+          git config user.name  "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+
+          # Ratio-based regression gate: mdsmith vs mado within each
+          # run, so the runner speed and the (growing) corpus cancel and
+          # only a real relative slowdown trips it. Capture the verdict
+          # for the PR body and defer failing the job until after the PR
+          # is opened, so a regressed run still leaves a reviewable diff.
+          verdict=$(go run ./cmd/mdsmith-release bench-check \
+            /tmp/baseline-data docs/research/benchmarks/data 2>&1) && rc=0 || rc=$?
+          echo "$verdict"
+
+          git add docs/research/benchmarks/data \
+                  docs/research/benchmarks/results.fragment.md \
+                  docs/research/benchmarks/headline.fragment.md \
+                  docs/research/benchmarks/README.md \
+                  README.md
+          if git diff --staged --quiet; then
+            echo "::notice::benchmark numbers unchanged -- no PR opened"
+            exit "$rc"
+          fi
+
+          branch="bench/refresh-${VERSION}-${GITHUB_RUN_ID}"
+          git checkout -b "$branch"
+          git commit -m "bench: refresh cross-tool benchmark numbers for ${VERSION}"
+          # Authenticate the push via a masked http.extraheader rather
+          # than embedding the token in the remote URL.
+          auth_header=$(printf 'x-access-token:%s' "$GH_TOKEN" | base64 -w0)
+          echo "::add-mask::$auth_header"
+          git -c "http.https://github.com/.extraheader=AUTHORIZATION: basic ${auth_header}" \
+            push origin "HEAD:refs/heads/${branch}"
+
+          # Build the PR body with echo appends (no heredoc) so dedent
+          # and backtick expansion cannot bite. Single quotes keep the
+          # Markdown backticks literal.
+          {
+            echo 'Automated refresh of the committed cross-tool benchmark snapshot,'
+            echo "re-measured on the CI runner during the ${VERSION} release."
+            echo ''
+            echo 'Review the number diff under `docs/research/benchmarks/data/` and the'
+            echo 'regenerated fragments. Absolute times wobble on a shared runner, so the'
+            echo '**ratio** is the signal -- the regression gate compares mdsmith to mado,'
+            echo 'which cancels both the runner speed and the corpus size:'
+            echo ''
+            echo '```'
+            echo "$verdict"
+            echo '```'
+          } > /tmp/pr-body.md
+          gh pr create \
+            --base main \
+            --head "$branch" \
+            --title "bench: refresh cross-tool benchmark numbers for ${VERSION}" \
+            --body-file /tmp/pr-body.md
+
+          if [ "$rc" -ne 0 ]; then
+            # rc!=0 is either a real regression or bench-check failing to
+            # read the data (missing tool/file); the verdict echoed above
+            # says which. Either way fail the job so it is visible.
+            echo "::error::bench-check failed — perf regression, or it could not read the benchmark data (see the verdict above). The refresh PR opened for review; this run is marked failed."
+            exit "$rc"
+          fi

AGENTS.md

@@ -93,7 +93,7 @@ row: "- [{summary}]({filename})"
 - [`mdsmith lsp` emits diagnostics, quick-fixes, and navigation — definition, references, symbol search, and a call-hierarchy over `<?include?>`, `<?catalog?>`, and cross-file links — consumed by any LSP-aware editor.](docs/features/live-diagnostics.md)
 - [`mdsmith extract` projects a schema-conformant Markdown file into a JSON, YAML, or msgpack data tree; `mdsmith export` writes a portable, directive-free copy that renders anywhere.](docs/features/markdown-as-data.md)
 - [Pin a Markdown convention to get a curated rule preset and a target renderer flavor in one switch. `MDS034` flags syntax the flavor will not render; a placeholder vocabulary spares template tokens.](docs/features/markdown-conventions.md)
-- [A single static Go binary, no runtime to start. The workspace walk runs in parallel, embeds are linted once, and `check` is built for the hot path — roughly 4x faster than Node markdownlint, with a CI gate against regression.](docs/features/performance.md)
+- [A single static Go binary, no runtime to start. The workspace walk runs in parallel, embeds are linted once, and `check` is built for the hot path — an order of magnitude faster than Node markdownlint, with a CI gate against regression.](docs/features/performance.md)
 - [CI badge, Go Report Card grade, and Codecov coverage badge report live project health. mdsmith lints its own docs with the rules it ships, and a coverage gate blocks any merge that drops below the line.](docs/features/quality.md)
 - [`mdsmith list query 'status: "✅"' plan/` selects files by a CUE expression on front matter; `mdsmith metrics rank` ranks files by any shared metric — both ready to pipe into a release script.](docs/features/release-gating.md)
 - [Rename a heading and every workspace anchor link that points at it is rewritten in one atomic edit. Link-reference labels rename with their uses. A colliding slug fails loudly instead of silently breaking cross-file links.](docs/features/rename.md)

CLAUDE.md

@@ -79,7 +79,7 @@ row: "- [{summary}]({filename})"
 - [`mdsmith lsp` emits diagnostics, quick-fixes, and navigation — definition, references, symbol search, and a call-hierarchy over `<?include?>`, `<?catalog?>`, and cross-file links — consumed by any LSP-aware editor.](docs/features/live-diagnostics.md)
 - [`mdsmith extract` projects a schema-conformant Markdown file into a JSON, YAML, or msgpack data tree; `mdsmith export` writes a portable, directive-free copy that renders anywhere.](docs/features/markdown-as-data.md)
 - [Pin a Markdown convention to get a curated rule preset and a target renderer flavor in one switch. `MDS034` flags syntax the flavor will not render; a placeholder vocabulary spares template tokens.](docs/features/markdown-conventions.md)
-- [A single static Go binary, no runtime to start. The workspace walk runs in parallel, embeds are linted once, and `check` is built for the hot path — roughly 4x faster than Node markdownlint, with a CI gate against regression.](docs/features/performance.md)
+- [A single static Go binary, no runtime to start. The workspace walk runs in parallel, embeds are linted once, and `check` is built for the hot path — an order of magnitude faster than Node markdownlint, with a CI gate against regression.](docs/features/performance.md)
 - [CI badge, Go Report Card grade, and Codecov coverage badge report live project health. mdsmith lints its own docs with the rules it ships, and a coverage gate blocks any merge that drops below the line.](docs/features/quality.md)
 - [`mdsmith list query 'status: "✅"' plan/` selects files by a CUE expression on front matter; `mdsmith metrics rank` ranks files by any shared metric — both ready to pipe into a release script.](docs/features/release-gating.md)
 - [Rename a heading and every workspace anchor link that points at it is rewritten in one atomic edit. Link-reference labels rename with their uses. A colliding slug fails loudly instead of silently breaking cross-file links.](docs/features/rename.md)

README.md

@@ -45,8 +45,8 @@ file: docs/research/benchmarks/headline.fragment.md
 <!-- Generated by docs/research/benchmarks/gen_fragments.py from
 docs/research/benchmarks/data/*.json — do not edit by hand. Re-run
 the harness (run.sh) and `mdsmith fix` to refresh. -->
-One static Go binary checks 523 Markdown files in about
-0.2 s. That is roughly 10x faster than Node markdownlint.
+One static Go binary checks 722 Markdown files in about
+0.5 s. That is roughly 9x faster than Node markdownlint.
 It does more per file than the Rust linters; closing the rest of
 that gap is active work.
 <?/include?>
@@ -121,8 +121,8 @@ is republished to Open VSX for Cursor, VSCodium, and Theia.
 **[Fast on every run](docs/features/performance.md).**
 One static Go binary, no runtime to start. The workspace walk runs
 across all cores, and includes are linted once. A full check of
-this repository's ~720 files takes about 1.3 s, roughly 4x faster
-than Node markdownlint.
+this repository's Markdown takes about 0.5 s, an order of
+magnitude faster than Node markdownlint.
 
 **[Installs everywhere](docs/features/install-everywhere.md).**
 The same version-stamped binary ships through go install, npm,