From d58b59dd4af82f4709643812abc0e001b4b14b0b Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 22 Jun 2026 21:08:25 +0000 Subject: [PATCH 1/7] docs(benchmarks): restructure the benchmark write-up for scannability The benchmark research README is the source of the published assets/benchmarks/pages/benchmark.md page. It read as walls of text: long bold-label paragraphs and 30-50 word sentences packed under a single "Reading the result" H2. Restructure for scannability without changing any measured fact: - Split the intro wall into a lead plus a two-source list. - Convert the "Reading the result" bold-label paragraphs into real subsections; list the two profiler-loop bugs and the per-corpus parity results. - Group the methodology caveats under a new "Measurement notes and fairness" H2 and promote "Gates" to its own H2. - Tighten sentences to the background voice and drop promotional wording (honest, crushes, home turf). Only prose outside the blocks changed; the generated fragment bodies and data/*.json are untouched, so bench-fragments stays green. `mdsmith check .` passes. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4 --- docs/research/benchmarks/README.md | 648 +++++++++++++++-------------- 1 file changed, 332 insertions(+), 316 deletions(-) diff --git a/docs/research/benchmarks/README.md b/docs/research/benchmarks/README.md index 8070b97e1..a2ce6020f 100644 --- a/docs/research/benchmarks/README.md +++ b/docs/research/benchmarks/README.md @@ -2,55 +2,57 @@ summary: >- First-party hyperfine benchmark of mdsmith against gomarklint, mado, rumdl, panache, and markdownlint-cli2 over - two corpora, with the exact commands, environment, and an - honest reading of where mdsmith trails the lighter per-file - linters. + two corpora, with the exact commands, the environment, and + where mdsmith trails the lighter per-file linters. --- # Markdown linter benchmark Our own benchmark run, not a re-quote of each project's README. Reproduce it with [`run.sh`](run.sh), a thin wrapper over -`mdsmith-release bench`. The published numbers come from the -committed `data/*.json` snapshot in this directory: it is the -`bench-fragments` gate's source of truth, what the tables below -render from in-repo, and what the website reads at build time. You -refresh it deliberately by running `run.sh` and reviewing the -result in a PR. The same harness also runs on every merge to -`main`, publishing a re-measured copy to the orphan `assets` -branch — but that per-merge run is a record-only drift signal (the -demo.gif-style artifact), not a live source the website or the -docs read. +`mdsmith-release bench`. + +Two copies of the numbers exist, and they do different jobs: + +- **The committed `data/*.json` snapshot** in this directory is + the published source. The tables below render from it in-repo, + the website reads it at build time, and the `bench-fragments` + gate checks it. A maintainer refreshes it on purpose: run + `run.sh`, then review the result in a PR. +- **The per-merge `assets`-branch copy** re-measures on every + merge to `main`, the same way `demo.gif` is rebuilt. It is a + record-only drift signal, never a live source the website or + these docs read. ## Method -- Driver: `hyperfine` 1.20.0, `--warmup 3 --runs 10 -N` - (markdownlint-cli2: `--warmup 2 --runs 6`). -- Each tool runs its check/lint over a directory of `.md` - files on its built-in defaults. The exceptions are the - `mdsmith--parity` rows: the same mdsmith binary run - with a per-peer +- **Driver.** `hyperfine` 1.20.0 with `--warmup 3 --runs 10 + -N`. markdownlint-cli2 uses `--warmup 2 --runs 6`. +- **What each tool runs.** Every tool checks a directory of + `.md` files on its built-in defaults. `mdsmith` with no `-c` + is the full default set users run. +- **The parity rows.** Each `mdsmith--parity` row is + the same mdsmith binary run with a per-peer [`bench--parity.mdsmith.yml`](bench-gomarklint-parity.mdsmith.yml) - profile, each matching one peer's default rule set for a - like-for-like row. `mdsmith` (no `-c`) is the full default - set users actually run. (The committed table still shows the - pre-split single `mdsmith-parity` column; the per-linter rows - land on the next benchmark run.) -- Caches disabled for the tools that have one (`rumdl - --no-cache`, `panache --no-cache`) so every run is - worst-case cold. mdsmith, mado, gomarklint, and - markdownlint-cli2 keep no on-disk cache. -- `mdsmith` vs the markdownlint tools is not like-for-like - (it does more per file); each `mdsmith--parity` row - vs its peer is the like-for-like one, with one residual - asymmetry noted in + profile that matches one peer's default rule set. (The + committed table still shows the pre-split single + `mdsmith-parity` column; the per-linter rows land on the next + run.) +- **Like-for-like.** `mdsmith` on full defaults does more per + file, so that row is not like-for-like. Each + `mdsmith--parity` row against its peer is. One + residual asymmetry remains; see [Reading the result](#reading-the-result). -- Integrity: every comparison binary is fetched at a pinned - version and verified by SHA-256 before it runs. - gomarklint, hyperfine, mado, panache, and rumdl come from - pinned GitHub release tarballs (rumdl moved off an - unpinned `uv tool install`); markdownlint-cli2 installs - via `npm ci` from the committed lockfile in `npm/`. A - tampered or silently-rebuilt download fails the run loud. +- **Cold cache.** Caches are disabled for the tools that have + one (`rumdl --no-cache`, `panache --no-cache`), so every run + is worst-case cold. mdsmith, mado, gomarklint, and + markdownlint-cli2 keep no on-disk cache. +- **Integrity.** Every comparison binary is fetched at a pinned + version and verified by SHA-256 before it runs. gomarklint, + hyperfine, mado, panache, and rumdl come from pinned GitHub + release tarballs (rumdl moved off an unpinned `uv tool + install`). markdownlint-cli2 installs via `npm ci` from the + committed lockfile in `npm/`. A tampered or silently-rebuilt + download fails the run loudly. ### Corpora @@ -61,43 +63,43 @@ docs read. ### Environment -- GitHub Actions `ubuntu-latest`: a shared 4-vCPU runner - (Linux 6.18.5 x86_64, Intel Xeon @ 2.10 GHz). Shared, not - dedicated — neighbouring load on the host moves the absolute - numbers, which is why the per-tool ratio within one run, not - the cross-tool absolute time across runs, is the signal (see - [Why absolute numbers move](#why-absolute-numbers-move-and-how-the-factor-stays-stable)). -- mdsmith (Go 1.25.8 build), gomarklint 3.2.3, mado 0.3.0, - rumdl 0.1.93, panache 2.46.0, markdownlint-cli2 0.22.1 - (markdownlint 0.40.0) -- Date: 2026-06-12 (the v0.43.0 release run's - `benchmark-publish` measurement, promoted from the - `assets` branch so the committed baseline shares the - release runner's environment) +- Runner: GitHub Actions `ubuntu-latest`, a shared 4-vCPU host + (Linux 6.18.5 x86_64, Intel Xeon @ 2.10 GHz). It is shared, + not dedicated, so a neighbour's load moves the absolute + numbers. Read the per-tool ratio within one run, not the + absolute time across runs; see + [Why the absolute numbers move](#why-the-absolute-numbers-move). +- Tools: mdsmith (Go 1.25.8 build), gomarklint 3.2.3, mado + 0.3.0, rumdl 0.1.93, panache 2.46.0, markdownlint-cli2 0.22.1 + (markdownlint 0.40.0). +- Date: 2026-06-12 — the v0.43.0 release run's + `benchmark-publish` measurement, promoted from the `assets` + branch so the committed baseline shares the release runner's + environment. ### No PGO in the benchmark build -The `mdsmith` binary the harness builds is a plain -`go build ./cmd/mdsmith` with no profile-guided -optimization, even though the release pipeline now builds the -shipped binaries with a generated profile (see -[the PGO page](../../development/pgo-profile.md)). The two are -deliberately separate. The benchmark measures a reproducible -build — one whose number depends only on the engine and the -corpus, not on a profile recorded earlier in the same run — -so a profile refresh cannot move the published figures -independently of an engine change. PGO measured within noise -(~0-2%) on this workload, so building the benchmark binary -without it costs the comparison nothing meaningful while -keeping the number honest. The released binaries carry the -profile; the benchmark binary does not. +The harness builds `mdsmith` with a plain +`go build ./cmd/mdsmith` and no profile-guided optimization. +The release pipeline does build the shipped binaries with a +generated profile (see +[the PGO page](../../development/pgo-profile.md)), so the +released binary carries the profile and the benchmark binary +does not. That split is deliberate. + +A profile-free build keeps the number reproducible: it depends +only on the engine and the corpus, not on a profile recorded +earlier in the same run. So a profile refresh cannot move the +published figures without an engine change behind it. PGO +measured within noise (~0-2%) on this workload, so dropping it +from the benchmark binary costs the comparison nothing. ## Results -Numbers below are spliced from -[`results.fragment.md`](results.fragment.md), which -`run.sh` regenerates from the hyperfine JSON. They are not -hand-maintained — re-run the harness and `mdsmith fix` to +The tables below are spliced from +[`results.fragment.md`](results.fragment.md), which `run.sh` +regenerates from the hyperfine JSON. They are not +hand-maintained. Re-run the harness and `mdsmith fix` to refresh. -parity` convention runs. +The parity claim above hinges on which rules each configuration +runs. For the full mdsmith-to-peer rule mapping (markdownlint, +rumdl, mado, panache, obsidian-linter), with each peer's +upstream default-enabled state, see the [peer-linter coverage +matrix](../markdownlint-coverage/README.md). It is generated +from rule front matter, so it stays in sync with the actual rule +set. This subsection adds the benchmark-specific layer: which +rules `mdsmith` (full) runs, and which each `-parity` +convention runs. | Configuration | Rule class | | ------------------- | ------------------------------------------------------------ | @@ -353,19 +376,18 @@ runs and which each `-parity` convention runs. #### Per-linter parity rule sets -Each `-parity` convention runs the rule set its -peer enables by default. It turns on the mdsmith opt-in -rules the peer runs. It turns off the mdsmith defaults -the peer skips. The single `mdsmith-parity` column uses -`mado-parity`, the mid-size peer. The per-linter +Each `-parity` convention runs the rule set its peer +enables by default. It turns on the mdsmith opt-in rules the +peer runs, and turns off the mdsmith defaults the peer skips. +The single `mdsmith-parity` column uses `mado-parity`, the +mid-size peer. The per-linter [`bench--parity.mdsmith.yml`](bench-mado-parity.mdsmith.yml) profiles drive a head-to-head run against each peer. -The tables are generated from the conventions +These tables are generated from the conventions (`mdsmith-release sync-parity-rules`). CI checks each set -against the coverage matrix, so neither can drift — the -same fragment the [conventions reference][conv-parity] -embeds: +against the coverage matrix, so neither can drift. It is the +same fragment the [conventions reference][conv-parity] embeds: Date: Mon, 22 Jun 2026 22:41:23 +0000 Subject: [PATCH 2/7] docs(benchmarks): correct stale comparative claims and counts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The interpretive prose predated the current measured numbers and the MDS070 addition. Fix the claims against the committed results table and the live rule set. Benchmark README: - Full mdsmith (214/154 ms) now beats rumdl (324/195) and panache on both corpora, so "same class as the per-file Rust linters / compare with rumdl" and "pick mado or rumdl for throughput" were wrong. rumdl is not a throughput leader; only mado leads. - gomarklint is "the fastest tool in the benchmark", not "in the table" — the committed results table has no gomarklint row (only the per-merge copy does). - Peer default-coverage counts refreshed to the current coverage matrix the prose cites: rumdl/markdownlint 42->43, gomarklint 21/22->22/23 (MDS070 added a default cover for both, after the 2026-06-12 measurement). markdown-linters.md: - Same rumdl/mado ranking fix in the Benchmarks section ("lands close to rumdl" -> ties mado; "mado and rumdl lead, mdsmith trails" -> mado leads, mdsmith beats rumdl and panache). - mdsmith rule count 67->69 (MDS001-MDS070, MDS060 unused) in the intro prose and the two comparison tables. mado-parity "27 rules" verified correct (43 default-enabled - 24 disabled + 8 opt-in). `mdsmith check .` passes. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4 --- docs/background/markdown-linters.md | 35 +++++++++++++++-------------- docs/research/benchmarks/README.md | 19 ++++++++-------- 2 files changed, 28 insertions(+), 26 deletions(-) diff --git a/docs/background/markdown-linters.md b/docs/background/markdown-linters.md index 005e90497..f35a2c8f5 100644 --- a/docs/background/markdown-linters.md +++ b/docs/background/markdown-linters.md @@ -11,12 +11,12 @@ use of LLMs as linters. ### mdsmith -Go binary with zero runtime deps. It ships 67 rules, -[MDS001][mds001] through [MDS068][mds068] (MDS060 is +Go binary with zero runtime deps. It ships 69 rules, +[MDS001][mds001] through [MDS070][mds070] (MDS060 is unused). They cover structure, readability, cross-file links, and generated content. -26 of the 67 rules are opt-in (off by default), among them +26 of the 69 rules are opt-in (off by default), among them conciseness-scoring ([MDS029][mds029]); the rest run by default. @@ -151,7 +151,7 @@ gates. | Aspect | gomarklint | mdsmith | | ------------- | ---------------------- | -------------------------------- | | Distribution | static Go binary | static Go binary | -| Rule set | 23, kebab-case IDs | 67, `MDSxxx` IDs | +| Rule set | 23, kebab-case IDs | 69, `MDSxxx` IDs | | Autofix | no | `mdsmith fix`, multi-pass | | LSP / editor | no (VS Code planned) | LSP for any editor | | Config | `.gomarklint.json` | `.mdsmith.yml` | @@ -629,7 +629,7 @@ R Markdown constructs the others flatten away. | ----------------------- | ------------ | -------------------- | -------------------- | ------------ | | Language | Go | Rust | Rust | Rust | | Rule IDs | own `MDSxxx` | markdownlint `MDxxx` | markdownlint `MDxxx` | own | -| Rule count | 67 | 71 | ~41 | unenumerated | +| Rule count | 69 | 71 | ~41 | unenumerated | | Autofix / format | `fix` | `--fix`, `fmt` | no | `format` | | LSP / editor | yes (LSP) | yes (LSP) | no | yes (LSP) | | Config format | YAML | TOML | TOML | TOML | @@ -757,19 +757,20 @@ access and is non-deterministic. ## Benchmarks -Default mdsmith is substantially faster than -markdownlint-cli2 on its own Markdown. With the -mdsmith-only rules disabled (`mdsmith-parity`), it lands -close to rumdl. See the [benchmark doc][bench] for the -current ratios. +Default mdsmith is more than an order of magnitude faster +than markdownlint-cli2 on its own Markdown. With the +mdsmith-only rules disabled (`mdsmith-parity`), it runs at +mado's speed: the two tie on the repo corpus. See the +[benchmark doc][bench] for the current ratios. -mado and rumdl lead the per-file race. mdsmith trails -them today because it also walks the cross-file graph, -scores readability, and validates generated sections. +mado leads the per-file race. Default mdsmith does more +per file: it also walks the cross-file graph, scores +readability, and validates generated sections. Even so, it +comes in ahead of rumdl and panache on both corpora. -Pick mado or rumdl for raw markdownlint throughput. -Pick panache for Quarto or R Markdown. Pick mdsmith for -the cross-file and self-maintaining-section layer. +Pick mado for raw markdownlint throughput. Pick panache +for Quarto or R Markdown. Pick mdsmith for the cross-file +and self-maintaining-section layer. See the [benchmark doc][bench] for the full method, both corpora, every tool's command, the result tables, and @@ -1020,7 +1021,7 @@ if you need a stable rule set across upgrades. [mds025]: ../../internal/rules/MDS025-table-format/README.md [mds027]: ../../internal/rules/MDS027-cross-file-reference-integrity/README.md [mds067]: ../../internal/rules/MDS067-callout-type/README.md -[mds068]: ../../internal/rules/MDS068-link-style/README.md +[mds070]: ../../internal/rules/MDS070-same-file-anchor/README.md [mds028]: ../../internal/rules/MDS028-token-budget/README.md [mds029]: ../../internal/rules/MDS029-conciseness-scoring/README.md [mds035]: ../../internal/rules/MDS035-toc-directive/README.md diff --git a/docs/research/benchmarks/README.md b/docs/research/benchmarks/README.md index a2ce6020f..2392ec20e 100644 --- a/docs/research/benchmarks/README.md +++ b/docs/research/benchmarks/README.md @@ -163,9 +163,9 @@ validates generated sections, all here under this repository's own `.mdsmith.yml`, which enables opt-in rules a stock install leaves off. -Even with that extra work, the `mdsmith` row runs in the same -class as the per-file Rust linters. Compare it with the `rumdl` -row on both corpora: it lands at roughly 3x the check-only mado. +Even with that extra work, the `mdsmith` row comes in ahead of +the `rumdl` and `panache` rows on both corpora, at roughly 3x +the check-only mado. ### Like-for-like: the parity conventions @@ -191,8 +191,8 @@ the rule class measured here.) ### Why parity trails gomarklint -gomarklint is the fastest tool in the table because it never -builds an AST. It is a pure line scanner. The +gomarklint is the fastest tool in the benchmark because it +never builds an AST. It is a pure line scanner. The `gomarklint-parity` convention turns mdsmith down to gomarklint's 20-rule default set, and that set excludes MDS027: the coverage matrix marks gomarklint's single-file @@ -234,9 +234,10 @@ intended. ### Which tool to pick -Pick mado or rumdl for raw markdownlint-rule throughput. Pick +Pick mado for the fastest markdownlint-rule throughput. Pick mdsmith when the cross-file graph, readability budgets, and -self-maintaining sections are the point. +self-maintaining sections are the point; on full defaults it +already comes in ahead of rumdl and panache on both corpora. ## Measurement notes and fairness @@ -337,13 +338,13 @@ it can still predate gomarklint and omit the row. The per-merge already carries the row. Read that row as the lightest-workload entry. gomarklint's -defaults cover 21 of mdsmith's rules (22 in all, one off by +defaults cover 22 of mdsmith's rules (23 in all, one off by default, every one a full cover). That is the smallest default rule set of the markdownlint-family tools here: - gomarklint: 22 - mado: 28 -- rumdl and markdownlint: 42 each +- rumdl and markdownlint: 43 each (Counts from the [peer-linter coverage matrix][mdcov], generated from rule front matter.) So gomarklint's fastest time partly From de98c864dff7de7fef878524fd0a32e40ea857fe Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 22 Jun 2026 22:44:39 +0000 Subject: [PATCH 3/7] docs(benchmarks): scope the headline gap claim to the fastest tool The generated headline said mdsmith is "closing the rest of that gap" to the Rust linters, implying it trails them. Full mdsmith (214/154 ms) now beats rumdl and panache and trails only mado, so the gap is to the fastest of them, not to the group. Fix the hardcoded narrative in gen_fragments.py and regenerate headline.fragment.md plus the performance.md include. The results table is byte-identical (same data), so bench-fragments stays green. `mdsmith check .` passes. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4 --- docs/features/performance.md | 4 ++-- docs/research/benchmarks/gen_fragments.py | 4 ++-- docs/research/benchmarks/headline.fragment.md | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/features/performance.md b/docs/features/performance.md index 9bd276d64..65341b9e4 100644 --- a/docs/features/performance.md +++ b/docs/features/performance.md @@ -39,8 +39,8 @@ docs/research/benchmarks/data/*.json — do not edit by hand. Re-run the harness (run.sh) and `mdsmith fix` to refresh. --> One static Go binary checks 766 Markdown files in about 0.2 s. That is roughly 22x faster than Node markdownlint. -It does more per file than the Rust linters; closing the rest of -that gap is active work. +It does more per file than the Rust linters; closing +the gap to the fastest of them is active work. These numbers are re-measured each release, not hand-typed. A CI diff --git a/docs/research/benchmarks/gen_fragments.py b/docs/research/benchmarks/gen_fragments.py index 8828dfbba..fe6e51287 100755 --- a/docs/research/benchmarks/gen_fragments.py +++ b/docs/research/benchmarks/gen_fragments.py @@ -102,8 +102,8 @@ def main() -> None: f"One static Go binary checks {repo_n} Markdown files in " f"about\n{ms / 1000:.1f} s. That is roughly {mult}x " "faster than Node markdownlint.\nIt does more per file " - "than the Rust linters; closing the rest of\nthat gap " - "is active work.\n") + "than the Rust linters; closing\nthe gap to the fastest " + "of them is active work.\n") (frag_dir / "headline.fragment.md").write_text(head) print(f"fragments regenerated in {frag_dir}") diff --git a/docs/research/benchmarks/headline.fragment.md b/docs/research/benchmarks/headline.fragment.md index 1700f4612..900f9487d 100644 --- a/docs/research/benchmarks/headline.fragment.md +++ b/docs/research/benchmarks/headline.fragment.md @@ -3,5 +3,5 @@ docs/research/benchmarks/data/*.json — do not edit by hand. Re-run the harness (run.sh) and `mdsmith fix` to refresh. --> One static Go binary checks 766 Markdown files in about 0.2 s. That is roughly 22x faster than Node markdownlint. -It does more per file than the Rust linters; closing the rest of -that gap is active work. +It does more per file than the Rust linters; closing +the gap to the fastest of them is active work. From e4bd643fdaedc1d8c4cff2ee696c1ade24a0ec41 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 22 Jun 2026 23:08:48 +0000 Subject: [PATCH 4/7] docs(benchmarks): fix three claims contradicted by source Statement-by-statement validation against the data, conventions, and code surfaced three errors: - "gomarklint's 20-rule default set" -> 21. gomarklint ships 23 rules with 2 off by default (external-link, max-line-length), and gomarklint-parity resolves to 43-25+3 = 21 mdsmith rules. - Residual asymmetries claimed MDS031 unclosed-code-block is "not in the parity disable list" and "kept on". It is disabled in mado-parity, rumdl-parity, and markdownlint-parity (the embedded fragment lists it), and mdsmith-parity now selects mado-parity. Reworded: the slip-through risk is the two opt-in rules MDS034 and MDS067; MDS031 is no longer an exception. - check-bench budgets/baselines were stale. bench_test.go sets Small = 250 ms budget / ~14 ms baseline and Large = 2 s budget / ~90 ms baseline, not "2 s / 12 s" and "~0.09 s / ~0.8 s". `mdsmith check .` passes. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4 --- docs/research/benchmarks/README.md | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/docs/research/benchmarks/README.md b/docs/research/benchmarks/README.md index 2392ec20e..123c5af9c 100644 --- a/docs/research/benchmarks/README.md +++ b/docs/research/benchmarks/README.md @@ -194,7 +194,7 @@ the rule class measured here.) gomarklint is the fastest tool in the benchmark because it never builds an AST. It is a pure line scanner. The `gomarklint-parity` convention turns mdsmith down to -gomarklint's 20-rule default set, and that set excludes MDS027: +gomarklint's 21-rule default set, and that set excludes MDS027: the coverage matrix marks gomarklint's single-file `link-fragments` check a partial cover of mdsmith's cross-file MDS027, so the set is fully parse-skip-safe. @@ -575,16 +575,17 @@ level playing field: enabled in parity. So rumdl and markdownlint, which both ship MD054, do marginally more than `mdsmith-parity` on that rule. -- Three mdsmith-only rules (no markdownlint analog) are - **not** in the parity disable list: - [MDS031 unclosed-code-block](../../../internal/rules/MDS031-unclosed-code-block/README.md) - (default, kept on as a cheap structural check), - [MDS034 markdown-flavor](../../../internal/rules/MDS034-markdown-flavor/README.md), +- Two opt-in mdsmith-only rules (no markdownlint analog) sit + outside the parity disable lists: + [MDS034 markdown-flavor](../../../internal/rules/MDS034-markdown-flavor/README.md) and - [MDS067 callout-type](../../../internal/rules/MDS067-callout-type/README.md) - (opt-in, rely on default-off). A user config that - enables MDS034 or MDS067 would slip through - `mdsmith-parity`. + [MDS067 callout-type](../../../internal/rules/MDS067-callout-type/README.md). + Both rely on being off by default, so a user config that + enables either would slip through `mdsmith-parity`. The + default-on + [MDS031 unclosed-code-block](../../../internal/rules/MDS031-unclosed-code-block/README.md), + also mdsmith-only, is no longer an exception: the per-linter + parity sets (mado, rumdl, markdownlint) all disable it. ### Fairness note on panache @@ -612,11 +613,11 @@ Four gates run in CI. - **Tiered check budgets.** `BenchmarkCheckCorpus{Small,Large}` in `internal/engine/bench_test.go` lint a 60-file and a 600-file synthetic workspace with the full rule set. Small - (2 s budget, baseline ~0.09 s) catches per-file overhead; - Large (12 s budget, baseline ~0.8 s) catches superlinear + (250 ms budget, baseline ~14 ms) catches per-file overhead; + Large (2 s budget, baseline ~90 ms) catches superlinear scaling. The CI job `check-bench` is modelled on `lsp-bench`. - Baselines reflect the two plan-175 fixes (LineOfOffset - line-index; MDS024 tokenizer skip). + Baselines reflect the plan-175 fixes (LineOfOffset line-index; + MDS024 tokenizer skip) and the later engine passes. - **Doc-number drift.** The CI job `bench-fragments` regenerates the fragments from the committed `data/*.json` via `gen_fragments.py`, re-runs `mdsmith fix`, and From 32715495a7235d32861069146432da72426f8c23 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 23 Jun 2026 20:39:18 +0000 Subject: [PATCH 5/7] docs(benchmarks): make the fastest-tool story consistent Adversarial cross-statement validation found the doc told two different "fastest tool" stories and a backwards parity claim: - "gomarklint is the fastest tool in the benchmark" (unscoped) vs "Pick mado for the fastest..." vs the committed table where mado is fastest and gomarklint has no row. Scope every gomarklint "fastest" to the per-merge copy / lightest workload; frame mado as fastest at a full markdownlint rule set and gomarklint as faster-but-checks-least. Headline and fairness note already agree. - "gomarklint-parity runs no anchor check at all" is backwards: convention.go keeps same-file-anchor (MDS070) enabled in gomarklint-parity (plan 2606210840); only mado-parity disables it. Rewritten to say the parse-skip-safe anchor check stays in. - "more than an order of magnitude behind every native tool" is false for panache (cli2 is ~6x, not >10x, behind it); "all finish in tens of milliseconds" is false (mdsmith/rumdl/panache are hundreds). Scoped both. - Dropped the "21-rule" count that clashed with the "22" coverage figure; the set is now described as "enables 3, disables 25". - markdown-linters.md: scope "mado leads the per-file race" to the Rust linters; note parity trails mado on the neutral corpus. `mdsmith check .` passes. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4 --- docs/background/markdown-linters.md | 16 +++++----- docs/research/benchmarks/README.md | 45 ++++++++++++++++------------- 2 files changed, 34 insertions(+), 27 deletions(-) diff --git a/docs/background/markdown-linters.md b/docs/background/markdown-linters.md index f35a2c8f5..ec83ec2b5 100644 --- a/docs/background/markdown-linters.md +++ b/docs/background/markdown-linters.md @@ -760,13 +760,15 @@ access and is non-deterministic. Default mdsmith is more than an order of magnitude faster than markdownlint-cli2 on its own Markdown. With the mdsmith-only rules disabled (`mdsmith-parity`), it runs at -mado's speed: the two tie on the repo corpus. See the -[benchmark doc][bench] for the current ratios. - -mado leads the per-file race. Default mdsmith does more -per file: it also walks the cross-file graph, scores -readability, and validates generated sections. Even so, it -comes in ahead of rumdl and panache on both corpora. +mado's speed: the two tie on the repo corpus, and parity +trails mado only narrowly on the longer neutral corpus. See +the [benchmark doc][bench] for the current ratios. + +Among the Rust linters, mado leads the per-file race. +Default mdsmith does more per file: it walks the +cross-file graph, scores readability, and validates +generated sections. Even so, it comes in ahead of rumdl +and panache on both corpora. Pick mado for raw markdownlint throughput. Pick panache for Quarto or R Markdown. Pick mdsmith for the cross-file diff --git a/docs/research/benchmarks/README.md b/docs/research/benchmarks/README.md index 123c5af9c..edc1b7c83 100644 --- a/docs/research/benchmarks/README.md +++ b/docs/research/benchmarks/README.md @@ -147,11 +147,11 @@ results stand out. ### Every native binary beats the Node baseline -The table puts markdownlint-cli2 more than an order of magnitude -behind every native tool on both corpora. mado, rumdl, panache, -and mdsmith all finish in tens of milliseconds; the Node tool -takes seconds. If the alternative is a Node markdownlint, any -native tool here is a large speed win. +Every native tool finishes in tens to hundreds of milliseconds; +markdownlint-cli2 takes seconds. It runs more than an order of +magnitude behind the faster native tools and about 6x behind +even the slowest one, panache. If the alternative is a Node +markdownlint, any native tool here is a large speed win. ### Default mdsmith does the most work per run @@ -191,19 +191,23 @@ the rule class measured here.) ### Why parity trails gomarklint -gomarklint is the fastest tool in the benchmark because it -never builds an AST. It is a pure line scanner. The -`gomarklint-parity` convention turns mdsmith down to -gomarklint's 21-rule default set, and that set excludes MDS027: -the coverage matrix marks gomarklint's single-file -`link-fragments` check a partial cover of mdsmith's cross-file -MDS027, so the set is fully parse-skip-safe. - -So mdsmith can follow gomarklint onto the no-parse path here. -What remains is per-rule and fixed overhead, not the goldmark -parse. The trade-off: `gomarklint-parity` then runs no anchor -check at all, where gomarklint runs a same-file one (see plan -2606210840). [gomarklint architecture and the parity +When the per-merge copy includes it, gomarklint is the fastest +tool measured, because it never builds an AST: it is a pure line +scanner. (The committed table omits its row; see the +[fairness note](#fairness-note-on-gomarklint).) The +`gomarklint-parity` convention turns mdsmith down to that same +light footprint: it enables 3 opt-in rules and disables 25 +defaults. The result excludes the cross-file MDS027, because the +coverage matrix marks gomarklint's single-file `link-fragments` +a partial cover of it, so the set is fully parse-skip-safe. + +On that footing mdsmith follows gomarklint onto the no-parse +path; what remains is per-rule and fixed overhead, not the +goldmark parse. The anchor check stays in: `gomarklint-parity` +keeps the parse-skip-safe MDS070 same-file-anchor rule (plan +2606210840), which matches gomarklint's own same-file +`link-fragments`, and both leave the cross-file walk off. +[gomarklint architecture and the parity gap](gomarklint-architecture.md) breaks the profile down bucket by bucket and records the optimization levers and their ceilings. @@ -234,8 +238,9 @@ intended. ### Which tool to pick -Pick mado for the fastest markdownlint-rule throughput. Pick -mdsmith when the cross-file graph, readability budgets, and +Pick mado for the fastest run of a full markdownlint rule set; +gomarklint is faster still but checks the least. Pick mdsmith +when the cross-file graph, readability budgets, and self-maintaining sections are the point; on full defaults it already comes in ahead of rumdl and panache on both corpora. From 983bedf4d49640ea5ace51088af28b618d16293a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 23 Jun 2026 20:56:28 +0000 Subject: [PATCH 6/7] docs(benchmarks): choose-tool advice no longer leans on noise-level gaps On the parity workload mado and mdsmith-parity tie (67 ms = 67 ms on repo; 47 vs 53, within noise, on neutral), so recommending one over the other "for throughput" implies a difference that is not there. Reframe "Which tool to pick": the fast tools cluster within run-to-run noise (mado, mdsmith-parity, with gomarklint quicker because it runs the fewest rules); only rumdl and panache clearly trail. Choose on capability, not on these margins. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4 --- docs/research/benchmarks/README.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/research/benchmarks/README.md b/docs/research/benchmarks/README.md index edc1b7c83..cb8f66d6e 100644 --- a/docs/research/benchmarks/README.md +++ b/docs/research/benchmarks/README.md @@ -238,11 +238,17 @@ intended. ### Which tool to pick -Pick mado for the fastest run of a full markdownlint rule set; -gomarklint is faster still but checks the least. Pick mdsmith -when the cross-file graph, readability budgets, and -self-maintaining sections are the point; on full defaults it -already comes in ahead of rumdl and panache on both corpora. +mado and mdsmith on a parity profile run at the same speed: they +tie on the repo corpus and sit within noise of each other on the +neutral one, so there is no throughput reason to pick between +them. gomarklint is quicker still, because it runs the fewest +rules and builds no AST. rumdl and panache are the ones that +clearly trail, on both corpora. + +So choose on capability, not on these margins. Pick mdsmith when +the cross-file graph, readability budgets, and self-maintaining +sections are the point; on full defaults it still comes in ahead +of rumdl and panache while doing that extra work. ## Measurement notes and fairness From bc307f5666ef2065407afac2d51b3acf393d8e06 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 23 Jun 2026 21:09:06 +0000 Subject: [PATCH 7/7] docs(benchmarks): parity-row note matches the committed snapshot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The committed data predates the per-linter parity split, so it carries a single `mdsmith-parity` row — but the generated results note described per-linter `mdsmith--parity` rows that aren't in the table, which read as a contradiction. Make gen_fragments.py emit the note that matches the rows the data actually holds: the single pre-split column now, and the per-peer wording automatically once a refresh adds the four columns. Align the "Like-for-like" parenthetical to the same framing (single column reads closest to mado-parity; the next refresh replaces it with one column per peer). Regenerated; results table byte-identical; `mdsmith check .` passes. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4 --- docs/research/benchmarks/README.md | 14 ++++++------ docs/research/benchmarks/gen_fragments.py | 23 +++++++++++++++----- docs/research/benchmarks/results.fragment.md | 8 +++---- 3 files changed, 29 insertions(+), 16 deletions(-) diff --git a/docs/research/benchmarks/README.md b/docs/research/benchmarks/README.md index cb8f66d6e..dcde79ff4 100644 --- a/docs/research/benchmarks/README.md +++ b/docs/research/benchmarks/README.md @@ -109,10 +109,10 @@ file: results.fragment.md docs/research/benchmarks/data/*.json — do not edit by hand. Re-run the harness (run.sh) and `mdsmith fix` to refresh. --> -`mdsmith` is the default rule set. Each `mdsmith--parity` -row runs the rule set that peer enables by default, for a -like-for-like comparison against that peer (the -`bench--parity.mdsmith.yml` profiles). +`mdsmith` is the default rule set. The `mdsmith-parity` +row is the single pre-split parity column; the per-linter +`mdsmith--parity` columns, one per peer, land when +the benchmark is next re-run. **Repo corpus — 766 Markdown files** (median wall time, lower is better; `vs mado` is the ratio to mado's median): @@ -185,9 +185,9 @@ The `mdsmith` → `mdsmith-parity` delta is the measured cost of the cross-file and generated-content layer, work users opt into. The residual gap to mado on long prose is engine headroom: the number to drive down, and the profiler loop below is how. (These -figures predate the per-linter split; the `mdsmith-parity` -column now selects `mado-parity`, whose 27-rule set is close to -the rule class measured here.) +figures predate the per-linter split: the single `mdsmith-parity` +column shown here reads closest to the `mado-parity` 27-rule set, +and the next refresh replaces it with one column per peer.) ### Why parity trails gomarklint diff --git a/docs/research/benchmarks/gen_fragments.py b/docs/research/benchmarks/gen_fragments.py index fe6e51287..8907f4bcc 100755 --- a/docs/research/benchmarks/gen_fragments.py +++ b/docs/research/benchmarks/gen_fragments.py @@ -80,11 +80,24 @@ def main() -> None: repo_tbl, repo = rows(json_dir, "corpus_repo") neut_tbl, _ = rows(json_dir, "corpus_neutral") - note = ("`mdsmith` is the default rule set. Each " - "`mdsmith--parity`\nrow runs the rule set that " - "peer enables by default, for a\nlike-for-like comparison " - "against that peer (the\n`bench--parity.mdsmith.yml` " - "profiles).\n\n") + # The parity columns differ across the per-linter split: the pre-split + # snapshot carries a single `mdsmith-parity` row, while the current + # harness emits one `mdsmith--parity` column per peer. Describe + # whichever the data actually holds so the note never contradicts the + # table below it. + per_linter_parity = [c for c in repo if c.startswith("mdsmith-") + and c.endswith("-parity") and c != "mdsmith-parity"] + if per_linter_parity: + note = ("`mdsmith` is the default rule set. Each " + "`mdsmith--parity`\nrow runs the rule set that " + "peer enables by default, for a\nlike-for-like comparison " + "against that peer (the\n`bench--parity.mdsmith.yml` " + "profiles).\n\n") + else: + note = ("`mdsmith` is the default rule set. The `mdsmith-parity`\n" + "row is the single pre-split parity column; the per-linter\n" + "`mdsmith--parity` columns, one per peer, land when\n" + "the benchmark is next re-run.\n\n") results = (GEN + "\n" + note + f"**Repo corpus — {repo_n} Markdown files** (median " "wall time, lower is\nbetter; `vs mado` is the ratio " diff --git a/docs/research/benchmarks/results.fragment.md b/docs/research/benchmarks/results.fragment.md index 2404ff34e..7aae080ab 100644 --- a/docs/research/benchmarks/results.fragment.md +++ b/docs/research/benchmarks/results.fragment.md @@ -2,10 +2,10 @@ docs/research/benchmarks/data/*.json — do not edit by hand. Re-run the harness (run.sh) and `mdsmith fix` to refresh. --> -`mdsmith` is the default rule set. Each `mdsmith--parity` -row runs the rule set that peer enables by default, for a -like-for-like comparison against that peer (the -`bench--parity.mdsmith.yml` profiles). +`mdsmith` is the default rule set. The `mdsmith-parity` +row is the single pre-split parity column; the per-linter +`mdsmith--parity` columns, one per peer, land when +the benchmark is next re-run. **Repo corpus — 766 Markdown files** (median wall time, lower is better; `vs mado` is the ratio to mado's median):