Skip to content

docs(benchmarks): restructure the benchmark write-up for scannability - #684

Merged
jeduden merged 7 commits into
mainfrom
claude/stoic-gauss-ieovtw
Jun 23, 2026
Merged

docs(benchmarks): restructure the benchmark write-up for scannability#684
jeduden merged 7 commits into
mainfrom
claude/stoic-gauss-ieovtw

Conversation

@jeduden

@jeduden jeduden commented Jun 22, 2026

Copy link
Copy Markdown
Owner

What

Restructure docs/research/benchmarks/README.md — the source of the
published assets/benchmarks/pages/benchmark.md page — so it reads as a
scannable report instead of walls of text.

The page held a lot of important detail but packed it into long
bold-label paragraphs and 30–50 word sentences, most of it crammed under a
single ## Reading the result heading.

Changes (prose only; no facts changed)

  • Intro → a lead plus a two-item list contrasting the committed
    snapshot vs the per-merge assets copy.
  • Reading the result → the bold-label paragraphs become real ###
    subsections; the two profiler-loop bugs and the per-corpus parity
    results are now lists.
  • New ## Measurement notes and fairness groups the methodology
    caveats (why the numbers move, the fairness notes, the rule-set parity
    detail) that were all nested under "Reading the result".
  • Gates promoted from ### to its own ##.
  • Sentences tightened to the background voice; promotional wording dropped
    (honest, crushes, home turf).

New top-level outline: Method → Results → Reading the result →
Measurement notes and fairness → Gates.

Safety

  • Only prose outside the <?include?> blocks changed. The generated
    fragment bodies, *.fragment.md, and data/*.json are untouched, so
    the bench-fragments drift gate stays green.
  • In-page anchor links updated/preserved; MDS070 (same-file-anchor) is on
    for this file and mdsmith check . passes (508 files, 0 failures).
  • mdsmith fix made zero changes after the edits.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4

Generated by Claude Code

@claude
docs(benchmarks): restructure the benchmark write-up for scannability
d58b59d 
The benchmark research README is the source of the published
assets/benchmarks/pages/benchmark.md page. It read as walls of
text: long bold-label paragraphs and 30-50 word sentences packed
under a single "Reading the result" H2.

Restructure for scannability without changing any measured fact:

- Split the intro wall into a lead plus a two-source list.
- Convert the "Reading the result" bold-label paragraphs into real
  subsections; list the two profiler-loop bugs and the per-corpus
  parity results.
- Group the methodology caveats under a new "Measurement notes and
  fairness" H2 and promote "Gates" to its own H2.
- Tighten sentences to the background voice and drop promotional
  wording (honest, crushes, home turf).

Only prose outside the <?include?> blocks changed; the generated
fragment bodies and data/*.json are untouched, so bench-fragments
stays green. `mdsmith check .` passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4
@codecov

codecov Bot commented Jun 22, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.62%. Comparing base (7d37ea5) to head (bc307f5).
⚠️ Report is 19 commits behind head on main.

Additional details and impacted files
Components Coverage Δ
Go 98.61% <ø> (ø)
TypeScript 99.54% <ø> (ø)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

claude added 6 commits June 22, 2026 22:41
@claude
docs(benchmarks): correct stale comparative claims and counts
a97dc63 
The interpretive prose predated the current measured numbers and
the MDS070 addition. Fix the claims against the committed results
table and the live rule set.

Benchmark README:
- Full mdsmith (214/154 ms) now beats rumdl (324/195) and panache
  on both corpora, so "same class as the per-file Rust linters /
  compare with rumdl" and "pick mado or rumdl for throughput"
  were wrong. rumdl is not a throughput leader; only mado leads.
- gomarklint is "the fastest tool in the benchmark", not "in the
  table" — the committed results table has no gomarklint row
  (only the per-merge copy does).
- Peer default-coverage counts refreshed to the current coverage
  matrix the prose cites: rumdl/markdownlint 42->43, gomarklint
  21/22->22/23 (MDS070 added a default cover for both, after the
  2026-06-12 measurement).

markdown-linters.md:
- Same rumdl/mado ranking fix in the Benchmarks section
  ("lands close to rumdl" -> ties mado; "mado and rumdl lead,
  mdsmith trails" -> mado leads, mdsmith beats rumdl and panache).
- mdsmith rule count 67->69 (MDS001-MDS070, MDS060 unused) in the
  intro prose and the two comparison tables.

mado-parity "27 rules" verified correct (43 default-enabled - 24
disabled + 8 opt-in). `mdsmith check .` passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4
@claude
docs(benchmarks): scope the headline gap claim to the fastest tool
de98c86 
The generated headline said mdsmith is "closing the rest of that
gap" to the Rust linters, implying it trails them. Full mdsmith
(214/154 ms) now beats rumdl and panache and trails only mado,
so the gap is to the fastest of them, not to the group.

Fix the hardcoded narrative in gen_fragments.py and regenerate
headline.fragment.md plus the performance.md include. The results
table is byte-identical (same data), so bench-fragments stays
green. `mdsmith check .` passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4
@claude
docs(benchmarks): fix three claims contradicted by source
e4bd643 
Statement-by-statement validation against the data, conventions,
and code surfaced three errors:

- "gomarklint's 20-rule default set" -> 21. gomarklint ships 23
  rules with 2 off by default (external-link, max-line-length),
  and gomarklint-parity resolves to 43-25+3 = 21 mdsmith rules.
- Residual asymmetries claimed MDS031 unclosed-code-block is
  "not in the parity disable list" and "kept on". It is disabled
  in mado-parity, rumdl-parity, and markdownlint-parity (the
  embedded fragment lists it), and mdsmith-parity now selects
  mado-parity. Reworded: the slip-through risk is the two opt-in
  rules MDS034 and MDS067; MDS031 is no longer an exception.
- check-bench budgets/baselines were stale. bench_test.go sets
  Small = 250 ms budget / ~14 ms baseline and Large = 2 s budget
  / ~90 ms baseline, not "2 s / 12 s" and "~0.09 s / ~0.8 s".

`mdsmith check .` passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4
@claude
docs(benchmarks): make the fastest-tool story consistent
3271549 
Adversarial cross-statement validation found the doc told two
different "fastest tool" stories and a backwards parity claim:

- "gomarklint is the fastest tool in the benchmark" (unscoped)
  vs "Pick mado for the fastest..." vs the committed table where
  mado is fastest and gomarklint has no row. Scope every
  gomarklint "fastest" to the per-merge copy / lightest workload;
  frame mado as fastest at a full markdownlint rule set and
  gomarklint as faster-but-checks-least. Headline and fairness
  note already agree.
- "gomarklint-parity runs no anchor check at all" is backwards:
  convention.go keeps same-file-anchor (MDS070) enabled in
  gomarklint-parity (plan 2606210840); only mado-parity disables
  it. Rewritten to say the parse-skip-safe anchor check stays in.
- "more than an order of magnitude behind every native tool" is
  false for panache (cli2 is ~6x, not >10x, behind it); "all
  finish in tens of milliseconds" is false (mdsmith/rumdl/panache
  are hundreds). Scoped both.
- Dropped the "21-rule" count that clashed with the "22" coverage
  figure; the set is now described as "enables 3, disables 25".
- markdown-linters.md: scope "mado leads the per-file race" to
  the Rust linters; note parity trails mado on the neutral corpus.

`mdsmith check .` passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4
@claude
docs(benchmarks): choose-tool advice no longer leans on noise-level gaps
983bedf 
On the parity workload mado and mdsmith-parity tie (67 ms = 67 ms
on repo; 47 vs 53, within noise, on neutral), so recommending one
over the other "for throughput" implies a difference that is not
there. Reframe "Which tool to pick": the fast tools cluster
within run-to-run noise (mado, mdsmith-parity, with gomarklint
quicker because it runs the fewest rules); only rumdl and panache
clearly trail. Choose on capability, not on these margins.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4
@claude
docs(benchmarks): parity-row note matches the committed snapshot
bc307f5 
The committed data predates the per-linter parity split, so it
carries a single `mdsmith-parity` row — but the generated results
note described per-linter `mdsmith-<linter>-parity` rows that
aren't in the table, which read as a contradiction.

Make gen_fragments.py emit the note that matches the rows the
data actually holds: the single pre-split column now, and the
per-peer wording automatically once a refresh adds the four
columns. Align the "Like-for-like" parenthetical to the same
framing (single column reads closest to mado-parity; the next
refresh replaces it with one column per peer).

Regenerated; results table byte-identical; `mdsmith check .`
passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PThu9DcNDDSAKaw1upanY4
@jeduden jeduden added queue Add to a PR to enqueue it queue:active Applied automatically when a PR is in an active batch and removed queue Add to a PR to enqueue it labels Jun 23, 2026
@jeduden

jeduden commented Jun 23, 2026

Copy link
Copy Markdown
Owner Author

🟢 Merge Queue — picked up

This PR is in the queue and will be batched with other queue-labelled PRs.

Next: No action needed — you'll get another comment when CI starts on the batch. View merge queue run.

@jeduden

jeduden commented Jun 23, 2026

Copy link
Copy Markdown
Owner Author

🔵 Merge Queue — CI running

Merged into batch branch merge-queue/batch-684-1782251014. View CI run.

Next: No action needed — you'll be notified when CI completes.

@jeduden jeduden removed the queue:active Applied automatically when a PR is in an active batch label Jun 23, 2026
@jeduden
jeduden merged commit 3d65431 into main Jun 23, 2026
36 checks passed
@jeduden

jeduden commented Jun 23, 2026

Copy link
Copy Markdown
Owner Author

✅ Merge Queue — merged

This PR landed on main via commit 3d65431. CI run that validated the merge.

Next: Done — nothing more to do here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jeduden @claude