bench(lint): add inline scanner benchmark and corpus hit-rate test - #682
Merged
Conversation
Adds corpus-backed benchmarks and a hit-rate test for the Layer 1 inline byte scanner (plan 2606202100 task 4). Uses the repo's own Markdown as corpus (no MDSMITH_SPIKE_CORPUS required). On the repo corpus (1917 inline-bearing runs from parse-skip-eligible files, 421 scanner-eligible / 22.0%): BenchmarkScanInlineRun_Eligible: 1799 ns/op 808 B/op 1 alloc/op BenchmarkParseInline_Eligible: 2123 ns/op 1845 B/op 15 allocs/op BenchmarkInlineRunNode_AllRuns: 3728 ns/op 2731 B/op 14 allocs/op BenchmarkParseInline_AllRuns: 3889 ns/op 2935 B/op 17 allocs/op The scanner is 1.18× faster with 15× fewer allocations on eligible runs, but 22% coverage means InlineBlocks is only ~4% cheaper overall — confirming the parse-skip is still not a net win until the scanner handles multi-line paragraphs (the follow-up). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MbLdsh1WnHLfHBFJ1hmQQj
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The InlineBlocks projection benchmark over the repo's parse-skip-eligible corpus shows the byte scanner is ~4.1x faster on scanner-eligible runs (446 vs 1842 ns/op, 1 vs 15 allocs) and ~24% faster across the full run set (3143 vs 4134 ns/op) versus the goldmark per-run parse. 26.5% of runs are scanner-eligible; 22.0% scan to completion. Mark plan tasks 4 (re-profile) and 5 (default-on decision) done. The global MDSMITH_LAYER0_SKIP default stays off pending the end-to-end neutral-corpus re-profile, which the in-package projection benchmark does not by itself measure; document that scoping in the plan. Regenerate PLAN.md catalog for the status/summary change. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MbLdsh1WnHLfHBFJ1hmQQj
…alysis Tasks 4 and 5 are done. The inline scanner removes the per-run goldmark parse for scanner-eligible runs (22% of corpus runs) and is measurably faster: - Scanner-eligible runs: 462 vs 2196 ns/op (~4.7x faster, 15× fewer allocs) - All runs (scanner-first + fallback): 5067 vs 5670 ns/op (~11% faster) The MDSMITH_LAYER0_SKIP gate stays default-off pending the end-to-end neutral-corpus re-profile on MDSMITH_SPIKE_CORPUS; that measurement is the precondition for the default-on flip, not the in-package benchmark. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MbLdsh1WnHLfHBFJ1hmQQj
Extract the run-grouping loop from scanInlineBlocks into inlineRunBounds so fileRuns in the bench test delegates to the single authoritative source rather than duplicating the loop verbatim. Delete the duplicate repoRootTB helper; widen repoRoot to testing.TB so both tests and benchmarks share it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MbLdsh1WnHLfHBFJ1hmQQj
…nInlineBlocks Pre-size the out slice in inlineRunBounds with make([][2]int, 0, len(f.Lines)) per the project's allocation-budget coding standard (CLAUDE.md: "Pre-size slices with make([]X, 0, n)"). Add an early nil return in scanInlineBlocks when inlineRunBounds finds no runs, preserving the pre-refactor nil-for-no-content contract and avoiding a needless arena.New() for all-code-block files. Update the corpusRuns comment and plan doc to accurately describe the corpus filter's intentional omission of the production gate's block-quote guard. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MbLdsh1WnHLfHBFJ1hmQQj
…corpusRuns Remove `defer f.inlineBlocksDone.Store(true)` in InlineBlocks: deferred stores fire during panic unwind before any outer recover() can intervene, permanently caching a nil f.inlineBlocks with done=true and silently suppressing all inline diagnostics for the file on subsequent rule checks. Changed to an unconditional Store after the successful assignment so a panic leaves inlineBlocksDone false and callers can retry normally. Add directory pruning (.git, node_modules, dist, build) to corpusRuns, matching collectMarkdownCorpus in the integration tests — demo/build/page.md was entering the bench corpus and inflating run counts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MbLdsh1WnHLfHBFJ1hmQQj
jeduden
added
queue
Owner
Author
|
🟢 Merge Queue — picked up This PR is in the queue and will be batched with other Next: No action needed — you'll get another comment when CI starts on the batch. View merge queue run. |
Owner
Author
|
🔵 Merge Queue — CI running Merged into batch branch Next: No action needed — you'll be notified when CI completes. |
jeduden
removed
the
queue:active
Owner
Author
|
✅ Merge Queue — merged This PR landed on Next: Done — nothing more to do here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MDSMITH_SPIKE_CORPUSrequiredBenchmark results (repo corpus, 1917 inline runs from parse-skip-eligible files)
BenchmarkScanInlineRun_Eligible(scanner, 421 eligible runs)BenchmarkParseInline_Eligible(goldmark on same runs)BenchmarkInlineRunNode_AllRuns(scanner-first + fallback, all 1917 runs)BenchmarkParseInline_AllRuns(goldmark on all runs)Finding: The scanner handles 22% of corpus runs and is 1.18× faster with 15× fewer allocations on those runs. Overall
InlineBlockscost is only ~4% lower — the parse-skip is still not a net win until the scanner handles multi-line paragraphs.Test plan
go test -run TestCorpusRunEligibility ./internal/lint/passes and logs hit-rate ≥ 20%go test -bench=. ./internal/lint/runs cleanlygo test ./...passesNotes
Addresses plan 2606202100 task 4 (measurement checkpoint). The scanner coverage needs to be widened to multi-line paragraphs before the parse-skip can be turned on by default (task 5).
🤖 Generated with Claude Code
https://claude.ai/code/session_01MbLdsh1WnHLfHBFJ1hmQQj
Generated by Claude Code