cmd: add per-runner concurrency benchmark harness and baseline#2573
Draft
bendrucker wants to merge 1 commit into
Draft
cmd: add per-runner concurrency benchmark harness and baseline#2573bendrucker wants to merge 1 commit into
bendrucker wants to merge 1 commit into
Conversation
Extract the per-runner check fan-out into checkRunners and add a benchmark suite covering the fan-out in isolation, serial runner construction, and the end-to-end fan-out over a generated fixture. The benchmarks report deterministic custom metrics (peak-concurrency, check-calls/op, runners) since the path is I/O-bound. Adds a make bench target and a non-blocking benchstat CI workflow that renders a HEAD-vs-base table into the job summary. No behavior change. This establishes the master baseline that the rest of the concurrency re-architecture stack is measured against. Claude-Session: https://claude.ai/code/session_01SFtKL5iaFWPVDqdcyvPiCJ
This was referenced Jun 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First PR in a benchmark-driven pass at tflint's concurrency. The repo has no
benchmarks, and the changes that follow (bounding the per-runner check fan-out, cutting
redundant per-runner work) only make sense measured against a baseline, so this
establishes one. No behavior change.
It extracts the inline per-runner fan-out in
inspectModuleintocheckRunners, theseam the benchmarks measure and that a later PR will bound. Three benchmarks cover the
path:
BenchmarkCheckRunnersruns the fan-out in isolation with a synthetic check,BenchmarkInspectFanoutruns it end-to-end over a generated N-module fixture with eachcheck hitting the real evaluator through a
plugin.GRPCServerplus a simulated remotedelay, and
BenchmarkBuildRunnersisolates runner construction. Sizes sweep1/10/100/1000.
The path is I/O-bound, so wall-clock is noisy and the signals that matter are the custom
metrics:
peak-concurrency,check-calls/op, andrunners.make benchruns thesuite;
bench.ymlposts abenchstattable to the job summary, non-blocking andHEAD-only until the base branch also carries the benchmarks.
Two things the baseline already shows. The fan-out is unbounded: peak concurrency tracks
runner count, ~900 in flight at 1000 module calls on 10 cores. And runner construction
is serial and superlinear, 7s and ~14 GiB allocated at 1000 modules, which may matter
more than the fan-out itself.
One correction to the plan that motivated this: the shared-root evaluation race it set
out to fix (#2094) doesn't exist on master.
Evaluator.scope()allocates a fresh scopeper call, so the
CallStack/ResolvedLocalValuescaches are never shared acrossgoroutines, and
integrationtest/race/eval_locals_on_root_ctxpasses under-race. If#2094 still reproduces, it's a logic bug in
CallStackordering.