Add dispatch-fairness benchmark scenario + bump default corpus scale - #489
Merged
Conversation
…488) The performance benchmark suite never caught the #471/#477 dispatch-stall regression because every scenario measures one operation in isolation (BenchmarkRunner/SessionScenario never race a slow op against a cheap one), and the default --files 10 undershoots the ~50-file/~1,350-step scale the #471 investigation needed to reproduce it. The fix that would have caught it (ConcurrencyProbeTests) landed as a one-off xUnit probe outside the benchmark suite instead, invisible to BenchmarkReport/JSON trend tracking. Adds WorkspaceReloadContentionScenario: models many editor tabs restoring (or a workspace-wide reload/config-change event) as a concurrent didChange storm across most open files, racing a cheap textDocument/foldingRange read on an untouched probe file. Wired into BenchmarkRunner so it's tracked in the report/JSON like every other scenario, gated via the existing --assert / reference-machine mechanism. Gates on a ratio-to-same-run-baseline ceiling (ContentionCheck), not an absolute-ms target: the #477 investigation found this class of measurement's absolute magnitude swings wildly with machine/CPU contention (~15x-20x locally, ~1.3x-40x+ under noise), so an absolute threshold would either never fire or fire constantly depending on the machine. Also bumps BenchmarkRunner/SessionRunner's default --files from 10 to 50 (the full committed corpus) so routine runs reach realistic scale instead of undershooting it. Issue: #488 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #488.
tests/Performance/...Benchmarks*) never caught the LSP server: workspace/inlayHint/refresh and workspace/semanticTokens/refresh scale badly on large solutions, blocking unrelated requests for tens of seconds #471/Discarded Task from _mediator.Publish(...) doesn't defer work — several fire-and-forget call sites actually run inline and block their caller #477 dispatch-stall regression: every existing scenario measures one operation in isolation (BenchmarkRunner/SessionScenarionever race a slow op against a cheap one), and the default--files 10undershot the ~50-file/~1,350-step scale the LSP server: workspace/inlayHint/refresh and workspace/semanticTokens/refresh scale badly on large solutions, blocking unrelated requests for tens of seconds #471 investigation needed to reproduce it. The fix that did catch it (ConcurrencyProbeTests) landed as a one-off xUnit probe outside the benchmark suite, invisible toBenchmarkReport/JSON trend tracking.WorkspaceReloadContentionScenario: models many editor tabs restoring (or a workspace-wide reload/config-change event) as a concurrentdidChangestorm across most open files, racing a cheaptextDocument/foldingRangeread on an untouched probe file. Wired intoBenchmarkRunnerso it's tracked in the report/JSON like every other scenario, gated via the existing--assert/ reference-machine mechanism.ContentionCheck), not an absolute-ms target — the Discarded Task from _mediator.Publish(...) doesn't defer work — several fire-and-forget call sites actually run inline and block their caller #477 investigation found this class of measurement's absolute magnitude swings wildly with machine/CPU contention (~15x-20x locally, ~1.3x-40x+ under noise), so an absolute threshold would either never fire or fire constantly depending on the machine.BenchmarkRunner/SessionRunner's default--filesfrom 10 to 50 (the full committed corpus) so routine runs reach realistic scale instead of undershooting it, and updates--helpto explain the new check.Test plan
dotnet buildon the full solution (Reqnroll.IdeSupport.slnx) succeedsdotnet testonReqnroll.IdeSupport.LSP.Server.Tests(Performancefilter) — 62/62 pass, including the newWorkspaceReloadContentionScenarioTestsdotnet run --project tests/Performance/Reqnroll.IdeSupport.LSP.Server.Benchmarks -- run --no-batchagainst currentmaster(pre-Discarded Task from _mediator.Publish(...) doesn't defer work — several fire-and-forget call sites actually run inline and block their caller #477): the new scenario measured a ~92.6x and, on a repeat run, ~28.3x stall ratio — reproducing the exact noisy-but-real regressionConcurrencyProbeTestsalready documented, confirming the new scenario would have caught LSP server: workspace/inlayHint/refresh and workspace/semanticTokens/refresh scale badly on large solutions, blocking unrelated requests for tens of seconds #471 and validating the ratio-ceiling (not absolute-ms) design--assertcorrectly fails/passes based on the ceiling--helpoutput describes the new check🤖 Generated with Claude Code