Skip to content

Run Scenario CodeLens: full-document resolveTestTargets walk is too slow on large feature files #495

Description

@clrudolphi

Issue Description

RunTestCodeLensService.GetTargetsAsync resolves every scenario in a .feature file on every call, not just the one line a particular CodeLens data point actually needs. On the VeryLargeFeature stress corpus (~2,000-2,400 scenarios in one file), a single walk takes 30-45 seconds — independently too slow relative to VS's own classic-CodeLens per-data-point timeout (observed at ~26 seconds).

This was masked, not fixed, by the caching work in #492/#493/#494:

No caching strategy that keeps re-running the full walk from scratch will fully paper over this on a corpus this large. As shipped today, this only manifests as a "stuck CodeLens" symptom the first time a fresh walk is required (cold open, or after a real invalidation from reqnroll/refreshCodeLens) — but on a sufficiently large file, that first walk can still exceed VS's timeout even with everything from #492-#494 in place.

Proposed direction (not yet designed)

Some options worth evaluating, not mutually exclusive:

  • Incremental/per-scenario caching inside RunTestCodeLensService/ScenarioTestTargetResolver, keyed by scenario identity + a cheap staleness check (e.g. tag/method-signature hash), so a real edit only needs to re-resolve the scenarios it actually touched instead of the whole file.
  • Resolving lazily/on-demand per visible line instead of eagerly walking the whole document up front, if the classic CodeLens API allows a data point to ask for just its own line's answer without the service needing every other scenario's answer too.
  • Revisiting whether RunTestCodeLensService.GetTargetsAsync's "resolve everything" contract is the right shape at all now that callers are individual line-scoped RunTestCodeLensDataPoints, not just the whole-file tagger.

Scope: add this traffic to the performance benchmark utility

tests/Performance/Reqnroll.IdeSupport.LSP.Server.Benchmarks* has no coverage at all today for reqnroll/resolveTestTargets / the Run Scenario CodeLens path — it's the one reqnroll/* operation with neither a PerfTargets entry (even an unpublished "measured but no threshold" one, which every sibling operation has) nor an InteractiveScenarios method, despite already being field-instrumented via IOperationDurationRecorder in ResolveTestTargetsHandler. Any fix for this issue should come with:

  1. A PerfTargets.ResolveTestTargets entry + InteractiveScenarios.ResolveTestTargetsAsync() for single-request latency, following the existing pattern used by every other reqnroll/* operation.
  2. A contention scenario modeling N concurrent CodeLens-driven callers for the same large file resolving together — WorkspaceReloadContentionScenario (Performance benchmark suite never exercises concurrent request contention, so it missed the #471/#477 dispatch-stall regression #488) already establishes the right shape for this ("many requests fire together, race a cheap read") and should be followed rather than reinvented.

Without this, a regression in either the per-scenario cost or the whole-document walk shape would ship silently, the same way this issue's underlying slowness did.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    designNeeds a design decisionenhancementNew feature or requestlsp-serverLSP servertestingTest coverage / verification work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions