Impact
Read-only production investigation covering 2026-09-04 09:08:28 UTC through 2026-09-07 09:08:28 UTC found 27 exception telemetry rows across 20 operations for VssServiceException at VssHttpClientBase.HandleResponseAsync. The signature combines different causes: 26 rows across 19 operations were DBCPU throttling; one unrelated build-state conflict is excluded below.
The throttling occurred on September 4 between 23:00:22 and 23:42:22 UTC. Matching both ambient and recorded work-item correlation fields identified 19 failed-attempt events across 19 operations. Six were attempt-3 exhaustion events: two BuildAnalysisRequestWorkItem events and four KnownIssueValidationRequest events. These are telemetry/event counts, not a verified count of distinct builds or logical work items; event dimensions do not include build identifiers. Sampling and duplicate exception reporting may affect row counts.
Representative Evidence
- Azure DevOps reports that a request was blocked for exceeding usage of resource DBCPU in namespace VSID.
- First actionable source frame:
BuildInsights.BuildAnalysis.Services.BuildDataProvider.GetTestHistoryAsync for 24 throttling rows; GetTestRuns for two rows.
- Local inspected source:
src/BuildInsights.BuildAnalysis/Services/BuildDataProvider.cs, GetTestHistoryAsync around line 426 and QueryTestHistoryAsync around line 451.
- Caller:
src/BuildInsights.BuildAnalysis/Services/BuildAnalysisService.cs, GetTestResults around line 540.
- No raw telemetry, operation identifiers, URLs with query strings, or entity payloads are included.
Likely Cause And Confidence
High confidence: upstream Azure DevOps database-CPU throttling caused the correlated failures.
Medium confidence: expensive history retrieval and whole-work-item retries amplify the pressure. GetTestHistoryAsync requests four seven-day windows, each with continuation paging. GetTestResults processes up to five tests, potentially making 20 initial history requests plus continuation requests per analysis.
A RetryAfterHandler from the Microsoft.DotNet.Services.Utility NuGet package is already registered in src/BuildInsights.Utilities.AzureDevOps/VssConnectionExtension.cs. Its implementation, effective cooldown, and response headers were not verified. Dependency failures observed during the interval did not match the selected exception operation identifiers; no HTTP status is attributed to these exceptions. Adequate existing cooldown or evidence that other consumers dominate the shared quota would weaken the local amplification hypothesis.
Suggested Remediation
First verify the existing retry handler and capture structured throttle/retry metadata. At the dominant test-history request boundary, apply bounded, cancellation-aware backoff for confirmed throttling, honoring server retry guidance and retrying the current page without replaying completed pages. Avoid layering another retry loop over an equivalent existing policy. Do not catch all VssServiceException instances or replace unavailable history with an empty successful result.
This targets the dominant history path, not the separate test-run query failures. If shared quota pressure remains sustained, evaluate per-organization request limiting; that is a broader change. The Build Insights call sites are first-party in the investigated workspace; ownership of any shared-package change must be confirmed before changing shared infrastructure.
Impact
Read-only production investigation covering 2026-09-04 09:08:28 UTC through 2026-09-07 09:08:28 UTC found 27 exception telemetry rows across 20 operations for VssServiceException at VssHttpClientBase.HandleResponseAsync. The signature combines different causes: 26 rows across 19 operations were DBCPU throttling; one unrelated build-state conflict is excluded below.
The throttling occurred on September 4 between 23:00:22 and 23:42:22 UTC. Matching both ambient and recorded work-item correlation fields identified 19 failed-attempt events across 19 operations. Six were attempt-3 exhaustion events: two BuildAnalysisRequestWorkItem events and four KnownIssueValidationRequest events. These are telemetry/event counts, not a verified count of distinct builds or logical work items; event dimensions do not include build identifiers. Sampling and duplicate exception reporting may affect row counts.
Representative Evidence
BuildInsights.BuildAnalysis.Services.BuildDataProvider.GetTestHistoryAsyncfor 24 throttling rows; GetTestRuns for two rows.src/BuildInsights.BuildAnalysis/Services/BuildDataProvider.cs, GetTestHistoryAsync around line 426 and QueryTestHistoryAsync around line 451.src/BuildInsights.BuildAnalysis/Services/BuildAnalysisService.cs, GetTestResults around line 540.Likely Cause And Confidence
High confidence: upstream Azure DevOps database-CPU throttling caused the correlated failures.
Medium confidence: expensive history retrieval and whole-work-item retries amplify the pressure. GetTestHistoryAsync requests four seven-day windows, each with continuation paging. GetTestResults processes up to five tests, potentially making 20 initial history requests plus continuation requests per analysis.
A RetryAfterHandler from the Microsoft.DotNet.Services.Utility NuGet package is already registered in src/BuildInsights.Utilities.AzureDevOps/VssConnectionExtension.cs. Its implementation, effective cooldown, and response headers were not verified. Dependency failures observed during the interval did not match the selected exception operation identifiers; no HTTP status is attributed to these exceptions. Adequate existing cooldown or evidence that other consumers dominate the shared quota would weaken the local amplification hypothesis.
Suggested Remediation
First verify the existing retry handler and capture structured throttle/retry metadata. At the dominant test-history request boundary, apply bounded, cancellation-aware backoff for confirmed throttling, honoring server retry guidance and retrying the current page without replaying completed pages. Avoid layering another retry loop over an equivalent existing policy. Do not catch all VssServiceException instances or replace unavailable history with an empty successful result.
This targets the dominant history path, not the separate test-run query failures. If shared quota pressure remains sustained, evaluate per-organization request limiting; that is a broader change. The Build Insights call sites are first-party in the investigated workspace; ownership of any shared-package change must be confirmed before changing shared infrastructure.