[Internal] CancellationToken: Fixes operation-level token being ignored in gateway HTTP retry delays and query pagination paths

[tvaron3]

Summary

Two independent improvements bundled in one PR:

1. Fault injection coverage for LeaseNotFound (410/1022) and gateway metadata-read failures — adds end-to-end tests that inject the lease-not-found substatus on point operations and gateway PKRange/Address metadata reads, exercising cross-region retry and address-cache invalidation paths that previously had no fault-injection coverage.
2. Operation-level CancellationToken honor in the gateway HTTP retry loop and query pagination call stack — the customer-observable fix. Inter-attempt Task.Delay backoffs and entry-point checks now respect the caller's token so a cancelled operation surfaces OperationCanceledException within the caller's deadline.

Why this matters — walkthrough of the customer scenario

ICM 21000001041367 (Costco Wallet Gateway Service): customer query with a 30 s CancellationToken returned TaskCanceledException after ~100 s wall-clock during a partition merge on paymentshub.wallets. Backend telemetry showed the legitimate LeaseNotFound (410/1022) window during lease handoff cutover was only 18 s. The remaining ~80 s of post-deadline time was inside the SDK.

Where the 80 s actually live

410/1022 returns ShouldRetryResult.RetryAfter(TimeSpan.Zero) — there is no deliberate backoff for this substatus. So why ~80 s?

The time isn't in RetryAfter. It's in HTTP transport timeouts between regions that the SDK sits on inside its own Task.Delays — and FeedIterator prefetch fans out parallel retry chains, each unwinding independently.

Per retry attempt	Time	Cancellable before fix?	Cancellable after fix?
DNS / TCP / TLS reuse	~1 ms	yes (socket await)	yes
Cross-region HTTP round-trip	50–80 ms	yes (socket await)	yes
HttpTimeoutPolicy retry wait inside CosmosHttpClientCore.ProcessMessageAsync	up to 5 s per attempt	❌ NO (Task.Delay(delay) had no token)	✅ YES (Task.Delay(delay, ct))
Consistency-barrier Task.Delay inside StoreReader	up to several seconds	❌ NO	✅ YES (covered by FaultInjectionMetadataBarrierRequestDelay_HonorsCancellationToken)
GetEffectiveRangesAsync / routing	ms, but uncancellable	❌ NO	✅ YES (entry-point ThrowIfCancellationRequested)

Multiply that uncancellable per-attempt budget by N parallel prefetch chains × M retry rounds before the policy gives up = the ~80 s Costco saw.

Before vs. after

sequenceDiagram
    autonumber
    participant App as Customer App (CT = 30s)
    participant SDK as SDK Retry Loop
    participant E as East US (source, merging)
    participant W as West US 2 (target)

    Note over App,W: t=0s  Query starts, CT armed for 30s
    App->>SDK: ReadFeedAsync(ct)
    SDK->>E: GET (attempt 1)
    E-->>SDK: 410.1022 (LeaseNotFound) - fast
    Note over SDK: RetryAfter(Zero) -> mark EUS<br/>unavailable, retry on WUS2
    SDK->>W: GET (attempt 2)
    W-->>SDK: 410.1022
    SDK->>E: GET (attempt 3, prefetch chain 2)
    E-->>SDK: 410.1022

    Note over App: t=30s  CT FIRES
    rect rgb(255, 230, 230)
        Note over SDK: BEFORE FIX:<br/>SDK is inside await Task.Delay(delayForNextRequest)<br/>with NO token. Sleeps the full HttpTimeoutPolicy budget.
        SDK->>W: GET (attempt 4)
        W-->>SDK: 410.1022 (5s later)
        SDK->>E: GET (attempt 5)
        E-->>SDK: 410.1022 (5s later)
        Note over SDK: Parallel prefetch chains<br/>(FeedIterator fans out 4-8)<br/>each unwind independently
    end
    Note over App: t~100s  All chains drain<br/>TaskCanceledException (70s LATE)
    SDK-->>App: TaskCanceledException

Loading

sequenceDiagram
    autonumber
    participant App as Customer App (CT = 30s)
    participant SDK as SDK Retry Loop
    participant E as East US
    participant W as West US 2

    Note over App,W: t=0s  Query starts, CT armed for 30s
    App->>SDK: ReadFeedAsync(ct)
    SDK->>E: GET (attempt 1)
    E-->>SDK: 410.1022
    SDK->>W: GET (attempt 2)
    W-->>SDK: 410.1022
    SDK->>E: GET (attempt 3)
    E-->>SDK: 410.1022

    Note over App: t=30s  CT FIRES
    rect rgb(220, 255, 220)
        Note over SDK: AFTER FIX:<br/>await Task.Delay(delay, ct) throws immediately.<br/>Next GetEffectiveRangesAsync also throws at entry-point.
        SDK--xSDK: OperationCanceledException
        Note over SDK: Each prefetch chain hits the same<br/>cancel point on its very next await
    end
    Note over App: t~30.x s
    SDK-->>App: TaskCanceledException (on time)

Loading

Net: the fix doesn't shorten any individual attempt. It makes the inter-attempt Task.Delays and the routing entry points respond to the CT — so the moment the customer's deadline fires, every chain bails out on its very next await instead of waiting out the full HTTP timeout first.

Cancellation token fixes — what changed

Customer-facing fix (transport layer)

• CosmosHttpClientCore.ProcessMessageAsync: the gateway HTTP retry loop's inter-attempt delay was await Task.Delay(delayForNextRequest). It is now await Task.Delay(delayForNextRequest, cancellationToken). Per-operation gateway requests (point ops, queries, etc.) flow the caller's token into this delay so a cancellation during backoff no longer overruns the caller's deadline. Background metadata refreshes that call this same path pass CancellationToken.None, so they are unaffected.

Entry-point fast-fail

• FeedRangeInternal.GetEffectiveRangesAsync (virtual + Epk / PartitionKey / PartitionKeyRange overrides): added cancellationToken.ThrowIfCancellationRequested() at the top so an already-cancelled caller fails fast before doing any cache work.

Caller-stack plumbing

CT is now threaded through (and ultimately reaches the FeedRangeInternal.GetEffectiveRangesAsync entry-point check above):

• ContainerCore.GetFeedRangesAsync / GetPartitionKeyRangesAsync
• CosmosQueryClient / CosmosQueryClientCore.TryGetTargetPartitionKeyRangeByFeedRangeAsync
• NetworkAttachedDocumentContainer.MonadicGetChildRangeAsync / pagination paths
• FeedRangeCompositeContinuation.HandleSplitAsync
• StandByFeedContinuationToken delegate signature

Deliberately NOT changed

• PartitionKeyRangeCache populator (the populator delegates inside AsyncCacheNonBlocking): the shared metadata refresh task is fetched once and awaited by all concurrent callers. Letting one caller's token cancel this shared task would cancel the refresh for every other caller (share-of-fate). Callers fast-fail at their own entry points instead; the populator itself runs with CancellationToken.None.
• CosmosQueryClient.GetTargetPartitionKeyRangesAsync: routes straight into TryGetOverlappingRangesAsync / the shared populator above. No CT parameter — same share-of-fate rationale.
• Retry policies (ClientRetryPolicy, BulkExecutionRetryPolicy, ResourceThrottleRetryPolicy, WebExceptionRetryPolicy): BackoffRetryUtility.ExecuteAsync already calls cancellationToken.ThrowIfCancellationRequested() before each attempt and uses Task.Delay(backoffTime, cancellationToken) for backoff. Short-circuiting inside ShouldRetryAsync would preempt important side-effects performed during exception classification — most notably the cross-region failover marking in ClientRetryPolicy's HttpRequestException handler (TryMarkEndpointUnavailableForPkRange / MarkEndpointUnavailableForRead). The existing CancellationTokenDoesNotCancelFailover tests pin that contract.

Test coverage

New tests

• Fault injection (Microsoft.Azure.Cosmos.EmulatorTests): lease-not-found injection on point operations and gateway metadata-read paths; verifies cross-region retry behavior and address-cache invalidation. FaultInjectionMetadataBarrierRequestDelay_HonorsCancellationToken covers the consistency-barrier loop using the new MetadataBarrierRequest operation type + ResponseHeaderOverride server-error result.
• CT honor regression tests (Microsoft.Azure.Cosmos.Tests): entry-point fast-fail on FeedRangeInternal.GetEffectiveRangesAsync overrides (Epk / PartitionKey / PartitionKeyRange); CT propagation through CosmosHttpClientCore retry-delay path; CT propagation through query pagination call sites; Container.IsFeedRangePartOfAsync end-to-end CT honor (emulator).

Existing tests preserved

• CancellationTokenDoesNotCancelFailover (Eventual) / (Session) — pins the rule that cross-region failover marking runs to completion even when the caller's token has fired during a retry classification step.
• All 30 tests matching CancellationToken|WithCancelledToken|PropagatesCancellation|AvailabilityStrategyUnit pass.

Risk

• Transport-layer Task.Delay(ct) is the only behavioral change for in-flight retries on the gateway path. Cancelled-during-retry-backoff now surfaces OperationCanceledException; previously, it would complete the delay and either retry or surface the underlying exception. This matches the documented CancellationToken contract.
• No shared-cache behavior changes (PKRC populator unchanged).
• No retry-policy classification or failover behavior changes.

[tvaron3]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tvaron3]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[github-actions]

📝 Changelog reminder (non-blocking)

This PR touches shipped source but does not appear to update the
corresponding changelog.md.

Touched (and missing entry for):

• Microsoft.Azure.Cosmos/FaultInjection/src/** ⇒ expected an entry in Microsoft.Azure.Cosmos/FaultInjection/changelog.md (### Unreleased)

How to decide

Use the rubric in .github/copilot-instructions.md ("Changelog
classifier") or in CONTRIBUTING.md ("Changelog entry"). Quick
version:

• Customer-observable change (behavior, perf, memory, API) ⇒ add an entry, even if the PR is [Internal].
• Test-only / CI-only / doc-only / pure-internal-refactor with zero customer-observable effect ⇒ no entry, add the no-changelog-needed label to silence this check.
• Unsure? Default to adding a one-line entry under #### Other Changes. Reviewer will adjust.

This check is non-blocking — merge is not gated on it. The
reviewer is responsible for the final classification.

[tvaron3]

/azp run

[azure-pipelines]

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

[tvaron3]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).