[azcosmos] Control-plane retry: cross-region failover preempted by caller ctx cancellation

[NaluTripician]

Summary

Same class of defect as Azure/azure-cosmos-dotnet-v3#5805 (fix in Azure/azure-cosmos-dotnet-v3#5806).

cosmos_client_retry_policy.go honors the caller's ctx.Err() and exits the retry loop before consulting the cross-region retry decision on control-plane reads. When an unhealthy preferred region triggers internal HTTP timeout escalation, a caller whose ctx deadline fires during escalation never gets the cross-region attempt that the SDK advertises.

The global endpoint manager policy (cosmos_global_endpoint_manager_policy.go:22-24) already uses context.WithoutCancel to shield its own internal retries from caller cancellation — the same pattern is missing from the client retry policy for control-plane reads.

Note: azcosmos has no collection / partition-key-range cache today, so the exact customer-visible failure mode of the .NET bug does not reproduce. This issue is filed to:

1. Fix the existing gap on GetAccountProperties and region discovery, which are affected today.
2. Prevent the defect class from silently recurring when / if a collection or partition cache is later added (at which point control-plane reads gate data-plane traffic, as in .NET / Python).

Files / lines

• sdk/data/azcosmos/cosmos_client_retry_policy.go — client retry policy (~lines 24-79)
• sdk/data/azcosmos/cosmos_global_endpoint_manager_policy.go:22-24 — reference for the context.WithoutCancel pattern already used locally.

Repro (sketch)

1. Fresh cosmos.Client.
2. Preferred regions [A, B]; A unreachable (blackhole / 503).
3. Call client.AccountProperties(...) with a context bounded by context.WithTimeout(ctx, 36*time.Second).
4. Observe: context deadline fires, no attempt against region B is made.

Expected behavior

When the caller's ctx is cancelled during an in-flight control-plane attempt, the retry policy is consulted; if it indicates a cross-region retry, one bounded attempt against the next region executes on a retry-scoped context derived with context.WithoutCancel + context.WithTimeout(grace).

Proposed fix direction

retryCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), graceWindow)
defer cancel()
// attempt against next region using retryCtx

Surface the original ctx.Err()-bearing error as the cause if the grace attempt also fails or expires. Grace window of 10s aligns with the .NET fix.

Cross-references

This defect class was identified in a cross-SDK investigation prompted by the .NET fix. Tracking issues:

• .NET (fix landed): Metadata retry: cross-region failover preempted by caller cancellation during control-plane HTTP timeout escalation azure-cosmos-dotnet-v3#5805 — PR Metadata Retry: Fixes cross-region failover preempted by caller cancellation azure-cosmos-dotnet-v3#5806
• Python: [Cosmos] Metadata retry: cross-region failover preempted when caller cancels during control-plane timeout escalation azure-sdk-for-python#46471
• Rust: [cosmos] Metadata retry: dropped caller future silently cancels cross-region failover azure-sdk-for-rust#4253
• Java: not filed — Reactor's retryWhen operator structurally isolates subscription cancellation from the retry decision, so the defect does not reproduce.

/cc @NaluTripician

[github-actions]

Thank you for your feedback. Tagging and routing to the team member(s) best able to assist.

//cc: @MehaKaushik @Pilchie @wmengmsft

[github-actions]

🎯 Agentic Issue Triage

Summary: The azcosmos client retry policy exits the retry loop on caller ctx cancellation before consulting the cross-region retry decision, meaning GetAccountProperties and region discovery calls can silently skip cross-region failover when the caller's context deadline fires during an in-flight control-plane request.

📋 Issue Details

Module: github.com/Azure/azure-sdk-for-go/sdk/data/azcosmos
Affected API: cosmos_client_retry_policy.go (~lines 24-79), cosmos_global_endpoint_manager_policy.go
Scenarios:

• Preferred region A is unreachable (blackhole / 503); region B is healthy
• Caller sets context.WithTimeout(ctx, 36*time.Second) on client.AccountProperties(...)
• Context deadline fires during in-flight attempt against region A; SDK never attempts region B

Root ask: Fix cosmos_client_retry_policy.go to use context.WithoutCancel + a grace-window timeout (like cosmos_global_endpoint_manager_policy.go already does) when executing a cross-region control-plane retry, so caller cancellation does not preempt the retry decision.

🔎 Debugging / Reproduction Notes

The issue author provides a clear sketch repro:

1. Create a cosmos.Client with preferred regions [A, B]; make A unreachable.
2. Call client.AccountProperties(...) with context.WithTimeout(ctx, 36s).
3. Observe: no attempt against region B is made.

The reference pattern (context.WithoutCancel + grace timeout of ~10s) already exists in cosmos_global_endpoint_manager_policy.go:22-24 and is the proposed fix direction. The original error should be surfaced as the cause if the grace attempt also fails.

Cross-SDK tracking: .NET fix landed (Azure/azure-cosmos-dotnet-v3#5806), Python filed (#46471), Rust filed (#4253).

Suggested investigation steps:

1. Confirm cosmos_client_retry_policy.go checks ctx.Err() before the cross-region retry branch and returns early.
2. Review whether GetAccountProperties and ListDatabases (all control-plane reads) funnel through this policy.
3. Implement fix: derive a retryCtx via context.WithTimeout(context.WithoutCancel(ctx), graceWindow) for the cross-region attempt; surface the original error as cause on failure.
4. Add a unit/integration test simulating context cancellation mid-flight with multi-region setup to verify the retry fires.

🏷️ Label Confidence

• Category: Client — module sdk/data/azcosmos is under sdk/data/, not sdk/resourcemanager/, so it is a client-side SDK issue
• Service: Cosmos — module is azcosmos (Azure Cosmos DB Go SDK); confirmed by reference issue #22868 using the same Cosmos service label
• Confidence: High — the issue title and body explicitly name the sdk/data/azcosmos module and specific files; no ambiguity between services or categories

👥 Owner Routing

• CODEOWNERS scan (bottom-to-top): First # ServiceLabel: %Cosmos entry encountered (line 45–46) matched before the second entry (line 41–43)
• Matched entry: # ServiceLabel: %Cosmos (line 45) — label matches predicted Cosmos service label
• AzureSdkOwners: none listed in matched entry
• ServiceOwners: @MehaKaushik, @Pilchie, @Wmengmsft
• Routing action: Applied "Service Attention" and "needs-team-attention"; posted mention_owners ping to ServiceOwners
• Scan notes: Entry at line 41–43 (also %Cosmos, with AzureSdkOwners: @jhendrixMSFT``) was skipped — the bottom-to-top scan stops at the first match (line 45–46)

Warning

⚠️ Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• releaseassets.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "releaseassets.githubusercontent.com"

See Network Configuration for more information.

Generated by Agentic Triage for issue #26649 · ● 332.2K · ◷