[Internal] DTS: Adds retries in DTS when isRetriable is true and on timeout#5689
Open
Meghana-Palaparthi wants to merge 25 commits intomainfrom
Open
[Internal] DTS: Adds retries in DTS when isRetriable is true and on timeout#5689Meghana-Palaparthi wants to merge 25 commits intomainfrom
Meghana-Palaparthi wants to merge 25 commits intomainfrom
Conversation
Meghana-Palaparthi
requested review from
FabianMeiswinkel,
Pilchie,
adityasa,
khdang,
kirankumarkolli,
kirillg,
neildsh and
sboshra
as code owners
Member
|
@sdkReviewAgent |
Member
|
sdkReviewAgent | Status: ⏳ Queued Review requested by @xinlian12. I'll start shortly. |
Member
|
sdkReviewAgent | Status: 🔍 Reviewing I'm reviewing this PR now. I'll post my findings as comments when done. |
xinlian12
reviewed
Mar 19, 2026
xinlian12
reviewed
Mar 19, 2026
xinlian12
reviewed
Mar 19, 2026
xinlian12
reviewed
Mar 19, 2026
# Conflicts: # Microsoft.Azure.Cosmos/src/DistributedTransaction/DistributedTransactionCommitter.cs # Microsoft.Azure.Cosmos/src/DistributedTransaction/DistributedTransactionOperationResult.cs # Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/DistributedTransaction/DistributedTransactionCommitterTests.cs
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Member
|
✅ Review complete (40:04) Posted 1 inline comment(s). Steps: ✓ context, correctness, cross-sdk, design, history, past-prs, synthesis, test-coverage |
xinlian12
reviewed
Apr 17, 2026
Member
|
✅ Review complete (37:13) Posted 1 inline comment(s). Steps: ✓ context, correctness, cross-sdk, design, history, past-prs, synthesis, test-coverage |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Refactor DTX retry policy into ClientRetryPolicy
- Add DTX-specific retry block in ClientRetryPolicy.ShouldRetryInternalAsync gated on IsDistributedTransactionRequest, covering:
* 449/5352 (DtcCoordinatorRaceConflict): honors Retry-After header
* 500/5411,5412,5413 (DtcLedgerFailure, DtcAccountConfigFailure, DtcDispatchFailure): transient infra failures safe to retry via idempotency token.
- Uses dedicated dtxRetryCount field (max 10) to avoid the single-master write restrictions in ShouldRetryOnUnavailableEndpointStatusCodes.
- 408 and 403.3 were already handled by the pipeline for all request types.
- 429/3200 (DtcLedgerThrottled) was already handled by ResourceThrottleRetryPolicy.
- Simplify DistributedTransactionCommitter.ExecuteCommitWithRetryAsync to only loop on the isRetriable JSON body flag (the one signal not visible at the pipeline level). Remove IsRetriableByStatusCode and the CosmosException-when-408 catch block.
- Update ClientRetryPolicyTests with DTX-specific tests:
* 408, 449/5352, 500/5411-5413 retry correctly for DTX requests
* Non-DTX requests are NOT retried on the same codes (gating tests)
* Budget exhaustion (10 retries) returns NoRetry
- Update DistributedTransactionCommitterTests: remove all status-code retry tests (now covered in ClientRetryPolicyTests), rename tests that assumed CosmosException was caught by the outer loop.
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Meghana-Palaparthi
requested review from
jeet1995 and
kundadebdatta
as code owners
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
| { | ||
| private static readonly TimeSpan DefaultRetryBaseDelay = TimeSpan.FromSeconds(1); | ||
|
|
||
| internal const int MaxIsRetriableRetryCount = 100; |
Member
There was a problem hiding this comment.
I am curious, if this retries 100 times, and inside ClientRetryPolicy we retry 100 times, wouldn't it be like 100k retries for a single request?
To avoid this confusion, maybe we can combine these two retry mechanisms into a single layer?
Contributor
There was a problem hiding this comment.
Good catch, I think worst this is 10k retries on a single request (before throttling) . If we want to combine maybe a passing response header to the client retry policy could work? If we wanted to keep the 2 retry layers and not have a new header we can probably just limit it to 3-5 retries here to be safe.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request introduces robust retry logic for distributed transaction commits and improves the handling and parsing of distributed transaction responses in the Cosmos DB SDK. The changes ensure that commit operations are retried safely in the event of timeouts or retriable errors, enhance diagnostics, and make response parsing more resilient. Additionally, the request and response classes are refactored for safer stream handling and improved reliability.
Reference doc for DTX retries: dtx-sdk-response-status-codes.md
Distributed transaction commit improvements:
DistributedTransactionCommitterResponse parsing and diagnostics enhancements:
isRetriableandserverDiagnosticsfields, and improved resilience to partial JSON parsing failures.IsRetriableproperty toDistributedTransactionResponseand ensured it is correctly populated from server responses.Request stream handling improvements:
DistributedTransactionServerRequestto use a pre-serialized byte array for the request body, enabling safe creation of new memory streams for each retry and preventing disposal issues.Reliability and correctness fixes:
DistributedTransactionResponse.DistributedTransactionOperationResultto throw explicit exceptions on failure.Miscellaneous:
Type of change
Please delete options that are not relevant.
Closing issues
To automatically close an issue: closes #IssueNumber