Session Consistency: Adds SessionTokenMismatchRetryPolicy optimization through customer supplied region switch hints#4976
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
dibahlfi
changed the title
dibahlfi
changed the title
dibahlfi
changed the title
dibahlfi
changed the title
| /// <summary> | ||
| /// hints which guide SDK-internal retry policies on how early to switch retries to a different region. | ||
| /// </summary> | ||
| public Boolean RemoteRegionPreferred { get; set; } = false; |
There was a problem hiding this comment.
Can we please expand the text on when the switch will be performed? Like after all the replicas are tried locally?
There was a problem hiding this comment.
return type 'bool' (Object vs primitive type)
| /// The minimum value is 100ms - this minimum is enforced to provide a way for the local region to catch-up on replication lag. The default value is 500ms - as a recommendation ensure that this value is higher than the steady-state | ||
| /// replication latency between the regions you chose | ||
| /// </summary> | ||
| public TimeSpan MinInRegionRetryTime { get; set; } = ConfigurationManager.GetMinRetryTimeInLocalRegionWhenRemoteRegionPreferred(); |
There was a problem hiding this comment.
For a single master write region, after one iteration over all replicas we can yield fast right?
There was a problem hiding this comment.
- PPAF will make it non-trivial as this decision is at the partition level
- In-case of regional fail-overs, Client might have stale regions info and might result in availability loss too, so some MinInRegion retry might be better.
There was a problem hiding this comment.
Yes, this one seems like an optimization. As discussed, I will set up a meeting with Fabien and Abhijeet and you to go over it and we will plan as a follow up PR.
| /// hints which guide SDK-internal retry policies on how early to switch retries to a different region. | ||
| /// </summary> | ||
| Boolean RemoteRegionPreferred { get;} | ||
|
|
|
|
||
| private Boolean shouldRetryLocally() | ||
| { | ||
| if(! this.sessionRetryOptions.RemoteRegionPreferred) |
There was a problem hiding this comment.
Should it guarantee that primary in the region is tried?
I think current implementation doesn't guarantee it.
There was a problem hiding this comment.
That is correct - it does not guarantee it. A write within a region is only not just false progress when it has been committed to a quorum of replica - so, it would never be just the primary (unless quorum loss) that has a change - and for pending writes the primary replica is not the first one writing a change. So, there is no need to force including the primary (unless for reduce replica set size consistency reader/quorum reader do it anyway)
| private IDictionary<string, System.Uri> writeRegionMap; | ||
|
|
||
| // to run code before running each test | ||
| [TestInitialize] |
There was a problem hiding this comment.
Who sets the ENV variable?
There was a problem hiding this comment.
client can either set the environment variable or pass the value in the sessionRetryOptions. I am using the later in the Test cases written.
| { | ||
| SessionRetryOptions = new SessionRetryOptions() | ||
| { | ||
| RemoteRegionPreferred = true |
There was a problem hiding this comment.
For MM, min threshold also a good to have right? (esp for session consistency)
There was a problem hiding this comment.
Other unit tests have those options. Am I missing something?
| } | ||
|
|
||
| [TestMethod] | ||
| public void SessionRetryOptionsMinMaxRetriesCountEnforcedTest() |
There was a problem hiding this comment.
nit: UT units with out emulator dependency can be moved to UT project
| HttpStatusCode statusCode = operationExecutionResult.StatusCode; | ||
|
|
||
| int executionDuration = operationExecutionResult.Duration; | ||
| Console.WriteLine($" status code is {statusCode}"); |
There was a problem hiding this comment.
nit: Prefer System.Diagnostics.Trace.Trace.... API's
| { | ||
| Assert.IsTrue(executionDuration > sessionTokenMismatchDefaultWaitTime); | ||
| } | ||
|
|
There was a problem hiding this comment.
Validation: coverage for the expected retry count?
There was a problem hiding this comment.
Already happening in the ReadOperationWithReadSessionUnavailableTest and WriteOperationWithReadSessionUnavailableTest tests methods.
| private const int defaultInitialBackoffTimeInMilliseconds = 5; | ||
| private const int defaultMaximumBackoffTimeInMilliseconds = 500; | ||
| private const int backoffMultiplier = 2; | ||
| private const int backoffMultiplier = 5; // before it was very aggressive |
There was a problem hiding this comment.
Is it same on JAVA SDK?
| @@ -8,18 +8,26 @@ | |||
| /// </summary> | |||
| public sealed class SessionRetryOptions : ISessionRetryOptions | |||
There was a problem hiding this comment.
Two possible options
- In CosmsoClientOptions just take a bool as contract
- Expose a new type of SessionRetryOptions
If we environ this surface area expanding richly forward then later makes sense, otherwise feels like overkill thoughts?
There was a problem hiding this comment.
It also then makes the RetryTime and RetryCunt as implementation details not as public contract.
There was a problem hiding this comment.
make sense. I will make the changes to take bool as a contract.
| /// provides SessionTokenMismatchRetryPolicy optimization through customer supplied region switch hints | ||
| /// </summary> | ||
| internal SessionRetryOptions SessionRetryOptions | ||
| internal bool EnableRemoteRegionPreferredForSessionRetry |
There was a problem hiding this comment.
This should be conditionally public or internal - it is teh public surface area that custoemrs would use
Like
#if PREVIEW public #else internal #endif
See this PR for an example - #5077
| /// <param name="enableRemoteRegionPreferredForSessionRetry"></param> | ||
| /// <returns>The <see cref="CosmosClientBuilder"/> object</returns> | ||
| public CosmosClientBuilder WithSessionRetryOptions(SessionRetryOptions sessionRetryOptions) | ||
| public CosmosClientBuilder WithEnableRemoteRegionPreferredForSessionRetry(bool enableRemoteRegionPreferredForSessionRetry) |
There was a problem hiding this comment.
Same public for preview / internal for GA package - see #5077
4 participants
Pull Request Template
Description
This PR will allow application developers to configure hints through a SessionRetryOptions instance which will signal to the SDK whether to pin retries on the local region or move quicker to a remote region when READ_SESSION_NOT_AVAILABLE errors are thrown.
Adding 2 Diagnostics:
1st Diagnostic for method ReadOperationWithReadSessionUnavailableTest()
[DataRow(FaultInjectionOperationType.ReadItem, 2, true , DisplayName = "Validate Read Item operation with remote region preferred.")]
{"Summary":{"DirectCalls":{"(404, 1002)":12,"(200, 0)":1},"GatewayCalls":{"(200, 0)":1}},"name":"ReadItemAsync","start datetime":"2025-02-10T23:29:31.706Z","duration in milliseconds":1360.4163,"data":{"Client Configuration":{"Client Created Time Utc":"2025-02-10T23:29:30.4112757Z","MachineId":"hashedMachineName:397d2b79-d17a-106f-f8e6-1556dfbd2678","NumberOfClientsCreated":2,"NumberOfActiveClients":2,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.43.1|2|X64|Microsoft Windows 10.0.22631|.NET 6.0.36|L|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: Session, prgns:[South Central US, East US, Central US], apprgn: )","ProcessorCount":22}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":1358.5185,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0017},{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":1358.326,"data":{"System Info":{"systemHistory":[{"dateUtc":"2025-02-10T23:29:29.8038049Z","cpu":4.469,"memory":18640920.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32765,"minThreads":22,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":1358.3105,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":1358.297,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":514.3986,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":514.394,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":387.6739,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2025-02-10T23:29:31.8449636Z","DurationInMs":133.4363,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":0,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14067/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133835124557722688s/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14025 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14067 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14339 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:31.7114029Z", "durationInMs": 0.0417},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:31.7114446Z", "durationInMs": 114.2982},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:31.8257428Z", "durationInMs": 12.7226},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:31.8384654Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:31.8385681Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0},"requestSizeInBytes":550},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:29:31.8514739Z","DurationInMs":0.126,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":0,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14379/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133833555352673277p/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14025 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14067 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14339 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:31.8462297Z", "durationInMs": 0.1933},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:31.8464230Z", "durationInMs": 0.7099},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:31.8471329Z", "durationInMs": 3.8461},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:31.8509790Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:31.8510141Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0},"requestSizeInBytes":544},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:29:31.9563605Z","DurationInMs":104.6865,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":0,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14025/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133833074895650718s/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14025 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14067 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14339 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:31.8515427Z", "durationInMs": 0.043},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:31.8515857Z", "durationInMs": 104.0384},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:31.9556241Z", "durationInMs": 0.2686},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:31.9558927Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:31.9559413Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0},"requestSizeInBytes":544},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:29:32.0715836Z","DurationInMs":115.0568,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":0,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14339/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133821512564091753s/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14025 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14067 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14339 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:31.9564158Z", "durationInMs": 0.0405},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:31.9564563Z", "durationInMs": 114.1598},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:32.0706161Z", "durationInMs": 0.4751},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:32.0710912Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:32.0711597Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0},"requestSizeInBytes":544},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:29:32.0779410Z","DurationInMs":0.0878,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":0,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14339/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133821512564091753s/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14025 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14067 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14339 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:32.0775212Z", "durationInMs": 0.0195},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:32.0775407Z", "durationInMs": 0.017},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:32.0775577Z", "durationInMs": 0.1281},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:32.0776858Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:32.0776966Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0},"requestSizeInBytes":550},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:29:32.0782158Z","DurationInMs":0.0531,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":0,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14025/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133833074895650718s/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14025 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14067 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14339 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:32.0779734Z", "durationInMs": 0.0084},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:32.0779818Z", "durationInMs": 0.0054},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:32.0779872Z", "durationInMs": 0.0674},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:32.0780546Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:32.0780595Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0},"requestSizeInBytes":544},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:29:32.0784498Z","DurationInMs":0.0473,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":0,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14067/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133835124557722688s/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14025 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14067 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14339 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:32.0782334Z", "durationInMs": 0.0072},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:32.0782406Z", "durationInMs": 0.0054},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:32.0782460Z", "durationInMs": 0.059},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:32.0783050Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:32.0783097Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0},"requestSizeInBytes":544},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:29:32.0786743Z","DurationInMs":0.0484,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":0,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14379/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133833555352673277p/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14025 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14067 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14339 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:32.0784654Z", "durationInMs": 0.0062},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:32.0784716Z", "durationInMs": 0.0056},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:32.0784772Z", "durationInMs": 0.0575},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:32.0785347Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:32.0785400Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0},"requestSizeInBytes":544},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:29:32.0944597Z","DurationInMs":0.1071,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":0,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14339/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133821512564091753s/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14025 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14067 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14339 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:32.0938710Z", "durationInMs": 0.0209},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:32.0938919Z", "durationInMs": 0.0227},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:32.0939146Z", "durationInMs": 0.1675},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:32.0940821Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:32.0940940Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0},"requestSizeInBytes":550},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:29:32.0948532Z","DurationInMs":0.0746,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":0,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14025/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133833074895650718s/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14025 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14067 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14339 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:32.0945065Z", "durationInMs": 0.0125},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:32.0945190Z", "durationInMs": 0.0063},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:32.0945253Z", "durationInMs": 0.0951},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:32.0946204Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:32.0946277Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0},"requestSizeInBytes":544},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:29:32.0951847Z","DurationInMs":0.0669,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":0,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14379/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133833555352673277p/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14025 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14067 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14339 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:32.0948783Z", "durationInMs": 0.0094},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:32.0948877Z", "durationInMs": 0.0061},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:32.0948938Z", "durationInMs": 0.0863},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:32.0949801Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:32.0949859Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0},"requestSizeInBytes":544},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:29:32.0955021Z","DurationInMs":0.0612,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":0,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14067/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133835124557722688s/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14025 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14067 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)","(port: 14339 | status: Unknown | lkt: 2/10/2025 11:29:31 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:32.0952080Z", "durationInMs": 0.0094},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:32.0952174Z", "durationInMs": 0.0048},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:32.0952222Z", "durationInMs": 0.0875},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:32.0953097Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:32.0953154Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0},"requestSizeInBytes":544},"TransportException":null}}]},"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","ResponseTimeUtc":"2025-02-10T23:29:32.2222780Z","StatusCode":404,"SubStatusCode":1002,"RequestCharge":0,"RequestUri":"dbs/availabilityStrategyTestDb/colls/sessionRetryPolicy/docs/0b8a35a9-1396-4d14-bb2d-d83abbeecb5c","ErrorMessage":"Microsoft.Azure.Documents.NotFoundException: The read session is not available for the input session token.\r\nActivityId: 7ef8e2a5-4fb7-41b6-86ff-60a313abb789, Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, Windows/10.0.22631 cosmos-netstandard-sdk/3.18.0\r\n at Microsoft.Azure.Documents.ConsistencyReader.ReadSessionAsync(DocumentServiceRequest entity, ReadMode readMode) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\ConsistencyReader.cs:line 335\r\n at Microsoft.Azure.Documents.BackoffRetryUtility
1.ExecuteRetryAsync[TParam,TPolicy](Func1 callbackMethod, Func3 callbackMethodWithParam, Func2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy1 retryPolicyWithArg, Func1 inBackoffAlternateCallbackMethod, Func2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\BackoffRetryUtility.cs:line 171\r\n at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\IRetryPolicy.cs:line 40\r\n at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync[TParam,TPolicy](Func1 callbackMethod, Func3 callbackMethodWithParam, Func2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy1 retryPolicyWithArg, Func1 inBackoffAlternateCallbackMethod, Func2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\BackoffRetryUtility.cs:line 202\r\n at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync[TParam,TPolicy](Func1 callbackMethod, Func3 callbackMethodWithParam, Func2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy1 retryPolicyWithArg, Func1 inBackoffAlternateCallbackMethod, Func2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\BackoffRetryUtility.cs:line 246\r\n at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<b__0>d.MoveNext() in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\ReplicatedResourceClient.cs:line 158\r\n--- End of stack trace from previous location ---\r\n at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func1 executeAsync, Func1 prepareRequest, IRequestRetryPolicy2 policy, CancellationToken cancellationToken, Func1 inBackoffAlternateCallbackMethod, Nullable1 minBackoffForInBackoffCallback) in C:\\Users\\dikshibahl\\source\\repos\\azure-cosmos-dotnet-v3\\Microsoft.Azure.Cosmos\\src\\direct\\RequestRetryUtility.cs:line 79\r\n at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) in C:\\Users\\dikshibahl\\source\\repos\\azure-cosmos-dotnet-v3\\Microsoft.Azure.Cosmos\\src\\direct\\IRetryPolicy.cs:line 40\r\n at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func1 executeAsync, Func1 prepareRequest, IRequestRetryPolicy2 policy, CancellationToken cancellationToken, Func1 inBackoffAlternateCallbackMethod, Nullable1 minBackoffForInBackoffCallback) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\RequestRetryUtility.cs:line 102\r\n at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func1 executeAsync, Func1 prepareRequest, IRequestRetryPolicy2 policy, CancellationToken cancellationToken, Func1 inBackoffAlternateCallbackMethod, Nullable`1 minBackoffForInBackoffCallback) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\RequestRetryUtility.cs:line 164\r\n at Microsoft.Azure.Documents.StoreClient.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken, IRetryPolicy retryPolicy) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\StoreClient.cs:line 135\r\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Handler\TransportHandler.cs:line 122\r\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Handler\TransportHandler.cs:line 33","RequestSessionToken":null,"ResponseSessionToken":null,"BELatencyInMs":null}}}]}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":841.3216,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":841.317,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":841.2083,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be170.documents.azure.com:14371/apps/ee056cc0-380c-4306-a3a7-9cd9cfd25680/services/da8775e3-b152-438d-bcef-24e0bb407c36/partitions/7a3c58d8-3d15-4841-9156-4db4aff2fbb3/replicas/133763831364082482s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2025-02-10T23:29:32.2257205Z","EndTimeUTC":"2025-02-10T23:29:32.6095926Z","TargetEndpoint":"https://sdkdev-dikshi-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fTFMgAA%3d%3d%2fcolls%2fTFMgAI8vlks%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2025-02-10T23:29:33.0623921Z","DurationInMs":451.8791,"ResourceType":"Document","OperationType":"Read","RequestSessionToken":"0:19#367#1=560#3=-1","LocationEndpoint":"https://sdkdev-dikshi-eastus.documents.azure.com/","StoreResult":{"ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"Ok","SubStatusCode":"Unknown","LSN":617,"PartitionKeyRangeId":"0","GlobalCommittedLSN":367,"ItemLSN":617,"UsingLocalLSN":true,"QuorumAckedLSN":-1,"SessionToken":"19#367#1=560#3=-1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":2,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be170.documents.azure.com:14371/apps/ee056cc0-380c-4306-a3a7-9cd9cfd25680/services/da8775e3-b152-438d-bcef-24e0bb407c36/partitions/7a3c58d8-3d15-4841-9156-4db4aff2fbb3/replicas/133763831364082482s/","RequestCharge":1,"RetryAfterInMs":null,"BELatencyInMs":"0.241","ReplicaHealthStatuses":["(port: 14345 | status: Unknown | lkt: 2/10/2025 11:29:32 PM)","(port: 14004 | status: Unknown | lkt: 2/10/2025 11:29:32 PM)","(port: 14371 | status: Unknown | lkt: 2/10/2025 11:29:32 PM)","(port: 14408 | status: Unknown | lkt: 2/10/2025 11:29:32 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:29:32.6104169Z", "durationInMs": 0.0251},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:29:32.6104420Z", "durationInMs": 390.8181},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:29:33.0012601Z", "durationInMs": 0.5297},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:29:33.0017898Z", "durationInMs": 60.4034},{"event": "Received", "startTimeUtc": "2025-02-10T23:29:33.0621932Z", "durationInMs": 0.1928},{"event": "Completed", "startTimeUtc": "2025-02-10T23:29:33.0623860Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2025-02-10T23:29:32.9447337Z","lastSend":"2025-02-10T23:29:32.9448777Z","lastReceive":"2025-02-10T23:29:33.0007363Z"},"requestSizeInBytes":550,"responseMetadataSizeInBytes":501,"responseBodySizeInBytes":347},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2025-02-10T23:29:32.2257482Z","DurationInMs":383.7589,"RequestUri":"https://sdkdev-dikshi-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fTFMgAA%3d%3d%2fcolls%2fTFMgAI8vlks%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0","ResourceType":"Document","HttpMethod":"GET","ActivityId":"7ef8e2a5-4fb7-41b6-86ff-60a313abb789","StatusCode":"OK"}]}}}]}]}]}]}]}]},{"name":"Response Serialization","duration in milliseconds":0.1263}]}2nd Diagnostics for Test Method -
[DataRow(FaultInjectionOperationType.CreateItem, 2, true, DisplayName = "Validate Write Item operation with remote region preferred.")]
WriteOperationWithReadSessionUnavailableTest()
{"Summary":{"DirectCalls":{"(404, 1002)":3,"(201, 0)":1},"GatewayCalls":{"(200, 0)":4,"(304, 0)":1}},"name":"CreateItemAsync","start datetime":"2025-02-10T23:32:04.330Z","duration in milliseconds":1681.0311,"data":{"Client Configuration":{"Client Created Time Utc":"2025-02-10T23:32:03.8561394Z","MachineId":"hashedMachineName:397d2b79-d17a-106f-f8e6-1556dfbd2678","NumberOfClientsCreated":2,"NumberOfActiveClients":2,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.43.1|2|X64|Microsoft Windows 10.0.22631|.NET 6.0.36|L|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: Session, prgns:[South Central US, East US, Central US], apprgn: )","ProcessorCount":22}},"children":[{"name":"ItemSerialize","duration in milliseconds":5.3366},{"name":"Get PkValue From Stream","duration in milliseconds":85.9919,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0101},{"name":"Read Collection","duration in milliseconds":31.3418,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2025-02-10T23:32:04.3520803Z","DurationInMs":30.2406,"RequestUri":"https://sdkdev-dikshi-southcentralus.documents.azure.com/dbs/availabilityStrategyTestDb/colls/sessionRetryPolicy","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"ad375117-7a50-4584-83e6-e2ffcdc5e11d","StatusCode":"OK"}]}}}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":1581.5763,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0031},{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":1576.4022,"data":{"System Info":{"systemHistory":[{"dateUtc":"2025-02-10T23:32:03.3770308Z","cpu":4.470,"memory":18468000.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32765,"minThreads":22,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":1576.3781,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":1576.3177,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":495.2259,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":495.2231,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":380.8227,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14379/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133833555352673277p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14025/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133833074895650718s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14067/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133835124557722688s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14339/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133821512564091753s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2025-02-10T23:32:04.5098182Z","EndTimeUTC":"2025-02-10T23:32:04.5324247Z","TargetEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com//addresses/?$resolveFor=dbs%2fTFMgAA%3d%3d%2fcolls%2fTFMgAI8vlks%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2025-02-10T23:32:04.7769799Z","DurationInMs":236.7951,"ResourceType":"Document","OperationType":"Create","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"0c920ce4-1ac8-4a19-860f-6b331e0dcff5","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":-1,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":false,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14379/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133833555352673277p/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Connected | lkt: 2/10/2025 11:32:04 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:32:04.5401932Z", "durationInMs": 0.0497},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:32:04.5402429Z", "durationInMs": 217.4766},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:32:04.7577195Z", "durationInMs": 13.9983},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:32:04.7717178Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:32:04.7718667Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0},"requestSizeInBytes":627,"requestBodySizeInBytes":146},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:32:04.7869189Z","DurationInMs":5.1481,"ResourceType":"Document","OperationType":"Create","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"0c920ce4-1ac8-4a19-860f-6b331e0dcff5","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":-1,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":false,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14379/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133833555352673277p/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Connected | lkt: 2/10/2025 11:32:04 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:32:04.7817755Z", "durationInMs": 0.1915},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:32:04.7819670Z", "durationInMs": 0.7108},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:32:04.7826778Z", "durationInMs": 3.9451},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:32:04.7866229Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:32:04.7866506Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0},"requestSizeInBytes":627,"requestBodySizeInBytes":146},"TransportException":null}},{"ResponseTimeUTC":"2025-02-10T23:32:04.8117021Z","DurationInMs":0.4448,"ResourceType":"Document","OperationType":"Create","RequestSessionToken":"1:1#1#1=1#1=1","LocationEndpoint":"https://sdkdev-dikshi-southcentralus.documents.azure.com/","StoreResult":{"ActivityId":"0c920ce4-1ac8-4a19-860f-6b331e0dcff5","StatusCode":"NotFound","SubStatusCode":"PartitionKeyRangeGone","LSN":-1,"PartitionKeyRangeId":null,"GlobalCommittedLSN":-1,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":-1,"SessionToken":"1#1#1=1#1=1","CurrentWriteQuorum":-1,"CurrentReplicaSetSize":-1,"NumberOfReadRegions":-1,"IsValid":false,"StorePhysicalAddress":"rntbd://cdb-ms-prod-southcentralus1-be65.documents.azure.com:14379/apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133833555352673277p/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":null,"ReplicaHealthStatuses":["(port: 14379 | status: Connected | lkt: 2/10/2025 11:32:04 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:32:04.8112626Z", "durationInMs": 0.0224},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:32:04.8112850Z", "durationInMs": 0.0177},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:32:04.8113027Z", "durationInMs": 0.1577},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:32:04.8114604Z", "durationInMs": "Not Set"},{"event": "Completed", "startTimeUtc": "2025-02-10T23:32:04.8114734Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0},"requestSizeInBytes":627,"requestBodySizeInBytes":146},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2025-02-10T23:32:04.4396145Z","DurationInMs":26.5339,"RequestUri":"https://sdkdev-dikshi-southcentralus.documents.azure.com/dbs/TFMgAA==/colls/TFMgAI8vlks=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"034da6d0-8a92-41b5-94ea-e279074b4404","StatusCode":"OK"},{"StartTimeUTC":"2025-02-10T23:32:04.4675710Z","DurationInMs":23.7709,"RequestUri":"https://sdkdev-dikshi-southcentralus.documents.azure.com/dbs/TFMgAA==/colls/TFMgAI8vlks=/pkranges","ResourceType":"PartitionKeyRange","HttpMethod":"GET","ActivityId":"2dd21d7d-c3f0-4d51-a079-445726bbff4a","StatusCode":"NotModified","ReasonPhrase":"Not Modified"},{"StartTimeUTC":"2025-02-10T23:32:04.5098808Z","DurationInMs":22.3645,"RequestUri":"https://sdkdev-dikshi-southcentralus.documents.azure.com//addresses/?$resolveFor=dbs%2fTFMgAA%3d%3d%2fcolls%2fTFMgAI8vlks%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0","ResourceType":"Document","HttpMethod":"GET","ActivityId":"0c920ce4-1ac8-4a19-860f-6b331e0dcff5","StatusCode":"OK"}]},"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"0c920ce4-1ac8-4a19-860f-6b331e0dcff5","ResponseTimeUtc":"2025-02-10T23:32:04.9263815Z","StatusCode":404,"SubStatusCode":1002,"RequestCharge":0,"RequestUri":"dbs/availabilityStrategyTestDb/colls/sessionRetryPolicy","ErrorMessage":"Microsoft.Azure.Documents.NotFoundException: Message: Fault Injection Server Error: Read Session Not Available, rule: badSessionTokenRule\r\nActivityId: 0c920ce4-1ac8-4a19-860f-6b331e0dcff5, Request URI: /apps/af239937-7bb1-4306-9ac7-2042d643cef2/services/02ca8787-eae1-42ce-87d2-abc37f2608f6/partitions/65f73a29-2825-4147-ae43-580d57dd2c62/replicas/133833555352673277p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.22631 cosmos-netstandard-sdk/3.18.0\r\n at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\StoreResult.cs:line 453\r\n at Microsoft.Azure.Documents.ConsistencyWriter.WritePrivateAsync(DocumentServiceRequest request, TimeoutHelper timeout, Boolean forceRefresh) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\ConsistencyWriter.cs:line 303\r\n at Microsoft.Azure.Documents.BackoffRetryUtility
1.ExecuteRetryAsync[TParam,TPolicy](Func1 callbackMethod, Func3 callbackMethodWithParam, Func2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy1 retryPolicyWithArg, Func1 inBackoffAlternateCallbackMethod, Func2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\BackoffRetryUtility.cs:line 171\r\n at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\IRetryPolicy.cs:line 40\r\n at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync[TParam,TPolicy](Func1 callbackMethod, Func3 callbackMethodWithParam, Func2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy1 retryPolicyWithArg, Func1 inBackoffAlternateCallbackMethod, Func2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\BackoffRetryUtility.cs:line 202\r\n at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync[TParam,TPolicy](Func1 callbackMethod, Func3 callbackMethodWithParam, Func2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy1 retryPolicyWithArg, Func1 inBackoffAlternateCallbackMethod, Func2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\BackoffRetryUtility.cs:line 246\r\n at Microsoft.Azure.Documents.ConsistencyWriter.WriteAsync(DocumentServiceRequest entity, TimeoutHelper timeout, Boolean forceRefresh, CancellationToken cancellationToken) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\ConsistencyWriter.cs:line 135\r\n at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<b__0>d.MoveNext() in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\ReplicatedResourceClient.cs:line 158\r\n--- End of stack trace from previous location ---\r\n at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func1 executeAsync, Func1 prepareRequest, IRequestRetryPolicy2 policy, CancellationToken cancellationToken, Func1 inBackoffAlternateCallbackMethod, Nullable1 minBackoffForInBackoffCallback) in C:\\Users\\dikshibahl\\source\\repos\\azure-cosmos-dotnet-v3\\Microsoft.Azure.Cosmos\\src\\direct\\RequestRetryUtility.cs:line 79\r\n at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) in C:\\Users\\dikshibahl\\source\\repos\\azure-cosmos-dotnet-v3\\Microsoft.Azure.Cosmos\\src\\direct\\IRetryPolicy.cs:line 40\r\n at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func1 executeAsync, Func1 prepareRequest, IRequestRetryPolicy2 policy, CancellationToken cancellationToken, Func1 inBackoffAlternateCallbackMethod, Nullable1 minBackoffForInBackoffCallback) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\RequestRetryUtility.cs:line 102\r\n at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func1 executeAsync, Func1 prepareRequest, IRequestRetryPolicy2 policy, CancellationToken cancellationToken, Func1 inBackoffAlternateCallbackMethod, Nullable`1 minBackoffForInBackoffCallback) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\RequestRetryUtility.cs:line 164\r\n at Microsoft.Azure.Documents.StoreClient.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken, IRetryPolicy retryPolicy) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\direct\StoreClient.cs:line 135\r\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Handler\TransportHandler.cs:line 122\r\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken) in C:\Users\dikshibahl\source\repos\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Handler\TransportHandler.cs:line 33","RequestSessionToken":null,"ResponseSessionToken":"1:1#1#1=1#1=1","BELatencyInMs":null}}}]}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":1078.5664,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":1078.5604,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":1078.4367,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be170.documents.azure.com:14408/apps/ee056cc0-380c-4306-a3a7-9cd9cfd25680/services/da8775e3-b152-438d-bcef-24e0bb407c36/partitions/7a3c58d8-3d15-4841-9156-4db4aff2fbb3/replicas/133734972690132829p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be170.documents.azure.com:14004/apps/ee056cc0-380c-4306-a3a7-9cd9cfd25680/services/da8775e3-b152-438d-bcef-24e0bb407c36/partitions/7a3c58d8-3d15-4841-9156-4db4aff2fbb3/replicas/133734972036685089s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be170.documents.azure.com:14345/apps/ee056cc0-380c-4306-a3a7-9cd9cfd25680/services/da8775e3-b152-438d-bcef-24e0bb407c36/partitions/7a3c58d8-3d15-4841-9156-4db4aff2fbb3/replicas/133734972690132830s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-be170.documents.azure.com:14371/apps/ee056cc0-380c-4306-a3a7-9cd9cfd25680/services/da8775e3-b152-438d-bcef-24e0bb407c36/partitions/7a3c58d8-3d15-4841-9156-4db4aff2fbb3/replicas/133763831364082482s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[{"StartTimeUTC":"2025-02-10T23:32:04.9297706Z","EndTimeUTC":"2025-02-10T23:32:05.2787238Z","TargetEndpoint":"https://sdkdev-dikshi-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fTFMgAA%3d%3d%2fcolls%2fTFMgAI8vlks%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0"}],"StoreResponseStatistics":[{"ResponseTimeUTC":"2025-02-10T23:32:06.0001542Z","DurationInMs":720.7465,"ResourceType":"Document","OperationType":"Create","LocationEndpoint":"https://sdkdev-dikshi-eastus.documents.azure.com/","StoreResult":{"ActivityId":"0c920ce4-1ac8-4a19-860f-6b331e0dcff5","StatusCode":"Created","SubStatusCode":"Unknown","LSN":367,"PartitionKeyRangeId":"0","GlobalCommittedLSN":367,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":367,"SessionToken":"19#367#1=618#3=-1","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":2,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-eastus1-be170.documents.azure.com:14408/apps/ee056cc0-380c-4306-a3a7-9cd9cfd25680/services/da8775e3-b152-438d-bcef-24e0bb407c36/partitions/7a3c58d8-3d15-4841-9156-4db4aff2fbb3/replicas/133734972690132829p/","RequestCharge":22.29,"RetryAfterInMs":null,"BELatencyInMs":"4.022","ReplicaHealthStatuses":["(port: 14408 | status: Connected | lkt: 2/10/2025 11:32:06 PM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2025-02-10T23:32:05.2794120Z", "durationInMs": 0.0256},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2025-02-10T23:32:05.2794376Z", "durationInMs": 656.7086},{"event": "Pipelined", "startTimeUtc": "2025-02-10T23:32:05.9361462Z", "durationInMs": 0.9261},{"event": "Transit Time", "startTimeUtc": "2025-02-10T23:32:05.9370723Z", "durationInMs": 62.4402},{"event": "Received", "startTimeUtc": "2025-02-10T23:32:05.9995125Z", "durationInMs": 0.612},{"event": "Completed", "startTimeUtc": "2025-02-10T23:32:06.0001245Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2025-02-10T23:32:05.8787183Z","lastSend":"2025-02-10T23:32:05.8788281Z","lastReceive":"2025-02-10T23:32:05.9356819Z"},"requestSizeInBytes":627,"requestBodySizeInBytes":146,"responseMetadataSizeInBytes":515,"responseBodySizeInBytes":347},"TransportException":null}}],"HttpResponseStats":[{"StartTimeUTC":"2025-02-10T23:32:04.9297968Z","DurationInMs":348.8565,"RequestUri":"https://sdkdev-dikshi-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fTFMgAA%3d%3d%2fcolls%2fTFMgAI8vlks%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0","ResourceType":"Document","HttpMethod":"GET","ActivityId":"0c920ce4-1ac8-4a19-860f-6b331e0dcff5","StatusCode":"OK"}]}}}]}]}]}]}]}]},{"name":"Response Serialization","duration in milliseconds":2.7281}]}Type of change
Closing issues
To automatically close an issue: closes #IssueNumber