Per Partition Automatic Failover: Adds Hub Region Processing Only While Routing Requests Failed with 404/1002. #5447
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
aavasthy
requested review from
FabianMeiswinkel,
Pilchie,
adityasa,
khdang,
kirankumarkolli,
kirillg,
neildsh and
sboshra
as code owners
aavasthy
changed the title
ananth7592
left a comment
ananth7592
left a comment
There was a problem hiding this comment.
Approved with a comment
| } | ||
|
|
||
| if (statusCode == HttpStatusCode.NotFound | ||
| && subStatusCode == SubStatusCodes.ReadSessionNotAvailable) |
There was a problem hiding this comment.
also double check: does this change also targeted for MM as well? For MM, writes can happen in any region, also enable this for MM might cause regression
There was a problem hiding this comment.
confirmed this with backend team and this change is not intended to be used in multi-master.
| if (this.addHubRegionProcessingOnlyHeader) | ||
| { | ||
| request.Headers[HubRegionHeader] = bool.TrueString; | ||
| this.addHubRegionProcessingOnlyHeader = false; // reset after applying |
There was a problem hiding this comment.
what would be the errors returned if SDK try to read from non-hub region?
There was a problem hiding this comment.
Also falling back to new hub for that partition.
There was a problem hiding this comment.
SDK ends up getting 403/3(WriteForbidden) if it tries to read from a non-hub region. Then SDK has internal logic where every 403/3 response leads to new endpoint discovery. The SDK then retries the request to the hub region.
kirankumarkolli
left a comment
kirankumarkolli
left a comment
There was a problem hiding this comment.
Waiting for design document
| private const int MaxRetryCount = 120; | ||
| private const int MaxServiceUnavailableRetryCount = 1; | ||
| private const int MaxServiceUnavailableRetryCount = 1; | ||
| private const string HubRegionHeader = "x-ms-cosmos-hub-region-processing-only"; |
There was a problem hiding this comment.
Do we have any central place to host these types of headers in the code base?
There was a problem hiding this comment.
yes we are actually using the central file HttpConstants.cs. I removed this line as part of code clean up.
|
aavasthy
force-pushed
the
users/aavasthy/404_1002
branch
from
f989f93 to
d51b105
Compare
Pull Request Template
Description
During partition-level failover and failback under session consistency, a timing gap can cause read requests to fail with 404/1002 errors. When a partition temporarily fails over to a secondary region and later begins failing back to the primary region, the SDK’s read circuit breaker (PPCB) may start routing reads back to the primary region before it has fully caught up with the writes from the failover region. As a result, reads using session tokens from the previous write region may fail because the primary region does not yet have the corresponding session state. Since the SDK currently does not perform cross-regional retries for 404/1002 responses, these reads continue to fail until the primary region is fully synchronized. The goal is to leverage the new backend header x-ms-cosmos-hub-region-processing-only to detect such conditions and route retry requests to the correct write (hub) region, ensuring successful session-consistent reads during the failback window.
Type of change
Please delete options that are not relevant.
Closing issues
To automatically close an issue: closes #5440