GatewayStoreClient: Fixes stream consumption bug in GatewayStoreClient.CreateDocumentClientExceptionAsync

[Copilot]

Problem

During write region failover, Gateway returns a 403 response with application/json content type but invalid JSON content. This triggers a bug in GatewayStoreClient.CreateDocumentClientExceptionAsync where the HTTP response stream is consumed twice:

1. First consumption happens when attempting to deserialize the response as JSON (line 176)
2. When deserialization fails and the catch block is entered, execution continues to the fallback logic (line 195) which tries to read the same stream again
3. This results in an unhandled InvalidOperationException: The stream was already consumed. It cannot be read again.

The original DocumentClientException with proper diagnostics is lost, making debugging difficult.

Root Cause

The issue was introduced when an else clause was removed and an empty catch block was added to the JSON deserialization logic, causing the same stream to be processed twice if deserialization fails.

Solution

This PR implements a minimal fix that:

1. Buffers the HTTP response content once using ReadAsStringAsync() before attempting JSON deserialization
2. Creates a new MemoryStream from the buffered content for the JSON deserialization attempt
3. Reuses the buffered content in the fallback logic instead of trying to read from the response stream again
4. Fixes a typo in the generic type parameter from <e> to <Error>

Changes Made

• Modified CreateDocumentClientExceptionAsync method to buffer content once and reuse it
• Added explanatory comments for the stream consumption fix
• Added test case TestStreamConsumptionBugFixWhenJsonDeserializationFails to verify the fix
• All changes are surgical and preserve existing functionality

Code Example

Before (buggy):

// First read - consumes the stream
Stream contentAsStream = await responseMessage.Content.ReadAsStreamAsync();
Error error = JsonSerializable.LoadFrom<Error>(stream: contentAsStream);
// ... if this fails and throws, execution continues to:

// Second read - fails with "stream already consumed"
contextBuilder.AppendLine(await responseMessage.Content.ReadAsStringAsync());

After (fixed):

// Buffer content once
contentString = await responseMessage.Content.ReadAsStringAsync();
using (MemoryStream contentStream = new MemoryStream(Encoding.UTF8.GetBytes(contentString)))
{
    Error error = JsonSerializable.LoadFrom<Error>(stream: contentStream);
    // ... 
}
// ... if this fails and throws, execution continues to:

// Reuse buffered content - no stream re-reading
contextBuilder.AppendLine(contentString ?? await responseMessage.Content.ReadAsStringAsync());

This ensures that when Gateway returns invalid JSON during failover scenarios, clients get proper DocumentClientException instances with diagnostics instead of unhandled InvalidOperationException errors.

Fixes #5243.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[github-actions]

All good!

[kirankumarkolli]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).