Unobserved ArgumentException during background address-cache refresh with PPAF enabled [Starting in 3.62.0]

> We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the [latest SDK version](https://www.nuget.org/packages/Microsoft.Azure.Cosmos).

**Describe the bug**

After upgrading `Microsoft.Azure.Cosmos` from **3.61.0 to 3.62.0**, we began intermittently receiving unobserved task exceptions from the SDK's background address-cache refresh path.

`ClientSideRequestStatisticsTraceDatum.RecordAddressResolutionEnd` throws because the address-resolution identifier is not present:

```text
System.ArgumentException: Identifier {0} does not exist. Please call start before calling end.
```

We did not observe this exception while running SDK 3.61.0. Multiple occurrences with different generated identifiers were captured after deploying 3.62.0.

**To Reproduce**

The exception is intermittent, and we do not currently have a deterministic standalone reproduction.

1. Run a workload against a single-write-region Cosmos DB account with PPAF enabled using `Microsoft.Azure.Cosmos` 3.61.0.
2. Upgrade the application to `Microsoft.Azure.Cosmos` 3.62.0 without otherwise changing its Cosmos DB configuration.
3. Run sustained production traffic that exercises partition-address cache population and background refresh.
4. Subscribe to `TaskScheduler.UnobservedTaskException`.
5. Observe occasional `ArgumentException` instances from:

   ```text
   ClientSideRequestStatisticsTraceDatum.RecordAddressResolutionEnd
   GatewayAddressCache.GetServerAddressesViaGatewayAsync
   AsyncCacheNonBlocking.CreateAndWaitForBackgroundRefreshTaskAsync
   ```

The application uses Direct connection mode and cross-region SDK hedging. The affected Cosmos DB account has **Per-Partition Automatic Failover (PPAF) enabled**.

**Expected behavior**

Background partition-address cache refreshes should complete or fail without producing an unobserved task exception.

Every call to `RecordAddressResolutionEnd` should use the same request-statistics context in which the corresponding `RecordAddressResolutionStart` registered its identifier. A diagnostics inconsistency should not fault the SDK cache-refresh operation or surface through `TaskScheduler.UnobservedTaskException`.

**Actual behavior**

The SDK calls `RecordAddressResolutionEnd` with an identifier that does not exist in the current `ClientSideRequestStatisticsTraceDatum`. This throws `ArgumentException` from a background `AsyncCacheNonBlocking` refresh task.

The task exception is not observed until task finalization, at which point it is reported through `TaskScheduler.UnobservedTaskException`.

At least two occurrences were captured together with different generated identifiers, indicating a recurring SDK state-management issue rather than one malformed application value.

The foreground request impact is currently unknown. The address cache may continue serving its stale value while the refresh task fails, but the unobserved exception triggers critical application monitoring.

**Environment summary**

SDK Version: `Microsoft.Azure.Cosmos` 3.62.0; previously 3.61.0  
OS Version (e.g. Windows, Linux, MacOSX): Linux  
Connection Mode: Direct  
Write configuration: Single write region  
Cross-region SDK hedging: Enabled  
Per-Partition Automatic Failover (PPAF): Enabled

**Additional context**

The exception started only after the upgrade from 3.61.0 to 3.62.0.

The affected account has PPAF enabled. This is relevant because the 3.62.0 GA package enables its new metadata-hedging behavior based on the account's PPAF state.

### Complete captured stack traces

#### Occurrence 1

```text
[OnUnhandledException,UnhandledExceptionReporter.cs(34) <NoCorrelationId> ] An unhandled exception has occurred. The process may be in an inconsistent state.
Context: TaskScheduler_UnobservedTaskException CorrelationId:
ExpInfo(fullInfo->System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result,
the unobserved exception was rethrown by the finalizer thread. (Identifier {0} does not exist. Please call start before calling end. (Parameter 'c9f8567a-1196-496f-bab7-d37fd009d6dc'))
 ---> System.ArgumentException: Identifier {0} does not exist. Please call start before calling end. (Parameter 'c9f8567a-1196-496f-bab7-d37fd009d6dc')
   at Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum.RecordAddressResolutionEnd(String identifier)
   at Microsoft.Azure.Cosmos.Routing.GatewayAddressCache.GetServerAddressesViaGatewayAsync(
       DocumentServiceRequest request,
       String collectionRid,
       IEnumerable`1 partitionKeyRangeIds,
       Boolean forceRefresh)
   at Microsoft.Azure.Cosmos.Routing.GatewayAddressCache.GetAddressesForRangeIdAsync(
       DocumentServiceRequest request,
       PartitionAddressInformation cachedAddresses,
       String collectionRid,
       String partitionKeyRangeId,
       Boolean forceRefresh)
   at Microsoft.Azure.Cosmos.AsyncCacheNonBlocking`2.AsyncLazyWithRefreshTask`1.CreateAndWaitForBackgroundRefreshTaskAsync(
       Func`2 createRefreshTask)
   at Microsoft.Azure.Cosmos.AsyncCacheNonBlocking`2.UpdateCacheAndGetValueFromBackgroundTaskAsync(
       TKey key,
       AsyncLazyWithRefreshTask`1 initialValue,
       Func`2 callbackDelegate,
       String operationName)
   at Microsoft.Azure.Cosmos.AsyncCacheNonBlocking`2.GetAsync(
       TKey key,
       Func`2 singleValueInitFunc,
       Func`2 forceRefresh)
   --- End of inner exception stack trace ---
```

#### Occurrence 2

```text
[OnUnhandledException,UnhandledExceptionReporter.cs(34) <NoCorrelationId> ] An unhandled exception has occurred. The process may be in an inconsistent state.
Context: TaskScheduler_UnobservedTaskException CorrelationId:
ExpInfo(fullInfo->System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result,
the unobserved exception was rethrown by the finalizer thread. (Identifier {0} does not exist. Please call start before calling end. (Parameter '415e1e85-a5c8-4625-b756-a4704aa3e5c8'))
 ---> System.ArgumentException: Identifier {0} does not exist. Please call start before calling end. (Parameter '415e1e85-a5c8-4625-b756-a4704aa3e5c8')
   at Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum.RecordAddressResolutionEnd(String identifier)
   at Microsoft.Azure.Cosmos.Routing.GatewayAddressCache.GetServerAddressesViaGatewayAsync(
       DocumentServiceRequest request,
       String collectionRid,
       IEnumerable`1 partitionKeyRangeIds,
       Boolean forceRefresh)
   at Microsoft.Azure.Cosmos.Routing.GatewayAddressCache.GetAddressesForRangeIdAsync(
       DocumentServiceRequest request,
       PartitionAddressInformation cachedAddresses,
       String collectionRid,
       String partitionKeyRangeId,
       Boolean forceRefresh)
   at Microsoft.Azure.Cosmos.AsyncCacheNonBlocking`2.AsyncLazyWithRefreshTask`1.CreateAndWaitForBackgroundRefreshTaskAsync(
       Func`2 createRefreshTask)
   at Microsoft.Azure.Cosmos.AsyncCacheNonBlocking`2.UpdateCacheAndGetValueFromBackgroundTaskAsync(
       TKey key,
       AsyncLazyWithRefreshTask`1 initialValue,
       Func`2 callbackDelegate,
       String operationName)
   at Microsoft.Azure.Cosmos.AsyncCacheNonBlocking`2.AsyncLazyWithRefreshTask`1.CreateAndWaitForBackgroundRefreshTaskAsync(
       Func`2 createRefreshTask)
   at Microsoft.Azure.Cosmos.AsyncCacheNonBlocking`2.UpdateCacheAndGetValueFromBackgroundTaskAsync(
       TKey key,
       AsyncLazyWithRefreshTask`1 initialValue,
       Func`2 callbackDelegate,
       String operationName)
   at Microsoft.Azure.Cosmos.AsyncCacheNonBlocking`2.GetAsync(
       TKey key,
       Func`2 singleValueInitFunc,
       Func`2 forceRefresh)
   at Microsoft.Azure.Cosmos.AsyncCacheNonBlocking`2.GetAsync(
       TKey key,
       Func`2 singleValueInitFunc,
       Func`2 forceRefresh)
   --- End of inner exception stack trace ---
```

### Potential 3.62.0 regression areas

The following 3.62.0 changes appear relevant to the affected routing and cache paths:

- [PR #5999](https://github.com/Azure/azure-cosmos-dotnet-v3/pull/5999): adds concurrent cross-region metadata hedging for Collection and PartitionKeyRange metadata reads, including cache refreshes and cloned requests.
- [PR #5549](https://github.com/Azure/azure-cosmos-dotnet-v3/pull/5549): changes `GatewayAddressCache.GetServerAddressesViaGatewayAsync` to support an AAD token-revocation retry path.
- [PR #5970](https://github.com/Azure/azure-cosmos-dotnet-v3/pull/5970): enables thin-client mode by default when compatible endpoints are advertised. This appears less directly related because the captured stack uses `GatewayAddressCache`.

Our current hypothesis is that a 3.62.0 routing, cloning, retry, or hedging path causes `RecordAddressResolutionStart` and `RecordAddressResolutionEnd` to operate against different `ClientSideRequestStatisticsTraceDatum` instances. This has not yet been confirmed.

It may also be useful to verify that every nested background refresh task created by `AsyncCacheNonBlocking` has its exception observed.

### Related issues

The exception signature has appeared in earlier SDK versions, but this report concerns a regression specifically observed after moving from 3.61.0 to 3.62.0:

- [#5074: Unhandled ArgumentException](https://github.com/Azure/azure-cosmos-dotnet-v3/issues/5074)
- [#4164: Unobserved Exception thrown while running cache refresh task](https://github.com/Azure/azure-cosmos-dotnet-v3/issues/4164)
- [#4010: Unobserved task exception on gateway failure](https://github.com/Azure/azure-cosmos-dotnet-v3/issues/4010)


Navigation Menu

Uh oh!

Unobserved ArgumentException during background address-cache refresh with PPAF enabled [Starting in 3.62.0] #6067

Complete captured stack traces

Occurrence 1

Occurrence 2

Potential 3.62.0 regression areas

Related issues

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

Unobserved ArgumentException during background address-cache refresh with PPAF enabled [Starting in 3.62.0] #6067

Description

Complete captured stack traces

Occurrence 1

Occurrence 2

Potential 3.62.0 regression areas

Related issues

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions