AsyncCache: Adds support for stack trace optimization during exceptions for AsyncCache and AsyncCacheNonblocking

[ananth7592]

Pull Request Template

Description

This PR addresses the issue of shared exception objects in the AsyncCache class, which was causing significant performance degradation and memory bloat recently for open AI .

The fix involves creating shallow copies of known offenders and rethrowing the rest. This approach ensures that each AsyncCache.GetAsync call receives its own exception instance, preventing the dangerous growth of the exception object's StackTrace and reducing GC overhead.

All ICloneable implemenations of exceptions will benefit from this.

https://learn.microsoft.com/en-us/dotnet/api/system.icloneable?view=net-9.0

Key changes:

• Added support to post-process the exception in above mentioned two cache classes that creates a shallow copy of specific exception types to prevent stack trace proliferation and rethrows them, while propagating all other exceptions unchanged.
• Added support for any exception that is ICloneable supported , TimeoutException and TaskCanceledException in this drop
• Improved exception handling to avoid shared exception objects across threads.

Feature is enabled by default. To turn it off, set EnableAsyncCacheExceptionNoSharing CosmosClientOptions to false

Proof of testing:

Repro Steps
To reproduce the issue, I followed these high-level steps:

• Create a Cosmos client with the necessary settings to preempt gateway calls for address refresh to fail with a specific exception.

• Create a TaskCompletionSource that waits until all the threads are created.

• Initiate calls to createItem that will eventually block, with one of them erroring out with the pre-set exception.

• Observe the exception storm as the same exception trace is copied over to all the threads.

After the fix

• Tested with TimeoutException (2K Threads) and TaskCanceledException (10K threads) using TaskCompletionSource with up to 10,000 threads without issues on the devbox.

• Implemented the fix in AsyncNonBlockingCache by creating shallow copies of known offenders and rethrowing the rest.

• Re-ran the setup and observed a significant reduction in the number of stack trace frames.

TaskCancelledExcpetion with 10K threads ( Even the file size is giving away how bad the stack trace profileration problem was )

TaskCancExceptionWithFix.log

Filesize: 10 KB
Standard Output: 
With cache fix and 10K threads 4923736 bytes allocated in (Gen0:0 1:0 2:0) Frames=2

TaskCancExceptionWithoutFix.log

Filesize: 1,168KB
Standard Output: 
Without cache fix and 10K threads 5612888 bytes allocated in (Gen0:0 1:0 2:0) Frames=2

TImeoutException with 2K threads

TimeoutExceptionWithFix.log
TimeoutExceptionWithoutFix.log

Type of change

Please delete options that are not relevant.

• [] Bug fix (non-breaking change which fixes an issue)
• [] New feature (non-breaking change which adds functionality)
• [] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [] This change requires a documentation update

Closing issues

To automatically close an issue: closes #IssueNumber

[NaluTripician]

Is there any unit testing we can do to verify change?

[ananth7592]

Is there any unit testing we can do to verify change?

This is more of a performance optimization and will be materialized as such in high scale scenarios. Hence, I performed a local repro with high concurrency and captured the performance improvements in the log attached.

[Pilchie]

Looks fine to me.