[Per Partition Automatic Failover] Enable PPAF Dynamically Using Targeted Event-Based Updates with Thread-Safe Operations by Copilot · Pull Request #5326 · Azure/azure-cosmos-dotnet-v3

Copilot · 2025-07-30T16:22:38Z

This PR implements dynamic Per Partition Automatic Failover (PPAF) refresh capability that enables the Azure Cosmos DB .NET SDK to update PPAF settings at runtime without requiring SDK restart when account properties change.

Problem

Currently, the SDK fetches the EnablePartitionLevelFailover flag only during initialization via the GET ACCOUNT metadata call. If the account updates this flag after SDK initialization, customers need to restart the SDK to take the change into effect, creating operational challenges.

Solution

Leveraging Existing Background Refresh Infrastructure

Instead of creating a separate background task, this implementation enhances the existing GlobalEndpointManager.StartLocationBackgroundRefreshLoop() which already refreshes account properties every 5 minutes. This approach:

Avoids duplicate background tasks - Uses the well-tested existing refresh mechanism
Reduces resource consumption - Single centralized background refresh task
Maintains consistency - All account property updates flow through the same pathway

Targeted Event-Based Architecture

GlobalEndpointManager Enhancement:

Added OnEnablePartitionLevelFailoverConfigChanged event that fires only when PPAF status changes
Enhanced RefreshDatabaseAccountInternalAsync() to track previous PPAF values and emit targeted events
Eliminates unnecessary event invocations when PPAF settings haven't changed

DocumentClient Direct Integration:

Subscribes directly to GlobalEndpointManager's targeted OnEnablePartitionLevelFailoverConfigChanged events
Simplified HandleEnablePartitionLevelFailoverConfigChanged() method processes only actual PPAF changes
Handles dynamic PPAF configuration updates efficiently when changes are detected

Dynamic PPAF Configuration Updates

When EnablePartitionLevelFailover changes in account properties, the SDK automatically:

Updates Connection Policy: Sets EnablePartitionLevelFailover and EnablePartitionLevelCircuitBreaker flags
Configures Read Hedging: Enables default hedging strategy when PPAF is enabled (respects existing user configurations)
Updates Circuit Breaker: Enables per-partition circuit breaker when PPAF is enabled
Recreates Endpoint Manager: Creates new GlobalPartitionEndpointManagerCore instance with updated settings using thread-safe atomic operations
Updates User Agent Features: Refreshes user agent to reflect new PPAF configuration for proper telemetry

Thread Safety

Atomic Operations: Used Interlocked.Exchange for atomic updates to PartitionKeyRangeLocation to prevent thread contention during dynamic updates. The implementation:

Converts auto-property to field with property accessor for atomic reference updates
Uses Interlocked.Exchange(ref this.partitionKeyRangeLocation, newValue) to atomically swap references
Ensures thread-safe disposal of old instances while preventing race conditions during concurrent request processing

Usage

// No code changes required - everything happens automatically
CosmosClient client = new CosmosClient(connectionString);

// SDK now automatically:
// 1. Uses existing GlobalEndpointManager background refresh (every 5 minutes)
// 2. Detects PPAF enablement changes with targeted event handling
// 3. Updates read hedging and circuit breaker settings only when needed
// 4. Recreates partition endpoint manager with thread-safe atomic operations
// 5. Updates user agent features to reflect dynamic changes

Implementation Details

Efficient Event Handling: Events are only fired when PPAF status actually changes, reducing unnecessary processing
Thread Safety: All operations properly synchronized with Interlocked.Exchange for atomic reference updates, preventing thread contention during dynamic updates
Backward Compatibility: Zero breaking changes, respects client-level override settings
Resource Efficiency: Single background refresh task with targeted event handling
Observability: Comprehensive trace logging for monitoring dynamic updates
User Agent Telemetry: Automatically updates user agent features when PPAF configuration changes for accurate service telemetry

Testing

Updated unit tests to reflect the targeted event architecture:

Direct property update validation with change detection
Proper handling of null values and edge cases
Validation of thread-safe atomic operations

The implementation satisfies all acceptance criteria:

✅ Read hedging automatically enabled/configured when PPAF is enabled
✅ Per-partition circuit breaker enabled when PPAF is enabled
✅ Dynamic updates without SDK restart required
✅ Preserves user-configured availability strategies
✅ Thread-safe with proper resource management using atomic operations
✅ Efficient resource utilization with targeted event architecture
✅ User agent features updated to reflect dynamic PPAF changes for accurate telemetry

Fixes #5304.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

kundadebdatta · 2025-07-30T17:12:20Z

+        /// <summary>
+        /// Initializes and starts the background account properties refresh task
+        /// </summary>
+        private void InitializeBackgroundRefresh()


There is already a background task in GlobalEndpointManager.InitializeAccountPropertiesAndStartBackgroundRefresh.

Let's try to leverage that instead of creating a new one.

You're absolutely right! I've refactored the implementation to leverage the existing background task in GlobalEndpointManager.StartLocationBackgroundRefreshLoop() instead of creating a duplicate one.

Changes made:

Removed separate background refresh task from CosmosAccountServiceConfiguration

Enhanced existing GlobalEndpointManager.RefreshDatabaseAccountInternalAsync() to raise OnAccountPropertiesRefreshed events

Updated CosmosAccountServiceConfiguration to receive updates via UpdateAccountProperties() method

Modified DocumentClient to subscribe to GlobalEndpointManager events for PPAF changes

This approach is much cleaner as it:

✅ Avoids duplicate background tasks

✅ Reduces resource consumption

✅ Leverages the existing, well-tested refresh mechanism

✅ Keeps responsibilities clear

The functionality remains the same - dynamic PPAF updates when account properties change - but now uses a single, centralized background refresh task. (commit: 6941697)

…ground task Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

github-actions

Please follow the required format: "[Internal] Category: (Adds|Fixes|Refactors|Removes) Description"

Internal should be used for PRs that have no customer impact. This flag is used to help generate the changelog to know which PRs should be included. Examples:
Diagnostics: Adds GetElapsedClientLatency to CosmosDiagnostics
PartitionKey: Fixes null reference when using default(PartitionKey)
[v4] Client Encryption: Refactors code to external project
[Internal] Query: Adds code generator for CosmosNumbers for easy additions in the future.

kundadebdatta · 2025-07-30T18:22:55Z

+        /// <summary>
+        /// Event that is raised when account properties are refreshed and PPAF enablement status changes
+        /// </summary>
+        internal event Action<bool?> OnEnablePartitionLevelFailoverChanged;


Do we really need this event anymore ? Can we not leverage the event OnAccountPropertiesRefreshed from GlobalEndpointManager directly ?

You're absolutely right! I've removed the redundant OnEnablePartitionLevelFailoverChanged event and simplified the architecture to leverage the existing OnAccountPropertiesRefreshed event from GlobalEndpointManager directly.

Changes made:

Removed OnEnablePartitionLevelFailoverChanged event from CosmosAccountServiceConfiguration

Enhanced DocumentClient.HandleAccountPropertiesRefreshed() to directly compare previous and new PPAF values and handle changes

Simplified CosmosAccountServiceConfiguration.UpdateAccountProperties() to only update properties without event firing

Updated unit tests to reflect the new direct architecture

This eliminates the intermediate event layer and provides a cleaner, more direct flow:
GlobalEndpointManager.OnAccountPropertiesRefreshed → DocumentClient.HandleAccountPropertiesRefreshed() → direct PPAF change handling

The functionality remains the same but with reduced complexity and fewer event subscriptions. (commit: 395c4bd)

kundadebdatta · 2025-07-30T18:25:59Z

/azp run

azure-pipelines · 2025-07-30T18:26:13Z

Azure Pipelines successfully started running 1 pipeline(s).

…rage GlobalEndpointManager event directly Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

kundadebdatta · 2025-07-30T18:40:03Z

+        /// </summary>
+        /// <param name="defaultValue">An integer containing the default value for the refresh interval in seconds.</param>
+        /// <returns>An integer representing the refresh interval in seconds.</returns>
+        public static int GetAccountPropertiesRefreshIntervalInSeconds(


There is no usage as we got rid of the redundant code for account refresh. Please remove this.

You're correct! I've removed the unused GetAccountPropertiesRefreshIntervalInSeconds method and the associated AccountPropertiesRefreshIntervalInSeconds constant since they're no longer needed after refactoring to use the existing GlobalEndpointManager background task. (commit: 47dcb90)

kundadebdatta · 2025-07-30T18:42:50Z

-
-                GlobalEndpointManager.ParseThinClientLocationsFromAdditionalProperties(accountProperties);
+
+                GlobalEndpointManager.ParseThinClientLocationsFromAdditionalProperties(accountProperties);


Is this really a change ? Please fix the code diff.

Fixed the code diff by reverting the file to original state and reapplying only the necessary changes. Removed the unnecessary whitespace-only change that was appearing in the diff. (commit: 526d0d8)

kundadebdatta · 2025-07-30T18:43:12Z

+        /// A read-only string containing the environment variable name for capturing the account properties refresh interval
+        /// in seconds. The default value for this interval is 300 seconds (5 minutes).
+        /// </summary>
+        internal static readonly string AccountPropertiesRefreshIntervalInSeconds = "AZURE_COSMOS_ACCOUNT_PROPERTIES_REFRESH_INTERVAL_IN_SECONDS";


Remove this as well.

Removed the empty line that was left after removing the AccountPropertiesRefreshIntervalInSeconds constant. (commit: 526d0d8)

… constant Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

kundadebdatta · 2025-07-30T19:18:20Z

+        /// Updates the account properties
+        /// This method is called by GlobalEndpointManager when account properties are refreshed
+        /// </summary>
+        internal void UpdateAccountProperties(AccountProperties newProperties)


This is not needed. Let's get this removed.

Removed the UpdateAccountProperties method and updated the event handler to focus only on PPAF handling without updating accountServiceConfiguration. Also simplified the tests accordingly. (commit: e610a29)

…e only on PPAF changes Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

…erviceConfiguration Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

kundadebdatta · 2025-07-30T21:06:08Z

/azp run

azure-pipelines · 2025-07-30T21:06:23Z

Azure Pipelines successfully started running 1 pipeline(s).

kundadebdatta · 2025-07-30T23:38:07Z

+
+                // Update the GlobalPartitionEndpointManager
+                this.UpdateGlobalPartitionEndpointManager();
+


Please update the user agent features here as well using this.ConnectionPolicy.UserAgentContainer.AppendFeatures

Added the user agent features update using this.ConnectionPolicy.UserAgentContainer.AppendFeatures(this.GetUserAgentFeatures()) when PPAF configuration changes dynamically. This ensures the user agent reflects the current PPAF state after dynamic updates. (commit: 49b3135)

Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

kundadebdatta · 2025-07-31T00:16:54Z

/azp run

azure-pipelines · 2025-07-31T00:17:08Z

Azure Pipelines successfully started running 1 pipeline(s).

kundadebdatta · 2025-07-31T01:28:41Z

+            }
+
+            // Update the partition key range location
+            this.PartitionKeyRangeLocation = newPartitionKeyRangeLocation;


Could it cause any thread contention ? Can we use Interlocked.CompareExchange to alter the object ?

Fixed thread contention issue by using Interlocked.Exchange for atomic updates to PartitionKeyRangeLocation. Changed from auto-property to field with property accessor and used Interlocked.Exchange(ref this.partitionKeyRangeLocation, newPartitionKeyRangeLocation) to atomically update the reference while avoiding race conditions during concurrent access. (commit: ec9fd26)

…dates Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

kundadebdatta · 2025-07-31T19:03:32Z

/azp run

azure-pipelines · 2025-07-31T19:03:46Z

Azure Pipelines successfully started running 1 pipeline(s).

kundadebdatta · 2025-08-13T18:18:00Z

This is no longer needed. The issue was closed with PR-5310.

Initial plan

a7d02e0

Copilot AI assigned Copilot and kundadebdatta Jul 30, 2025

Copilot started work on behalf of kundadebdatta July 30, 2025 16:22 View session

Implement dynamic PPAF refresh mechanism

45ae5f2

Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] [Per Partition Automatic Failover] - Enable PPAF Dynamically upon change on Account Properties Metadata Response~~ [Per Partition Automatic Failover] Enable PPAF Dynamically upon Account Properties Metadata Response Changes Jul 30, 2025

Copilot AI requested a review from kundadebdatta July 30, 2025 16:49

Copilot finished work on behalf of kundadebdatta July 30, 2025 16:49

kundadebdatta reviewed Jul 30, 2025

View reviewed changes

Copilot started work on behalf of kundadebdatta July 30, 2025 17:12 View session

Refactor PPAF refresh to leverage existing GlobalEndpointManager back…

6941697

…ground task Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

Copilot AI changed the title ~~[Per Partition Automatic Failover] Enable PPAF Dynamically upon Account Properties Metadata Response Changes~~ [Per Partition Automatic Failover] Enable PPAF Dynamically Using Existing GlobalEndpointManager Background Refresh Jul 30, 2025

Copilot AI requested a review from kundadebdatta July 30, 2025 17:28

Copilot finished work on behalf of kundadebdatta July 30, 2025 17:28

github-actions Bot reviewed Jul 30, 2025

View reviewed changes

kundadebdatta reviewed Jul 30, 2025

View reviewed changes

Copilot started work on behalf of kundadebdatta July 30, 2025 18:23 View session

Remove redundant OnEnablePartitionLevelFailoverChanged event and leve…

395c4bd

…rage GlobalEndpointManager event directly Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

Copilot AI changed the title ~~[Per Partition Automatic Failover] Enable PPAF Dynamically Using Existing GlobalEndpointManager Background Refresh~~ [Per Partition Automatic Failover] Enable PPAF Dynamically Using GlobalEndpointManager Event Directly Jul 30, 2025

Copilot AI requested a review from kundadebdatta July 30, 2025 18:36

Copilot finished work on behalf of kundadebdatta July 30, 2025 18:36

kundadebdatta reviewed Jul 30, 2025

View reviewed changes

Copilot started work on behalf of kundadebdatta July 30, 2025 18:40 View session

kundadebdatta reviewed Jul 30, 2025

View reviewed changes

Remove unused GetAccountPropertiesRefreshIntervalInSeconds method and…

47dcb90

… constant Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

Copilot finished work on behalf of kundadebdatta July 30, 2025 18:49

Copilot AI requested a review from kundadebdatta July 30, 2025 18:49

Copilot finished work on behalf of kundadebdatta July 30, 2025 19:18

Copilot started work on behalf of kundadebdatta July 30, 2025 19:18 View session

kundadebdatta reviewed Jul 30, 2025

View reviewed changes

Rename event to OnEnablePartitionLevelFailoverConfigChanged and invok…

9f329f8

…e only on PPAF changes Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

Copilot AI changed the title ~~[Per Partition Automatic Failover] Enable PPAF Dynamically Using GlobalEndpointManager Event Directly~~ [Per Partition Automatic Failover] Enable PPAF Dynamically Using Targeted Event-Based Updates Jul 30, 2025

Copilot AI requested a review from kundadebdatta July 30, 2025 19:33

Copilot finished work on behalf of kundadebdatta July 30, 2025 19:33

Copilot started work on behalf of kundadebdatta July 30, 2025 19:33 View session

Remove unnecessary UpdateAccountProperties method from CosmosAccountS…

e610a29

…erviceConfiguration Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

Copilot finished work on behalf of kundadebdatta July 30, 2025 19:47

kundadebdatta reviewed Jul 30, 2025

View reviewed changes

Copilot started work on behalf of kundadebdatta July 30, 2025 23:38 View session

Update user agent features when PPAF configuration changes dynamically

49b3135

Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

Copilot AI requested a review from kundadebdatta July 30, 2025 23:47

Copilot finished work on behalf of kundadebdatta July 30, 2025 23:47

kundadebdatta reviewed Jul 31, 2025

View reviewed changes

Copilot started work on behalf of kundadebdatta July 31, 2025 01:28 View session

Use Interlocked.Exchange for thread-safe PartitionKeyRangeLocation up…

ec9fd26

…dates Co-authored-by: kundadebdatta <87335885+kundadebdatta@users.noreply.github.com>

Copilot AI changed the title ~~[Per Partition Automatic Failover] Enable PPAF Dynamically Using Targeted Event-Based Updates~~ [Per Partition Automatic Failover] Enable PPAF Dynamically Using Targeted Event-Based Updates with Thread-Safe Operations Jul 31, 2025

Copilot AI requested a review from kundadebdatta July 31, 2025 01:39

Copilot finished work on behalf of kundadebdatta July 31, 2025 01:39

Code changes to fix store model behaviors.

2cd6627

kundadebdatta closed this Aug 13, 2025


		GlobalEndpointManager.ParseThinClientLocationsFromAdditionalProperties(accountProperties);

		GlobalEndpointManager.ParseThinClientLocationsFromAdditionalProperties(accountProperties);


		// Update the GlobalPartitionEndpointManager
		this.UpdateGlobalPartitionEndpointManager();

Conversation

Copilot AI commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Leveraging Existing Background Refresh Infrastructure

Targeted Event-Based Architecture

Dynamic PPAF Configuration Updates

Thread Safety

Usage

Implementation Details

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

kundadebdatta commented Jul 30, 2025

Uh oh!

azure-pipelines Bot commented Jul 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

kundadebdatta commented Jul 30, 2025

Uh oh!

azure-pipelines Bot commented Jul 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

kundadebdatta commented Jul 31, 2025

Uh oh!

azure-pipelines Bot commented Jul 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

kundadebdatta commented Jul 31, 2025

Uh oh!

azure-pipelines Bot commented Jul 31, 2025

Uh oh!

kundadebdatta commented Aug 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Copilot AI commented Jul 30, 2025 •

edited

Loading