Database failovers aren't considered by retry mechanism

[webyhwh]

Experienced database failover in cloud instance of Dynamics 365 that was an infrastructure transitive issue just for a moment. This looks like a good candidate to be considered in a retry mechanism like other SQL related errors but it was failed immediately. Please see example logged by the client itself below, some sensitive info hidden by '***'.

*No specific settings are provided to the client, so default settings are being used including retry settings.
*Client version is 1.1.32

Current behavior: the request is being executed only once.
Expected behavior: the request is being executed multiple times based on retry settings.

===========================================

fail: Microsoft.PowerPlatform.Dataverse.Client.ServiceClient[2]
************ FaultException1 - Update : UpdateRequest To Dataverse via IOrganizationService |=> Initialization of the database connection or transaction failed. This operation should be retried later. Initialization exception message: Database 'db_***' on server '***' is not currently available. Please retry the connection later. If the problem persists, contact customer support, and provide them the session tracing ID of '***'. Source: System.ServiceModel.Primitives Method: HandleReply DateUTC: 12/06/2024 TimeUTC: 09:24:06 Error: Message: Initialization of the database connection or transaction failed. This operation should be retried later. Initialization exception message: Database 'db_***' on server '***' is not currently available. Please retry the connection later. If the problem persists, contact customer support, and provide them the session tracing ID of '***'. ErrorCode: -2147187375 Trace: Error Details : ApiExceptionSourceKey : Plugin/Microsoft.Crm.ObjectModel.TargetAttributeValidationPlugin ApiSourceActivityKey : ApiExceptionOwnerKey : OwnershipNotFoundInExceptionOwnerSettings ApiOriginalExceptionKey : Microsoft.Crm.CrmException: Initialization of the database connection or transaction failed. This operation should be retried later. Initialization exception message: Database 'db_***' on server '***' is not currently available. Please retry the connection later. If the problem persists, contact customer support, and provide them the session tracing ID of '***'. ---> System.Data.SqlClient.SqlException: Database 'db_***' on server '***' is not currently available. Please retry the connection later. If the problem persists, contact customer support, and provide them the session tracing ID of '***'. at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling, SqlAuthenticationProviderManager sqlAuthProviderManager) at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions) at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions) at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource1 retry) at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at Polly.Policy.<>c__DisplayClass144_0.b__0(Context ctx, CancellationToken ct)
at Polly.AdvancedCircuitBreakerSyntax.<>c__DisplayClass5_1.b__2(Context ctx, CancellationToken ct)
at Polly.CircuitBreaker.CircuitBreakerEngine.Implementation[TResult](Func3 action, Context context, CancellationToken cancellationToken, IEnumerable1 shouldHandleExceptionPredicates, IEnumerable1 shouldHandleResultPredicates, ICircuitController1 breakerController)
at Polly.AdvancedCircuitBreakerSyntax.<>c__DisplayClass5_0.b__1(Action2 action, Context context, CancellationToken cancellationToken) at Polly.Policy.ExecuteInternal(Action2 action, Context context, CancellationToken cancellationToken)
at Polly.Policy.Execute(Action2 action, Context context, CancellationToken cancellationToken) at Microsoft.Crm.Core.DataServices.Connection.SqlConnectivityCircuitBreakerPolicyCacheManager.OpenWithCircuitBreakerPolicy(IDbConnection connection, ILogger providedLogger) at Microsoft.Crm.CrmDbConnection.InternalOpenWithRetry(IDbConnection connection, String descriptionText, IConnectionRetryPolicy retryPolicy) at Microsoft.Crm.CrmDbConnection.GetCreateAndOpenConnectionInternal(IConnectionRetryPolicy retryPolicy) at Microsoft.Crm.CrmDbConnection.<>c__DisplayClass225_0.<GetCreateAndOpenConnection>b__0() at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute(ILogger logger, EventId eventId, ActivityType activityType, Action action, IEnumerable1 additionalCustomProperties)
at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute(ILogger logger, XrmTelemetryActivityType activityType, Action action)
at Microsoft.Crm.CrmDbConnection.OpenInternal(IConnectionRetryPolicy connectionOpenRetryPolicy)
at Microsoft.Crm.LegacySqlDataAccessContextImplementation.InitConnectionAndTransaction()
at Microsoft.Crm.LegacySqlDataAccessContextImplementation.EnsureBeginRequest()
at Microsoft.Crm.LegacySqlDataAccessContextImplementation.get_AbstractConnection()
at Microsoft.Crm.SqlExecutionContext.get_Connection()
at Microsoft.Crm.Extensibility.PipelineTracingService.WriteTrace(Object sender, Exception e, Boolean resetTraceInfo, Type pluginType)
at Microsoft.Crm.Extensibility.PipelineExecutionContext.SignalSuccessfulExecution(Type pluginType)
at Microsoft.Crm.Extensibility.VersionedPluginProxyStepBase.Execute(PipelineExecutionContext context)
--- End of inner exception stack trace ---
at Microsoft.Crm.LegacySqlDataAccessContextImplementation.VerifyDelayedInitializationDidNotFail()
at Microsoft.Crm.LegacySqlDataAccessContextImplementation.get_AbstractConnection()
at Microsoft.Crm.SqlExecutionContext.get_Connection()
at Microsoft.Crm.Extensibility.VersionedPluginProxyStepBase.ProcessException(Exception e, PipelineExecutionContext context)
at Microsoft.Crm.Extensibility.VersionedPluginProxyStepBase.Execute(PipelineExecutionContext context)
at Microsoft.Crm.Extensibility.PipelineInstrumentationHelper.Execute(Boolean instrumentationEnabled, String stopwatchName, ExecuteWithInstrumentation action, PipelineExecutionContext context)
at Microsoft.Crm.Extensibility.Pipeline.<>c__DisplayClass6_0.b__0()
ApiStepKey : fc743e7d-fbea-4695-bdb9-7d78334c8474
ApiDepthKey : 1
ApiActivityIdKey : 2ea3b0f6-898f-4d48-90b4-460e590f1a47
ApiPluginSolutionNameKey : System
ApiStepSolutionNameKey : System
ApiExceptionCategory : SystemFailure
ApiExceptionMessageName : DBConnectionOrTransactionInitializationFailed
ApiExceptionHttpStatusCode : 500

HelpLink Url: Not Provided
Stack Trace: at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
   at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannelProxy.TaskCreator.<>c__DisplayClass1_0.<CreateGenericTask>b__0(IAsyncResult asyncResult)
--- End of stack trace from previous location ---
   at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.Command_ExecuteAsyncImpl(OrganizationRequest req, String errorStringCheck, CancellationToken cancellationToken, Boolean bypassPluginExecution)
======================================================================================================================
Inner Exception Level 1	: 
==OrganizationServiceFault Info=======================================================================================
Error: Message: Exception output:: System.Data.SqlClient.SqlException (0x80131904): Database 'db_***' on server '***' is not currently available.  Please retry the connection later.  If the problem persists, contact customer support, and provide them the session tracing ID of '***'.
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling, SqlAuthenticationProviderManager sqlAuthProviderManager)
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Polly.Policy.<>c__DisplayClass144_0.<Execute>b__0(Context ctx, CancellationToken ct)
   at Polly.AdvancedCircuitBreakerSyntax.<>c__DisplayClass5_1.<AdvancedCircuitBreaker>b__2(Context ctx, CancellationToken ct)
   at Polly.CircuitBreaker.CircuitBreakerEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, IEnumerable`1 shouldHandleExceptionPredicates, IEnumerable`1 shouldHandleResultPredicates, ICircuitController`1 breakerController)
   at Polly.AdvancedCircuitBreakerSyntax.<>c__DisplayClass5_0.<AdvancedCircuitBreaker>b__1(Action`2 action, Context context, CancellationToken cancellationToken)
   at Polly.Policy.ExecuteInternal(Action`2 action, Context context, CancellationToken cancellationToken)
   at Polly.Policy.Execute(Action`2 action, Context context, CancellationToken cancellationToken)
   at Microsoft.Crm.Core.DataServices.Connection.SqlConnectivityCircuitBreakerPolicyCacheManager.OpenWithCircuitBreakerPolicy(IDbConnection connection, ILogger providedLogger)
   at Microsoft.Crm.CrmDbConnection.InternalOpenWithRetry(IDbConnection connection, String descriptionText, IConnectionRetryPolicy retryPolicy)
   at Microsoft.Crm.CrmDbConnection.GetCreateAndOpenConnectionInternal(IConnectionRetryPolicy retryPolicy)
   at Microsoft.Crm.CrmDbConnection.<>c__DisplayClass225_0.<GetCreateAndOpenConnection>b__0()
   at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute(ILogger logger, EventId eventId, ActivityType activityType, Action action, IEnumerable`1 additionalCustomProperties)
   at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute(ILogger logger, XrmTelemetryActivityType activityType, Action action)
   at Microsoft.Crm.CrmDbConnection.OpenInternal(IConnectionRetryPolicy connectionOpenRetryPolicy)
   at Microsoft.Crm.LegacySqlDataAccessContextImplementation.InitConnectionAndTransaction()
   at Microsoft.Crm.LegacySqlDataAccessContextImplementation.EnsureBeginRequest()
   at Microsoft.Crm.LegacySqlDataAccessContextImplementation.get_AbstractConnection()
   at Microsoft.Crm.SqlExecutionContext.get_Connection()
   at Microsoft.Crm.Extensibility.PipelineTracingService.WriteTrace(Object sender, Exception e, Boolean resetTraceInfo, Type pluginType)
   at Microsoft.Crm.Extensibility.PipelineExecutionContext.SignalSuccessfulExecution(Type pluginType)
   at Microsoft.Crm.Extensibility.VersionedPluginProxyStepBase.Execute(PipelineExecutionContext context)
ClientConnectionId:***
Error Number:40613,State:1,Class:14
ClientConnectionId before routing:***
Routing Destination:***
Error(s) details: 
Class: 0xE
LineNumber: 65536
Message: Database 'db_***' on server '***' is not currently available.  Please retry the connection later.  If the problem persists, contact customer support, and provide them the session tracing ID of '***'.
Number: 40613
Procedure: 
Server: ***
Source: .Net SqlClient Data Provider
State: 1
ToString(): System.Data.SqlClient.SqlError: Database 'db_***' on server '***' is not currently available.  Please retry the connection later.  If the problem persists, contact customer support, and provide them the session tracing ID of '***'.
--
: Microsoft Dynamics CRM has experienced an error. Reference number for administrators or support: ***
ErrorCode: -2147220970
Trace: 
Time: 12/06/2024 09:24:06
ErrorCode: -2147220970
DateUTC: 12/06/2024
TimeUTC: 09:24:06
HelpLink Url: Not Provided
Trace: Not Provided
======================================================================================================================

System.ServiceModel.FaultException`1[Microsoft.Xrm.Sdk.OrganizationServiceFault]: Initialization of the database connection or transaction failed. This operation should be retried later. Initialization exception message: Database 'db_***' on server '***' is not currently available.  Please retry the connection later.  If the problem persists, contact customer support, and provide them the session tracing ID of '***'. (Fault Detail is equal to Exception details: 
ErrorCode: 0x80048551
Message: Initialization of the database connection or transaction failed. This operation should be retried later. Initialization exception message: Database 'db_***' on server '***' is not currently available.  Please retry the connection later.  If the problem persists, contact customer support, and provide them the session tracing ID of '***'.
TimeStamp: 2024-12-06T09:24:06.3085349Z
--
Exception details: 
ErrorCode: 0x80040216
Message: Exception output:: System.Data.SqlClient.SqlException (0x80131904): Database 'db_***' on server '***' is not currently available.  Please retry the connection later.  If the problem persists, contact customer support, and provide them the session tracing ID of '***'.
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling, SqlAuthenticationProviderManager sqlAuthProviderManager)
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Polly.Policy.<>c__DisplayClass144_0.<Execute>b__0(Context ctx, CancellationToken ct)
   at Polly.AdvancedCircuitBreakerSyntax.<>c__DisplayClass5_1.<AdvancedCircuitBreaker>b__2(Context ctx, CancellationToken ct)
   at Polly.CircuitBreaker.CircuitBreakerEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, IEnumerable`1 shouldHandleExceptionPredicates, IEnumerable`1 shouldHandleResultPredicates, ICircuitController`1 breakerController)
   at Polly.AdvancedCircuitBreakerSyntax.<>c__DisplayClass5_0.<AdvancedCircuitBreaker>b__1(Action`2 action, Context context, CancellationToken cancellationToken)
   at Polly.Policy.ExecuteInternal(Action`2 action, Context context, CancellationToken cancellationToken)
   at Polly.Policy.Execute(Action`2 action, Context context, CancellationToken cancellationToken)
   at Microsoft.Crm.Core.DataServices.Connection.SqlConnectivityCircuitBreakerPolicyCacheManager.OpenWithCircuitBreakerPolicy(IDbConnection connection, ILogger providedLogger)
   at Microsoft.Crm.CrmDbConnection.InternalOpenWithRetry(IDbConnection connection, String descriptionText, IConnectionRetryPolicy retryPolicy)
   at Microsoft.Crm.CrmDbConnection.GetCreateAndOpenConnectionInternal(IConnectionRetryPolicy retryPolicy)
   at Microsoft.Crm.CrmDbConnection.<>c__DisplayClass225_0.<GetCreateAndOpenConnection>b__0()
   at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute(ILogger logger, EventId eventId, ActivityType activityType, Action action, IEnumerable`1 additionalCustomProperties)
   at Microsoft.Xrm.Telemetry.XrmTelemetryExtensions.Execute(ILogger logger, XrmTelemetryActivityType activityType, Action action)
   at Microsoft.Crm.CrmDbConnection.OpenInternal(IConnectionRetryPolicy connectionOpenRetryPolicy)
   at Microsoft.Crm.LegacySqlDataAccessContextImplementation.InitConnectionAndTransaction()
   at Microsoft.Crm.LegacySqlDataAccessContextImplementation.EnsureBeginRequest()
   at Microsoft.Crm.LegacySqlDataAccessContextImplementation.get_AbstractConnection()
   at Microsoft.Crm.SqlExecutionContext.get_Connection()
   at Microsoft.Crm.Extensibility.PipelineTracingService.WriteTrace(Object sender, Exception e, Boolean resetTraceInfo, Type pluginType)
   at Microsoft.Crm.Extensibility.PipelineExecutionContext.SignalSuccessfulExecution(Type pluginType)
   at Microsoft.Crm.Extensibility.VersionedPluginProxyStepBase.Execute(PipelineExecutionContext context)
ClientConnectionId:***
Error Number:40613,State:1,Class:14
ClientConnectionId before routing:***
Routing Destination:***
Error(s) details: 
Class: 0xE
LineNumber: 65536
Message: Database 'db_***' on server '***' is not currently available.  Please retry the connection later.  If the problem persists, contact customer support, and provide them the session tracing ID of '***'.
Number: 40613
Procedure: 
Server: ***
Source: .Net SqlClient Data Provider
State: 1
ToString(): System.Data.SqlClient.SqlError: Database 'db_***' on server '***' is not currently available.  Please retry the connection later.  If the problem persists, contact customer support, and provide them the session tracing ID of '***'.
--
: Microsoft Dynamics CRM has experienced an error. Reference number for administrators or support: ***
TimeStamp: 2024-12-06T09:24:06.3085349Z
--
).

[MattB-msft]

Thanks for reporting this..
This looks like a change server side that changed some of the error codes returned by the platform.
the code here: https://github.com/microsoft/PowerPlatform-DataverseServiceClient/blob/42bcd7ea76d18a6d1e5e8e612a260f75e746b144/src/GeneralTools/DataverseClient/Client/ServiceClient.cs#L2031C52-L2031C78 and specifically this error code: ex.HResult == -2146233087 is supposed to catch this case.

It looks like that changed to 2147187375 and 0x80048551

will get that sorted in a future release and discuss with the dataverse team.