[BUG]Job in AzureML Workspace can't authenticate with Azure.Identity 1.13.*

[lenniehwtw-new]

Library name and version

Azure.Identity 1.13.*

Describe the bug

We have a .NET pipleine job in AzureMl workspace that works just fine with Azure.Identity 1.12.1
But if we upgrade to 1.3.* the job fails to authenticate.
We were using DefaultAzureCredential() and having read the breaking changes for 1.13.0 I changed this to use a chained credential like this

var credentials = new ChainedTokenCredential(new EnvironmentCredential(), new ManagedIdentityCredential(azureClientId), new AzureCliCredential());

This works fine with 1.12.1, in this particular context we would expect it to pick up the ManagedIdentityCredential, the method is called in other contexts too, which is why the EnvironmentCredential is in the chain.

The error we see in the logs in AzureML looks like this:

Unhandled exception. Azure.Identity.AuthenticationFailedException: ManagedIdentityCredential authentication failed: [Managed Identity] The error response was either empty or could not be parsed.. Error response received from the server: Invalid secret token header: .
See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot
---> MSAL.NetCore.4.66.1.0.MsalServiceException:
ErrorCode: managed_identity_request_failed
Microsoft.Identity.Client.MsalServiceException: [Managed Identity] The error response was either empty or could not be parsed.. Error response received from the server: Invalid secret token header: .
at Microsoft.Identity.Client.ManagedIdentity.AbstractManagedIdentity.HandleResponseAsync(AcquireTokenForManagedIdentityParameters parameters, HttpResponse response, CancellationToken cancellationToken)
at Microsoft.Identity.Client.ManagedIdentity.AbstractManagedIdentity.AuthenticateAsync(AcquireTokenForManagedIdentityParameters parameters, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.ManagedIdentityAuthRequest.SendTokenRequestForManagedIdentityAsync(ILoggerAdapter logger, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.ManagedIdentityAuthRequest.GetAccessTokenAsync(CancellationToken cancellationToken, ILoggerAdapter logger)
at Microsoft.Identity.Client.Internal.Requests.ManagedIdentityAuthRequest.ExecuteAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.RequestBase.<>c__DisplayClass11_1.<b__1>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.Identity.Client.Utils.StopwatchService.MeasureCodeBlockAsync(Func1 codeBlock) at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken) at Microsoft.Identity.Client.ApiConfig.Executors.ManagedIdentityExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenForManagedIdentityParameters managedIdentityParameters, CancellationToken cancellationToken) at Azure.Identity.MsalManagedIdentityClient.AcquireTokenForManagedIdentityAsyncCore(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask1 task)
at Azure.Identity.MsalManagedIdentityClient.AcquireTokenForManagedIdentity(TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentityClient.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
StatusCode: 403
ResponseBody:
Headers:
--- End of inner exception stack trace ---
at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask1 task) at Azure.Identity.ManagedIdentityCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken) at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable) at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask1 task)
at Azure.Identity.DefaultAzureCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.SetResultOnTcsFromCredentialAsync(TokenRequestContext context, TaskCompletionSource1 targetTcs, Boolean async, CancellationToken cancellationToken) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetAuthHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](Task1 task)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.TokenRequestState.GetCurrentHeaderValue(Boolean async, Boolean checkForCompletion, CancellationToken cancellationToken)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetAuthHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask1 task) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequest(HttpMessage message, TokenRequestContext context) at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.AuthorizeRequestOnChallengeAsyncInternal(HttpMessage message, Boolean async) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask1 task)
at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.AuthorizeRequestOnChallenge(HttpMessage message)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
at Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken) at Azure.Core.Pipeline.HttpPipeline.SendRequest(Request request, CancellationToken cancellationToken) at Azure.Security.KeyVault.KeyVaultPipeline.SendRequest(Request request, CancellationToken cancellationToken) at Azure.Security.KeyVault.KeyVaultPipeline.GetPage[T](Uri firstPageUri, String nextLink, Func1 itemFactory, String operationName, CancellationToken cancellationToken)
at Azure.Security.KeyVault.Secrets.SecretClient.<>c__DisplayClass15_0.b__0(String nextLink)
at Azure.Core.PageResponseEnumerator.<>c__DisplayClass0_01.<CreateEnumerable>b__0(String continuationToken, Nullable1 pageSizeHint)
at Azure.Core.PageResponseEnumerator.FuncPageable1.AsPages(String continuationToken, Nullable1 pageSizeHint)+MoveNext()
at Azure.Pageable1.GetEnumerator()+MoveNext() at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.Load() at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList1 providers)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at Microsoft.Extensions.Hosting.HostBuilder.InitializeAppConfiguration()
at Microsoft.Extensions.Hosting.HostBuilder.Build()
at Program.

$(String[] args) in /build/src/Wtw.Model.Deployment.Worker/Program.cs:line 4
at Program.(String[] args)

Expected behavior

We would expect the code that works with Azure.Identity 1.12.1 to still work

Actual behavior

see above in Bug description

Reproduction Steps

Run a .NET executable as a pipeline job in AzureML workspace

Environment

The job is a pipeline job that runs in AzureML workspace and is responsible for deploying endpoints

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[christothes]

Hi @lenniehwtw-new
Would you mind providing the logging output (with any secrets redacted) after reproducing this with logging enabled?

[github-actions]

Hi @lenniehwtw-new. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[lenniehwtw-new]

My pipeline job is running now, will return shortly with the requested logs.....