Skip to content

[BUG] spring boot 3.4.5 containers hang at starting up for keyvault property-source with managed identity #45263

Open
@raja-anbazhagan

Description

@raja-anbazhagan

Describe the bug
Containers hang at starting up since upgrading from 3.0.7 to 3.4.5. Container starts fine if property source is disabled.

Exception or Stack Trace

13:03:42.327 [main] INFO com.example.redacted.ReweighJobApplication -- Starting ReweighJobApplication...
13:11:02.127 [main] ERROR org.springframework.boot.SpringApplication -- Application run failed
java.lang.IllegalStateException: Failed to configure KeyVault property source 'azure-key-vault-secret-property-source-0'
        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.buildKeyVaultPropertySource(KeyVaultEnvironmentPostProcessor.java:146)
        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.buildKeyVaultPropertySourceList(KeyVaultEnvironmentPostProcessor.java:130)
        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:93)
        at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:132)
        at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:115)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:138)
        at org.springframework.boot.context.event.EventPublishingRunListener.multicastInitialEvent(EventPublishingRunListener.java:136)
        at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:81)
        at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:64)
        at java.base/java.lang.Iterable.forEach(Unknown Source)
        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118)
        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112)
        at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:63)
        at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:353)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:313)
        at com.example.redacted.ReweighJobApplication.main(ReweighJobApplication.java:18)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:102)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:64)
        at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:40)
Caused by: com.azure.identity.CredentialUnavailableException: Managed Identity authentication is not available.
        at com.azure.identity.implementation.IdentityClient.lambda$getTokenFromMsalMIClient$28(IdentityClient.java:535)
        at reactor.core.publisher.Mono.lambda$onErrorMap$29(Mono.java:3862)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
        at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
        at reactor.core.publisher.MonoCompletionStage$MonoCompletionStageSubscription.apply(MonoCompletionStage.java:115)
        at reactor.core.publisher.MonoCompletionStage$MonoCompletionStageSubscription.apply(MonoCompletionStage.java:67)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
        Suppressed: java.lang.Exception: #block terminated with an error
                at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:104)
                at reactor.core.publisher.Mono.block(Mono.java:1779)
                at com.azure.core.credential.TokenCredential.getTokenSync(TokenCredential.java:110)
                at com.azure.core.implementation.AccessTokenCache.lambda$new$2(AccessTokenCache.java:65)
                at com.azure.core.implementation.AccessTokenCache.lambda$retrieveTokenSync$11(AccessTokenCache.java:230)
                at com.azure.core.implementation.AccessTokenCache.getTokenSync(AccessTokenCache.java:93)
                at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderHelperSync(BearerTokenAuthenticationPolicy.java:220)
                at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderSync(BearerTokenAuthenticationPolicy.java:207)
                at com.azure.security.keyvault.secrets.implementation.KeyVaultCredentialPolicy.authorizeRequestSync(KeyVaultCredentialPolicy.java:241)
                at com.azure.security.keyvault.secrets.implementation.KeyVaultCredentialPolicy.processSync(KeyVaultCredentialPolicy.java:373)
                at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:217)
                at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)
                at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)
                at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)
                at com.azure.core.http.policy.RetryPolicy.processSync(RetryPolicy.java:161)
                at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                at com.azure.core.http.policy.AddHeadersPolicy.processSync(AddHeadersPolicy.java:66)
                at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                at com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)
                at com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)
                at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                at com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)
                at com.azure.core.implementation.http.rest.SyncRestProxy.send(SyncRestProxy.java:62)
                at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:83)
                at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:124)
                at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:95)
                at jdk.proxy2/jdk.proxy2.$Proxy11.getSecretsSync(Unknown Source)
                at com.azure.security.keyvault.secrets.implementation.SecretClientImpl.getSecretsSinglePage(SecretClientImpl.java:1083)
                at com.azure.security.keyvault.secrets.SecretClient.lambda$listPropertiesOfSecrets$16(SecretClient.java:871)
                at com.azure.core.http.rest.PagedIterable.lambda$new$5(PagedIterable.java:193)
                at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:104)
                at com.azure.core.util.paging.ContinuablePagedByPageIterable$ContinuablePagedByPageIterator.<init>(ContinuablePagedByPageIterable.java:82)
                at com.azure.core.util.paging.ContinuablePagedByPageIterable.iterator(ContinuablePagedByPageIterable.java:58)
                at java.base/java.lang.Iterable.spliterator(Unknown Source)
                at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultOperation.lambda$listSecrets$0(KeyVaultOperation.java:49)
                at java.base/java.util.Optional.map(Unknown Source)
                at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultOperation.listSecrets(KeyVaultOperation.java:49)
                at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultPropertySource.loadProperties(KeyVaultPropertySource.java:77)
                at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultPropertySource.<init>(KeyVaultPropertySource.java:71)
                at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.buildKeyVaultPropertySource(KeyVaultEnvironmentPostProcessor.java:144)
                at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.buildKeyVaultPropertySourceList(KeyVaultEnvironmentPostProcessor.java:130)
                at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:93)
                at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:132)
                at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:115)
                at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185)
                at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178)
                at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156)
                at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:138)
                at org.springframework.boot.context.event.EventPublishingRunListener.multicastInitialEvent(EventPublishingRunListener.java:136)
                at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:81)
                at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:64)
                at java.base/java.lang.Iterable.forEach(Unknown Source)
                at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118)
                at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112)
                at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:63)
                at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:353)
                at org.springframework.boot.SpringApplication.run(SpringApplication.java:313)
                at com.example.redacted.ReweighJobApplication.main(ReweighJobApplication.java:18)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
                at java.base/java.lang.reflect.Method.invoke(Unknown Source)
                at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:102)
                at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:64)
                at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:40)
        Suppressed: com.azure.identity.CredentialUnavailableException: Managed Identity authentication is not available.
                ... 17 common frames omitted
                Suppressed: java.lang.Exception: #block terminated with an error
                        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:104)
                        at reactor.core.publisher.Mono.block(Mono.java:1779)
                        at com.azure.core.credential.TokenCredential.getTokenSync(TokenCredential.java:110)
                        at com.azure.core.implementation.AccessTokenCache.lambda$new$2(AccessTokenCache.java:65)
                        at com.azure.core.implementation.AccessTokenCache.lambda$retrieveTokenSync$11(AccessTokenCache.java:230)
                        at com.azure.core.implementation.AccessTokenCache.getTokenSync(AccessTokenCache.java:93)
                        at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderHelperSync(BearerTokenAuthenticationPolicy.java:220)
                        at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderSync(BearerTokenAuthenticationPolicy.java:207)
                        at com.azure.security.keyvault.secrets.implementation.KeyVaultCredentialPolicy.authorizeRequestOnChallengeSync(KeyVaultCredentialPolicy.java:340)
                        at com.azure.security.keyvault.secrets.implementation.KeyVaultCredentialPolicy.handleChallengeSync(KeyVaultCredentialPolicy.java:415)
                        at com.azure.security.keyvault.secrets.implementation.KeyVaultCredentialPolicy.processSync(KeyVaultCredentialPolicy.java:379)
                        at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                        at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:217)
                        at com.azure.core.http.policy.RetryPolicy.processSync(RetryPolicy.java:161)
                        at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                        at com.azure.core.http.policy.AddHeadersPolicy.processSync(AddHeadersPolicy.java:66)
                        at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                        at com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)
                        at com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)
                        at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                        at com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)
                        at com.azure.core.implementation.http.rest.SyncRestProxy.send(SyncRestProxy.java:62)
                        at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:83)
                        at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:124)
                        at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:95)
                        at jdk.proxy2/jdk.proxy2.$Proxy11.getSecretsSync(Unknown Source)
                        at com.azure.security.keyvault.secrets.implementation.SecretClientImpl.getSecretsSinglePage(SecretClientImpl.java:1083)
                        at com.azure.security.keyvault.secrets.SecretClient.lambda$listPropertiesOfSecrets$16(SecretClient.java:871)
                        at com.azure.core.http.rest.PagedIterable.lambda$new$5(PagedIterable.java:193)
                        at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:104)
                        at com.azure.core.util.paging.ContinuablePagedByPageIterable$ContinuablePagedByPageIterator.<init>(ContinuablePagedByPageIterable.java:82)
                        at com.azure.core.util.paging.ContinuablePagedByPageIterable.iterator(ContinuablePagedByPageIterable.java:58)
                        at java.base/java.lang.Iterable.spliterator(Unknown Source)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultOperation.lambda$listSecrets$0(KeyVaultOperation.java:49)
                        at java.base/java.util.Optional.map(Unknown Source)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultOperation.listSecrets(KeyVaultOperation.java:49)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultPropertySource.loadProperties(KeyVaultPropertySource.java:77)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultPropertySource.<init>(KeyVaultPropertySource.java:71)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.buildKeyVaultPropertySource(KeyVaultEnvironmentPostProcessor.java:144)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.buildKeyVaultPropertySourceList(KeyVaultEnvironmentPostProcessor.java:130)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:93)
                        at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:132)
                        at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:115)
                        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185)
                        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178)
                        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156)
                        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:138)
                        at org.springframework.boot.context.event.EventPublishingRunListener.multicastInitialEvent(EventPublishingRunListener.java:136)
                        at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:81)
                        at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:64)
                        at java.base/java.lang.Iterable.forEach(Unknown Source)
                        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118)
                        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112)
                        at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:63)
                        at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:353)
                        at org.springframework.boot.SpringApplication.run(SpringApplication.java:313)
                        at com.example.redacted.ReweighJobApplication.main(ReweighJobApplication.java:18)
                        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
                        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
                        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
                        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:102)
                        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:64)
                        at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:40)
        Caused by: com.microsoft.aad.msal4j.MsalServiceException: [Managed Identity] Unexpected exception occurred when parsing the response, HttpStatusCode: 500, Error message: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 1]
                at com.microsoft.aad.msal4j.AbstractManagedIdentitySource.handleResponse(AbstractManagedIdentitySource.java:66)
                at com.microsoft.aad.msal4j.IMDSManagedIdentitySource.handleResponse(IMDSManagedIdentitySource.java:117)
                at com.microsoft.aad.msal4j.AbstractManagedIdentitySource.getManagedIdentityResponse(AbstractManagedIdentitySource.java:46)
                at com.microsoft.aad.msal4j.ManagedIdentityClient.getManagedIdentityResponse(ManagedIdentityClient.java:48)
                at com.microsoft.aad.msal4j.AcquireTokenByManagedIdentitySupplier.fetchNewAccessTokenAndSaveToCache(AcquireTokenByManagedIdentitySupplier.java:90)
                at com.microsoft.aad.msal4j.AcquireTokenByManagedIdentitySupplier.execute(AcquireTokenByManagedIdentitySupplier.java:70)
                at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)
                at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)
                ... 7 common frames omitted
        Suppressed: com.azure.identity.CredentialUnavailableException: Managed Identity authentication is not available.
                ... 17 common frames omitted
                Suppressed: java.lang.Exception: #block terminated with an error
                        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:104)
                        at reactor.core.publisher.Mono.block(Mono.java:1779)
                        at com.azure.core.credential.TokenCredential.getTokenSync(TokenCredential.java:110)
                        at com.azure.core.implementation.AccessTokenCache.lambda$new$2(AccessTokenCache.java:65)
                        at com.azure.core.implementation.AccessTokenCache.lambda$retrieveTokenSync$11(AccessTokenCache.java:230)
                        at com.azure.core.implementation.AccessTokenCache.getTokenSync(AccessTokenCache.java:93)
                        at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderHelperSync(BearerTokenAuthenticationPolicy.java:220)
                        at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderSync(BearerTokenAuthenticationPolicy.java:207)
                        at com.azure.security.keyvault.secrets.implementation.KeyVaultCredentialPolicy.authorizeRequestSync(KeyVaultCredentialPolicy.java:241)
                        at com.azure.security.keyvault.secrets.implementation.KeyVaultCredentialPolicy.processSync(KeyVaultCredentialPolicy.java:373)
                        at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                        at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:217)
                        at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)
                        at com.azure.core.http.policy.RetryPolicy.processSync(RetryPolicy.java:161)
                        at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                        at com.azure.core.http.policy.AddHeadersPolicy.processSync(AddHeadersPolicy.java:66)
                        at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                        at com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)
                        at com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)
                        at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                        at com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)
                        at com.azure.core.implementation.http.rest.SyncRestProxy.send(SyncRestProxy.java:62)
                        at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:83)
                        at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:124)
                        at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:95)
                        at jdk.proxy2/jdk.proxy2.$Proxy11.getSecretsSync(Unknown Source)
                        at com.azure.security.keyvault.secrets.implementation.SecretClientImpl.getSecretsSinglePage(SecretClientImpl.java:1083)
                        at com.azure.security.keyvault.secrets.SecretClient.lambda$listPropertiesOfSecrets$16(SecretClient.java:871)
                        at com.azure.core.http.rest.PagedIterable.lambda$new$5(PagedIterable.java:193)
                        at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:104)
                        at com.azure.core.util.paging.ContinuablePagedByPageIterable$ContinuablePagedByPageIterator.<init>(ContinuablePagedByPageIterable.java:82)
                        at com.azure.core.util.paging.ContinuablePagedByPageIterable.iterator(ContinuablePagedByPageIterable.java:58)
                        at java.base/java.lang.Iterable.spliterator(Unknown Source)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultOperation.lambda$listSecrets$0(KeyVaultOperation.java:49)
                        at java.base/java.util.Optional.map(Unknown Source)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultOperation.listSecrets(KeyVaultOperation.java:49)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultPropertySource.loadProperties(KeyVaultPropertySource.java:77)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultPropertySource.<init>(KeyVaultPropertySource.java:71)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.buildKeyVaultPropertySource(KeyVaultEnvironmentPostProcessor.java:144)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.buildKeyVaultPropertySourceList(KeyVaultEnvironmentPostProcessor.java:130)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:93)
                        at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:132)
                        at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:115)
                        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185)
                        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178)
                        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156)
                        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:138)
                        at org.springframework.boot.context.event.EventPublishingRunListener.multicastInitialEvent(EventPublishingRunListener.java:136)
                        at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:81)
                        at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:64)
                        at java.base/java.lang.Iterable.forEach(Unknown Source)
                        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118)
                        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112)
                        at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:63)
                        at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:353)
                        at org.springframework.boot.SpringApplication.run(SpringApplication.java:313)
                        at com.example.redacted.ReweighJobApplication.main(ReweighJobApplication.java:18)
                        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
                        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
                        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
                        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:102)
                        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:64)
                        at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:40)
        Caused by: com.microsoft.aad.msal4j.MsalServiceException: [Managed Identity] Unexpected exception occurred when parsing the response, HttpStatusCode: 500, Error message: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 1]
                at com.microsoft.aad.msal4j.AbstractManagedIdentitySource.handleResponse(AbstractManagedIdentitySource.java:66)
                at com.microsoft.aad.msal4j.IMDSManagedIdentitySource.handleResponse(IMDSManagedIdentitySource.java:117)
                at com.microsoft.aad.msal4j.AbstractManagedIdentitySource.getManagedIdentityResponse(AbstractManagedIdentitySource.java:46)
                at com.microsoft.aad.msal4j.ManagedIdentityClient.getManagedIdentityResponse(ManagedIdentityClient.java:48)
                at com.microsoft.aad.msal4j.AcquireTokenByManagedIdentitySupplier.fetchNewAccessTokenAndSaveToCache(AcquireTokenByManagedIdentitySupplier.java:90)
                at com.microsoft.aad.msal4j.AcquireTokenByManagedIdentitySupplier.execute(AcquireTokenByManagedIdentitySupplier.java:70)
                at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)
                at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)
                ... 7 common frames omitted
        Suppressed: com.azure.identity.CredentialUnavailableException: Managed Identity authentication is not available.
                ... 17 common frames omitted
                Suppressed: java.lang.Exception: #block terminated with an error
                        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:104)
                        at reactor.core.publisher.Mono.block(Mono.java:1779)
                        at com.azure.core.credential.TokenCredential.getTokenSync(TokenCredential.java:110)
                        at com.azure.core.implementation.AccessTokenCache.lambda$new$2(AccessTokenCache.java:65)
                        at com.azure.core.implementation.AccessTokenCache.lambda$retrieveTokenSync$11(AccessTokenCache.java:230)
                        at com.azure.core.implementation.AccessTokenCache.getTokenSync(AccessTokenCache.java:93)
                        at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderHelperSync(BearerTokenAuthenticationPolicy.java:220)
                        at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderSync(BearerTokenAuthenticationPolicy.java:207)
                        at com.azure.security.keyvault.secrets.implementation.KeyVaultCredentialPolicy.authorizeRequestSync(KeyVaultCredentialPolicy.java:241)
                        at com.azure.security.keyvault.secrets.implementation.KeyVaultCredentialPolicy.processSync(KeyVaultCredentialPolicy.java:373)
                        at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                        at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:217)
                        at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)
                        at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:233)
                        at com.azure.core.http.policy.RetryPolicy.processSync(RetryPolicy.java:161)
                        at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                        at com.azure.core.http.policy.AddHeadersPolicy.processSync(AddHeadersPolicy.java:66)
                        at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                        at com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)
                        at com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)
                        at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
                        at com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)
                        at com.azure.core.implementation.http.rest.SyncRestProxy.send(SyncRestProxy.java:62)
                        at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:83)
                        at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:124)
                        at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:95)
                        at jdk.proxy2/jdk.proxy2.$Proxy11.getSecretsSync(Unknown Source)
                        at com.azure.security.keyvault.secrets.implementation.SecretClientImpl.getSecretsSinglePage(SecretClientImpl.java:1083)
                        at com.azure.security.keyvault.secrets.SecretClient.lambda$listPropertiesOfSecrets$16(SecretClient.java:871)
                        at com.azure.core.http.rest.PagedIterable.lambda$new$5(PagedIterable.java:193)
                        at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:104)
                        at com.azure.core.util.paging.ContinuablePagedByPageIterable$ContinuablePagedByPageIterator.<init>(ContinuablePagedByPageIterable.java:82)
                        at com.azure.core.util.paging.ContinuablePagedByPageIterable.iterator(ContinuablePagedByPageIterable.java:58)
                        at java.base/java.lang.Iterable.spliterator(Unknown Source)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultOperation.lambda$listSecrets$0(KeyVaultOperation.java:49)
                        at java.base/java.util.Optional.map(Unknown Source)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultOperation.listSecrets(KeyVaultOperation.java:49)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultPropertySource.loadProperties(KeyVaultPropertySource.java:77)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultPropertySource.<init>(KeyVaultPropertySource.java:71)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.buildKeyVaultPropertySource(KeyVaultEnvironmentPostProcessor.java:144)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.buildKeyVaultPropertySourceList(KeyVaultEnvironmentPostProcessor.java:130)
                        at com.azure.spring.cloud.autoconfigure.implementation.keyvault.environment.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:93)
                        at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:132)
                        at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:115)
                        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185)
                        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178)
                        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156)
                        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:138)
                        at org.springframework.boot.context.event.EventPublishingRunListener.multicastInitialEvent(EventPublishingRunListener.java:136)
                        at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:81)
                        at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:64)
                        at java.base/java.lang.Iterable.forEach(Unknown Source)
                        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118)
                        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112)
                        at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:63)
                        at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:353)
                        at org.springframework.boot.SpringApplication.run(SpringApplication.java:313)
                        at com.example.redacted.ReweighJobApplication.main(ReweighJobApplication.java:18)
                        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
                        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
                        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
                        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:102)
                        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:64)
                        at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:40)
        Caused by: com.microsoft.aad.msal4j.MsalServiceException: [Managed Identity] Unexpected exception occurred when parsing the response, HttpStatusCode: 500, Error message: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 1]
                at com.microsoft.aad.msal4j.AbstractManagedIdentitySource.handleResponse(AbstractManagedIdentitySource.java:66)
                at com.microsoft.aad.msal4j.IMDSManagedIdentitySource.handleResponse(IMDSManagedIdentitySource.java:117)
                at com.microsoft.aad.msal4j.AbstractManagedIdentitySource.getManagedIdentityResponse(AbstractManagedIdentitySource.java:46)
                at com.microsoft.aad.msal4j.ManagedIdentityClient.getManagedIdentityResponse(ManagedIdentityClient.java:48)
                at com.microsoft.aad.msal4j.AcquireTokenByManagedIdentitySupplier.fetchNewAccessTokenAndSaveToCache(AcquireTokenByManagedIdentitySupplier.java:90)
                at com.microsoft.aad.msal4j.AcquireTokenByManagedIdentitySupplier.execute(AcquireTokenByManagedIdentitySupplier.java:70)
                at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)
                at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)
                ... 7 common frames omitted
Caused by: com.microsoft.aad.msal4j.MsalServiceException: [Managed Identity] Unexpected exception occurred when parsing the response, HttpStatusCode: 500, Error message: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 1]
        at com.microsoft.aad.msal4j.AbstractManagedIdentitySource.handleResponse(AbstractManagedIdentitySource.java:66)
        at com.microsoft.aad.msal4j.IMDSManagedIdentitySource.handleResponse(IMDSManagedIdentitySource.java:117)
        at com.microsoft.aad.msal4j.AbstractManagedIdentitySource.getManagedIdentityResponse(AbstractManagedIdentitySource.java:46)
        at com.microsoft.aad.msal4j.ManagedIdentityClient.getManagedIdentityResponse(ManagedIdentityClient.java:48)
        at com.microsoft.aad.msal4j.AcquireTokenByManagedIdentitySupplier.fetchNewAccessTokenAndSaveToCache(AcquireTokenByManagedIdentitySupplier.java:90)
        at com.microsoft.aad.msal4j.AcquireTokenByManagedIdentitySupplier.execute(AcquireTokenByManagedIdentitySupplier.java:70)
        at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)
        at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)
        ... 7 common frames omitted

To Reproduce
Steps to reproduce the behavior: Not sure Yet. Will update on finding

Code Snippet
Add the code snippet that causes the issue.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

  • OS: AKS container
  • IDE: Intellij But the problem is with containers running on AKS
  • Library/Libraries: com.azure.spring:spring-cloud-azure-starter-keyvault:5.22.0
  • Java version: 17
  • App Server/Environment: [e.g. Tomcat, WildFly, Azure Function, Apache Spark, Databricks, IDE plugin or anything special]
  • Frameworks: [e.g. Spring Boot, Micronaut, Quarkus, etc]

If you suspect a dependency version mismatch (e.g. you see NoClassDefFoundError, NoSuchMethodError or similar), please check out Troubleshoot dependency version conflict article first. If it doesn't provide solution for the problem, please provide:

Additional context
Add any other context about the problem here.
Here is the process inside the container
Image

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

  • Bug Description Added
  • Repro Steps Added
  • Setup information Added

Metadata

Metadata

Assignees

Labels

azure-springAll azure-spring related issuesazure-spring-keyvaultSpring keyvault related issues.customer-reportedIssues that are reported by GitHub users external to the Azure organization.questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

Type

No type

Projects

Status

Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions