Missing trust store file or invalid trust store password are fatal

[mgorovoy]

Using Spring 6.2.11, Spring Boot 3.5.6, Spring Cloud Vault 4.3.0, application is configured in application.properties to disable fail fast.

spring.cloud.vault.fail-fast=false

Vault is accessed by retrieving VaultTemplate bean in the application code and obtaining key value operations object. Application fails to start both when the configured trust store file is not present or when the password is invalid. I was able to replace the VaultProperties bean and implement detection when resource is missing, however invalid password is still killing the application on startup.

10:42:14 [main] ERROR SpringApplication - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clientHttpRequestFactoryWrapper' defined in class path resource [org/springframework/cloud/vault/config/VaultAutoConfiguration.class]: Failed to instantiate [org.springframework.vault.config.AbstractVaultConfiguration$ClientFactoryWrapper]: Factory method 'clientHttpRequestFactoryWrapper' threw exception with message: java.io.IOException: keystore password was incorrect
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:657) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:489) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1375) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1205) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:569) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:529) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:339) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:373) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:337) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.instantiateSingleton(DefaultListableBeanFactory.java:1221) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingleton(DefaultListableBeanFactory.java:1187) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:1123) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:987) ~[spring-context-6.2.11.jar:6.2.11]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:627) ~[spring-context-6.2.11.jar:6.2.11]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.5.6.jar:3.5.6]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:752) [spring-boot-3.5.6.jar:3.5.6]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:439) [spring-boot-3.5.6.jar:3.5.6]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:318) [spring-boot-3.5.6.jar:3.5.6]
	at com.nectar.ucf.UCFServerLoader.main(UCFServerLoader.java:38) [classes/:?]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.vault.config.AbstractVaultConfiguration$ClientFactoryWrapper]: Factory method 'clientHttpRequestFactoryWrapper' threw exception with message: java.io.IOException: keystore password was incorrect
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.lambda$instantiate$0(SimpleInstantiationStrategy.java:200) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiateWithFactoryMethod(SimpleInstantiationStrategy.java:89) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:169) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[spring-beans-6.2.11.jar:6.2.11]
	... 19 more
Caused by: java.lang.IllegalStateException: java.io.IOException: keystore password was incorrect
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.create(ClientHttpRequestFactoryFactory.java:123) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.cloud.vault.config.VaultConfiguration.createClientHttpRequestFactory(VaultConfiguration.java:114) ~[spring-cloud-vault-config-4.3.0.jar:4.3.0]
	at org.springframework.cloud.vault.config.VaultAutoConfiguration.clientHttpRequestFactoryWrapper(VaultAutoConfiguration.java:132) ~[spring-cloud-vault-config-4.3.0.jar:4.3.0]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.lambda$instantiate$0(SimpleInstantiationStrategy.java:172) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiateWithFactoryMethod(SimpleInstantiationStrategy.java:89) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:169) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[spring-beans-6.2.11.jar:6.2.11]
	... 19 more
Caused by: java.io.IOException: keystore password was incorrect
	at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159) ~[?:?]
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221) ~[?:?]
	at java.base/java.security.KeyStore.load(KeyStore.java:1473) ~[?:?]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.loadKeyStore(ClientHttpRequestFactoryFactory.java:231) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.getKeyStore(ClientHttpRequestFactoryFactory.java:180) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.createTrustManagerFactory(ClientHttpRequestFactoryFactory.java:205) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.getTrustManagers(ClientHttpRequestFactoryFactory.java:189) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.getSSLContext(ClientHttpRequestFactoryFactory.java:137) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory$HttpComponents.getHttpClientBuilder(ClientHttpRequestFactoryFactory.java:324) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory$HttpComponents.usingHttpComponents(ClientHttpRequestFactoryFactory.java:287) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.create(ClientHttpRequestFactoryFactory.java:115) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.cloud.vault.config.VaultConfiguration.createClientHttpRequestFactory(VaultConfiguration.java:114) ~[spring-cloud-vault-config-4.3.0.jar:4.3.0]
	at org.springframework.cloud.vault.config.VaultAutoConfiguration.clientHttpRequestFactoryWrapper(VaultAutoConfiguration.java:132) ~[spring-cloud-vault-config-4.3.0.jar:4.3.0]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.lambda$instantiate$0(SimpleInstantiationStrategy.java:172) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiateWithFactoryMethod(SimpleInstantiationStrategy.java:89) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:169) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[spring-beans-6.2.11.jar:6.2.11]
	... 19 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
	at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159) ~[?:?]
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221) ~[?:?]
	at java.base/java.security.KeyStore.load(KeyStore.java:1473) ~[?:?]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.loadKeyStore(ClientHttpRequestFactoryFactory.java:231) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.getKeyStore(ClientHttpRequestFactoryFactory.java:180) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.createTrustManagerFactory(ClientHttpRequestFactoryFactory.java:205) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.getTrustManagers(ClientHttpRequestFactoryFactory.java:189) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.getSSLContext(ClientHttpRequestFactoryFactory.java:137) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory$HttpComponents.getHttpClientBuilder(ClientHttpRequestFactoryFactory.java:324) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory$HttpComponents.usingHttpComponents(ClientHttpRequestFactoryFactory.java:287) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.vault.client.ClientHttpRequestFactoryFactory.create(ClientHttpRequestFactoryFactory.java:115) ~[spring-vault-core-3.2.0.jar:3.2.0]
	at org.springframework.cloud.vault.config.VaultConfiguration.createClientHttpRequestFactory(VaultConfiguration.java:114) ~[spring-cloud-vault-config-4.3.0.jar:4.3.0]
	at org.springframework.cloud.vault.config.VaultAutoConfiguration.clientHttpRequestFactoryWrapper(VaultAutoConfiguration.java:132) ~[spring-cloud-vault-config-4.3.0.jar:4.3.0]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.lambda$instantiate$0(SimpleInstantiationStrategy.java:172) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiateWithFactoryMethod(SimpleInstantiationStrategy.java:89) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:169) ~[spring-beans-6.2.11.jar:6.2.11]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[spring-beans-6.2.11.jar:6.2.11]
	... 19 more

[mp911de]

As for fail-fast, it is defined as:

In some cases, it may be desirable to fail startup of a service if it cannot connect to the Vault Server.

There is clearly some room for interpretation what cannot connect actually means. Fail-fast was built with I/O, not-found and permission problems in mind to avoid Vault downtime from application startup when using config data or the older Cloud Bootstrap context.

The stack trace above indicates usage of VaultAutoConfiguration. It is producing beans that must be fully initialized to provide integrity to their callers. From the configuration arrangement, there is no way the problem could be remediated during runtime in contrast to a Vault server that could come alive.

You could work around the issue by providing your own ClientFactoryWrapper bean utilizing a lazy ClientHttpRequestFactory configuration approach along the lines of:

@Configuration(proxyBeanMethods = false)
public class LenientVaultConfiguration {

	@Bean
	public ClientFactoryWrapper clientHttpRequestFactoryWrapper() {

		ClientHttpRequestFactory factory = it -> {
			ClientOptions clientOptions = new ClientOptions(…);
			SslConfiguration sslConfiguration = VaultConfiguration.createSslConfiguration(this.vaultProperties.getSsl());

			// TODO: Use e.g. Spring Data's Lazy to defer creation and avoid repeated creation
			return ClientHttpRequestFactoryFactory.create(clientOptions, sslConfiguration);
		};

		return new ClientFactoryWrapper(factory);
	}
}