io.quarkus.vault.runtime.config.VaultRuntimeConfig.connectTimeout() " because "vaultRuntimeConfig" is null

[rahulrathor-brightly]

Hi Folks,

I am updating the quarkus bom version from 3.2.11.Final-redhat-00001 to 3.15.3.redhat-00002. But getting following Error while deploying using openshift:

java.lang.RuntimeException: Error injecting io.quarkus.vault.runtime.VaultDynamicCredentialsManager 
    io.quarkus.vault.runtime.VaultCredentialsProvider.vaultDynamicCredentialsManager
    at io.quarkus.vault.runtime.VaultCredentialsProvider_Bean.doCreate(Unknown Source)
    at io.quarkus.vault.runtime.VaultCredentialsProvider_Bean.create(Unknown Source)
    at io.quarkus.vault.runtime.VaultCredentialsProvider_Bean.create(Unknown Source)
    at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
    at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.c55(Unknown Source)
    at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source)
    at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
    at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:21)
    at io.quarkus.vault.runtime.VaultCredentialsProvider_ClientProxy.arc$delegate(Unknown Source)
    at io.quarkus.vault.runtime.VaultCredentialsProvider_ClientProxy.getCredentials(Unknown Source)
    at io.quarkus.agroal.runtime.AgroalVaultCredentialsProviderPassword.asProperties(AgroalVaultCredentialsProviderPassword.java:21)
    at io.agroal.api.security.AgroalDefaultSecurityProvider.getSecurityProperties(AgroalDefaultSecurityProvider.java:23)
    at io.agroal.pool.ConnectionFactory.securityProperties(ConnectionFactory.java:211)
    at io.agroal.pool.ConnectionFactory.securityProperties(ConnectionFactory.java:200)
    at io.agroal.pool.ConnectionFactory.jdbcProperties(ConnectionFactory.java:181)
    at io.agroal.pool.ConnectionFactory.createConnection(ConnectionFactory.java:225)
    at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:580)
    at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:561)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at io.agroal.pool.util.PriorityScheduledExecutor.beforeExecute(PriorityScheduledExecutor.java:75)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.NullPointerException: Cannot invoke "io.quarkus.vault.runtime.config.VaultRuntimeConfig.connectTimeout()" because "vaultRuntimeConfig" is null
    at io.quarkus.vault.runtime.client.MutinyVertxClientFactory.createHttpClient(MutinyVertxClientFactory.java:22)
    at io.quarkus.vault.runtime.client.VaultClientProducer.sharedVaultClient(VaultClientProducer.java:47)
    at io.quarkus.vault.runtime.client.VaultClientProducer_ProducerMethod_sharedVaultClient_mYWxQxBV57LFo2E7aMyQXz4-ECo_Bean.doCreate(Unknown Source)
    at io.quarkus.vault.runtime.client.VaultClientProducer_ProducerMethod_sharedVaultClient_mYWxQxBV57LFo2E7aMyQXz4-ECo_Bean.create(Unknown Source)
    at io.quarkus.vault.runtime.client.VaultClientProducer_ProducerMethod_sharedVaultClient_mYWxQxBV57LFo2E7aMyQXz4-ECo_Bean.create(Unknown Source)
    at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
    at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
    at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
    at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
    at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
    at io.quarkus.vault.runtime.client.VaultClientProducer_ProducerMethod_sharedVaultClient_mYWxQxBV57LFo2E7aMyQXz4-ECo_Bean.get(Unknown Source)
    at io.quarkus.vault.runtime.client.VaultClientProducer_ProducerMethod_sharedVaultClient_mYWxQxBV57LFo2E7aMyQXz4-ECo_Bean.get(Unknown Source)
    at io.quarkus.vault.runtime.VaultDynamicCredentialsManager_Bean.doCreate(Unknown Source)
    at io.quarkus.vault.runtime.VaultDynamicCredentialsManager_Bean.create(Unknown Source)
    at io.quarkus.vault.runtime.VaultDynamicCredentialsManager_Bean.create(Unknown Source)
    at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
    at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
    at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
    at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
    at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
    at io.quarkus.vault.runtime.VaultDynamicCredentialsManager_Bean.get(Unknown Source)
    at io.quarkus.vault.runtime.VaultDynamicCredentialsManager_Bean.get(Unknown Source)
    ... 25 more

Note: Also, with the quarkus version 3.2.11.Final-redhat-00001 We are using following vault configuration property:

quarkus.vault.url=
quarkus.vault.credentials-provider.mydatabase.kv-path=
quarkus.vault.credentials-provider.mydatabase.credentials-role=
quarkus.vault.credentials-provider.mydatabase.credentials-mount=
quarkus.vault.authentication.kubernetes.role=
quarkus.vault.authentication.kubernetes.jwt-token-path=
quarkus.vault.authentication.kubernetes.auth-mount-path=
quarkus.vault.tls.use-kubernetes-ca-cert=false

Could anybody help with the above issue I am facing

[quarkus-bot[bot]]

/cc @geoand (kubernetes,openshift), @iocanel (kubernetes,openshift), @radcortez (config), @vsevel (vault)

[rahulrathor-brightly]

/cc @geoand (kubernetes,openshift), @iocanel (kubernetes,openshift), @radcortez (config), @vsevel (vault)

[vsevel]

how come your configuration is null?
is this your dev config?

[rahulrathor-brightly]

@vsevel I have removed the values due to data integrity reason but through environment pipelines we are providing the exact values.

[rahulrathor-brightly]

@vsevel Also with the values we are providing, the application is working with the 3.2.11.Final-redhat-00001 but same values not working with 3.15.3.redhat-00002 version. Is any new configuration needs to be added with newer version

[vsevel]

you are probably moving from quarkus vault 3.1, to 4.1
check the release notes in https://github.com/quarkiverse/quarkus-vault/releases
you can try to upgrade to quarkus 3.8 first
I do not recall having changes in the config
it would help to make sure you do value the properties you are showing

[rahulrathor-brightly]

Thanks for the reply @vsevel
I have firstly tried to upgrade feom 3.2.11 to 3.8.3 but having the same issue. I have dig down in the library further where I have found in version 3.2.11 the class used to initialize VaultConfigHolder was VaultBootstrapConfig but in version 3.8.3 or higher it is VaultRuntimeConfig.
Properties with values

quarkus.vault.url=vault-URL
quarkus.vault.credentials-provider.mydatabase.kv-path=database/aw-ue1-dv1/creds/app-dev
quarkus.vault.credentials-provider.mydatabase.credentials-role=app-dev
quarkus.vault.credentials-provider.mydatabase.credentials-mount=database/aw-ue1-dv1
quarkus.vault.authentication.kubernetes.role=application
quarkus.vault.authentication.kubernetes.jwt-token-path=/run/secrets/kubernetes.io/serviceaccount/token
quarkus.vault.authentication.kubernetes.auth-mount-path=auth/kubernetes/vault-config-application-configurator/aw-ue1-dv1
quarkus.vault.tls.use-kubernetes-ca-cert=false

[a86rbchn]

Have you managed to solve this? We have the same. Interesting is that this property exists if you try to get through ConfigProvider.getConfig()

[rahulrathor-brightly]

@a86rbchn No it hasn't been resolved yet!
@vsevel Could you please have a look?

[rahulrathor-brightly]

@a86rbchn Any Success?

[hamycern]

I experienced the same problem, but I wrote a VaultEngineAdapter using the following constructor:

@ApplicationScoped
public class VaultEngineAdapter {

	@Inject
	public VaultEngineAdapter(VaultKVSecretEngine kvSecretEngine,
			VaultRuntimeConfig vaultRuntimeConfig, VaultConfigHolder vaultConfigHolder) {

		this.kvSecretEngine = Objects.requireNonNull(kvSecretEngine);
		this.vaultRuntimeConfig = vaultRuntimeConfig;

		if (vaultConfigHolder.getVaultRuntimeConfig() == null) {
			vaultConfigHolder.setVaultRuntimeConfig(vaultRuntimeConfig);
		}
	}
}

Injects both the config and the holder (which is supposed to already have the config, but this is not the case (it is not assigned yet). After manually setting it, I can use the secret-engine normally on OpenShift. Problem does not occur in local development environment but only on OpenShift platform.

The above workaround is not "pretty" but it overcomes the problem.

[rahulrathor-brightly]

I have tried but this didn't worked for me. Also removed the following code snippet from constructor

this.kvSecretEngine = Objects.requireNonNull(kvSecretEngine);
this.vaultRuntimeConfig = vaultRuntimeConfig;

[vsevel]

I have a similar behavior if I do not provide a vault url, such as in -Dquarkus.vault.url="https://myvault" for instance.
could this be the situation you are facing?

[hamycern]

I do specify quarkus.vault.url in application.yaml and this application.yaml is configured via QUARKUS_CONFIG_LOCATIONS environment variable.

[vsevel]

do you have a reoproducer?

[satnamt]

@vsevel Application able to load vaultRuntimeConfig locally, however same is not working on OCP/Kubernetes. We are using OCP/vault version.

Vault v1.18.3
OCP - 4.16.37

Sample piece of code: `

ResourceAccessor resourceAccessor = new ClassLoaderResourceAccessor(
Thread.currentThread().getContextClassLoader());

        DatabaseConnection dbConnection = DatabaseFactory.getInstance().openConnection(
                datasourceUrl,
                credentialsProvider.getCredentials("mydatabase").get("user"),
                credentialsProvider.getCredentials("mydatabase").get("password"),
                null,
                resourceAccessor);

        Database database = DatabaseFactory.getInstance().findCorrectDatabaseImplementation(dbConnection);

`

[cescoffier]

I'm wondering if the problem comes from https://github.com/quarkiverse/quarkus-vault/blob/main/runtime/src/main/java/io/quarkus/vault/runtime/config/VaultConfigSourceFactory.java#L20. Could it be that this line is not executed?

[satnamt]

I explored the significant updates in the latest Quarkus version (3.2.x to 3.15.x) concerning configuration circular dependencies, which guided my investigation. While debugging, I discovered that the “quarkus-vault” library was unable to fetch the correct configuration during application startup due to its predefined ordinal value (270).

To address this, I switched from using “application.properties” and Kubernetes config maps to environment variables for supplying configurations. This adjustment effectively resolved the issue.

[cescoffier]

@radcortez came up with the same conclusion.

[radcortez]

Configuration sources like Vault, Consul, Database, and others require configurations to be able to configure themselves (like URLs, passwords, etc). We call these ConfigurableConfigSources. These sources can be configured using the main sources: System Properties, Environment Variables, application.properties, and a few more (in reality, any source that does not depend on any other configuration).

As you can imagine, it would be extremely complicated to allow any source to configure another, as this could also cause dependency cycles (imagine the Vault Config stored in the Database, the Database Config stored in Consul, and the Consul Config stored in Vault).

If quarkus.vault.url is set through quarkus.config.locations, any source loaded by quarkus.config.location is itself a ConfigurableConfigSource (because it requires configuration to configure itself), and its content is not available to configure other Configurable sources (like Vault).

To conclude, configurations required by Vault, must be set in the main sources, listed here: https://quarkus.io/guides/config-reference#configuration-sources, or any other custom source that is not itself configurable.