Vault-based deployments fail #244
Description
Vault-based deployments fail due to missing GCP IAM permissions (terraform destroy failure + cluster unable to retrieve secrets)
Description
When provisioning Jenkins-X infrastructure on GKE using Vault as the secret backend (not GSM), two problems consistently occur:
1. GKE cluster cannot retrieve Vault secrets
After provisioning completes, the GKE cluster displays a status of "missing properties" for each secret.
jx secret verify
SECRET STATUS
jx-production/tekton-container-registry-auth key secret/data/tekton/container/registry/auth missing properties: .dockerconfigjson
jx-staging/tekton-container-registry-auth key secret/data/tekton/container/registry/auth missing properties: .dockerconfigjson
jx/jenkins-maven-settings key secret/data/jx/mavenSettings missing properties: settingsXml, securityXml
jx/jenkins-x-chartmuseum key secret/data/jx/adminUser missing properties: password, username
jx/jx-basic-auth-htpasswd key secret/data/jx/basic/auth/htpasswd missing properties: auth
jx/jx-basic-auth-user-password key secret/data/jx/basic/auth/user missing properties: password
jx/jx-basic-auth-user-password key secret/data/jx/basic/auth/user/password missing properties: username
jx/lighthouse-hmac-token key secret/data/lighthouse/hmac missing properties: token
jx/lighthouse-oauth-token key secret/data/lighthouse/oauth missing properties: token
jx/nexus key secret/data/nexus missing properties: password
jx/tekton-container-registry-auth key secret/data/tekton/container/registry/auth missing properties: .dockerconfigjson
jx/tekton-git key secret/data/jx/pipelineUser missing properties: token, username
The cluster does not get provisioned successfully.
2. terraform destroy fails
The same GCP user/service account that successfully runs terraform apply cannot run terraform destroy.
The destroy phase attempts to remove the KMS key ring created for Vault, but the user does not have sufficient permissions to delete the associated crypto keys.
Terraform reports:
Error: googleapi: Error 403: Permission 'cloudkms.cryptoKeyVersions.destroy' denied on resource 'projects/XXXXXX/locations/global/keyRings/keyring-XXXXX-5784660be812/cryptoKeys/crypto-key-XXXX-5784660be812/cryptoKeyVersions/1' (or it may not exist)., forbidden
The configuration is partially deleted but there remains module components
terraform state list
module.jx.random_id.random
module.jx.random_pet.current
module.jx.module.vault[0].google_kms_crypto_key.vault_crypto_key[0]
module.jx.module.vault[0].google_kms_key_ring.vault_keyring[0]
module.jx.module.vault[0].google_project_service.cloudkms_api[0]
Root Cause
The Terraform modules assume the user has full KMS and Secret Manager permission needed for Vault integration, but these permissions are not part of the default setup.
GSM deployments work because Secret Manager is provisioned automatically and Terraform does not attempt to manage or destroy crypto keys directly.
Vault deployments require explicit permissions that are not documented.
Required Permissions (Vault Only)
Adding the following IAM roles to the user/service account resolves both issues:
roles/cloudkms.admin
roles/secretmanager.secretAccessor
Notes
- These permissions are only required for Vault-based Jenkins-X environments.
- GSM-based environments work without these roles and are unaffected.
- After granting these roles,
terraform apply,terraform destroy, andjx secret verifyall work as intended.
Proposed Fix
- Update documentation to clearly list required IAM roles when using the Vault option.
- Add preflight validation (optional but recommended).
- Adjust
prevent_destroyon crypto keys or document that users must have sufficient permissions to destroy the key ring.