Generalize VaultStaticSecret eventing to support future dynamic secrets

[jaireddjawed]

Description

With the introduction of Vault event notifications, VSO can now stream events from Vault in real time, enabling near-immediate updates to static secrets when they are modified or deleted. We plan to extend this capability to dynamic secrets as well.

Because the core event-streaming logic in VSO is largely shared between VaultStaticSecret and VaultDynamicSecret, this PR refactors and generalizes portions of the existing static-secret eventing implementation. These changes lay the groundwork for supporting real-time updates for dynamic secrets without altering current behavior. The PR that introduces instant updates for dynamic secrets is located here. Testing has been done on static secrets to ensure behavior remains consistent.

Setup KIND cluster with a kv secret and event notifications enabled

#!/usr/bin/env bash
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1

make setup-kind
make setup-integration-test

set -e
#!/usr/bin/env bash
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1

set -e

cat <<EOF | kubectl -n vault exec -i vault-0 -- sh -e
vault --version
vault secrets disable kvv2/
vault secrets enable -path=kvv2 kv-v2
vault kv put kvv2/secret1 username="db-readonly-username" password="db-secret-password"

cat <<EOT > /tmp/policy.hcl
path "kvv2/data/secret1" {
   capabilities = ["read", "list", "subscribe"]
   subscribe_event_types = ["kv*"]
}
path "sys/events/subscribe/*" {
    capabilities = ["read"]
}
EOT
vault policy write demo /tmp/policy.hcl

# setup the necessary auth backend
vault auth disable kubernetes
vault auth enable kubernetes
vault write auth/kubernetes/config \
    kubernetes_host=https://kubernetes.default.svc
vault write auth/kubernetes/role/demo \
    bound_service_account_names=default \
    bound_service_account_namespaces=tenant-1,tenant-2 \
    policies=demo \
    ttl=1h
EOF

for ns in tenant-{1,2} ; do
    kubectl delete namespace --wait --timeout=30s "${ns}" &> /dev/null || true
    kubectl create namespace "${ns}"
done

make build docker-build deploy-kind

kubectl apply -f config/samples/secrets_v1beta1_vaultconnection.yaml
kubectl apply -f config/samples/secrets_v1beta1_vaultauth.yaml
kubectl apply -f config/samples/secrets_v1beta1_vaultstaticsecret.yaml

VaultConnection CRD

# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  labels:
    app.kubernetes.io/name: vaultconnection
    app.kubernetes.io/instance: vaultconnection-sample
    app.kubernetes.io/part-of: vault-secrets-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: vault-secrets-operator
  name: vaultconnection-sample
  namespace: tenant-1
spec:
  address: http://vault.vault.svc.cluster.local:8200
  skipTLSVerify: true
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  labels:
    app.kubernetes.io/name: vaultconnection
    app.kubernetes.io/instance: vaultconnection-sample
    app.kubernetes.io/part-of: vault-secrets-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: vault-secrets-operator
  name: vaultconnection-sample
  namespace: tenant-2
spec:
  address: http://vault.vault.svc.cluster.local:8200
  skipTLSVerify: true

VaultAuth CRD

# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  labels:
    app.kubernetes.io/name: vaultauth
    app.kubernetes.io/instance: vaultauth-sample
    app.kubernetes.io/part-of: vault-secrets-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: vault-secrets-operator
  name: vaultauth-sample
  namespace: tenant-1
spec:
  vaultConnectionRef: vaultconnection-sample
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: demo
    serviceAccount: default
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  labels:
    app.kubernetes.io/name: vaultauth
    app.kubernetes.io/instance: vaultauth-sample
    app.kubernetes.io/part-of: vault-secrets-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: vault-secrets-operator
  name: vaultauth-sample
  namespace: tenant-2
spec:
  vaultConnectionRef: vaultconnection-sample
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: demo
    serviceAccount: default

VaultStaticSecret CRD

---
apiVersion: v1
kind: Secret
metadata:
  name: secret1
  namespace: tenant-1
type: Opaque
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  namespace: tenant-1
  name: vaultstaticsecret-sample-tenant-1
spec:
  # namespace: cluster1/tenant-1
  vaultAuthRef: vaultauth-sample
  mount: kvv2
  type: kv-v2
  path: secret1
  refreshAfter: 10m
  destination:
    name: secret1
  syncConfig:
    instantUpdates: true

Manual Testing

Get the Secret Stored in k8s To Confirm The Original Value of the Secret

{
    "apiVersion": "v1",
    "data": {
        "_raw": "eyJkYXRhIjp7InBhc3N3b3JkIjoiZGItc2VjcmV0LXBhc3N3b3JkIiwidXNlcm5hbWUiOiJkYi1yZWFkb25seS11c2VybmFtZSJ9LCJtZXRhZGF0YSI6eyJjcmVhdGVkX3RpbWUiOiIyMDI1LTEyLTAzVDA2OjMyOjEzLjM0OTc1NzYzOVoiLCJjdXN0b21fbWV0YWRhdGEiOm51bGwsImRlbGV0aW9uX3RpbWUiOiIiLCJkZXN0cm95ZWQiOmZhbHNlLCJ2ZXJzaW9uIjoxfX0=",
        "password": "ZGItc2VjcmV0LXBhc3N3b3Jk",
        "username": "ZGItcmVhZG9ubHktdXNlcm5hbWU="
    },
    "kind": "Secret",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{},\"name\":\"secret1\",\"namespace\":\"tenant-1\"},\"type\":\"Opaque\"}\n"
        },
        "creationTimestamp": "2025-12-03T06:33:30Z",
        "name": "secret1",
        "namespace": "tenant-1",
        "resourceVersion": "815",
        "uid": "de782965-1c56-4dfa-ad5d-a4ae396aaa8a"
    },
    "type": "Opaque"
}

Perform an update to the password kv store in Vault and ensured that the password changed in k8s

{
    "apiVersion": "v1",
    "data": {
        "_raw": "eyJkYXRhIjp7InBhc3N3b3JkIjoiZGItc2VjcmV0LXBhc3N3b3JkLTEyMyIsInVzZXJuYW1lIjoiZGItcmVhZG9ubHktdXNlcm5hbWUifSwibWV0YWRhdGEiOnsiY3JlYXRlZF90aW1lIjoiMjAyNS0xMi0wM1QwNzowMDowMi45NzA0ODQzODNaIiwiY3VzdG9tX21ldGFkYXRhIjpudWxsLCJkZWxldGlvbl90aW1lIjoiIiwiZGVzdHJveWVkIjpmYWxzZSwidmVyc2lvbiI6Mn19",
        "password": "ZGItc2VjcmV0LXBhc3N3b3JkLTEyMw==",
        "username": "ZGItcmVhZG9ubHktdXNlcm5hbWU="
    },
    "kind": "Secret",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{},\"name\":\"secret1\",\"namespace\":\"tenant-1\"},\"type\":\"Opaque\"}\n"
        },
        "creationTimestamp": "2025-12-03T06:57:32Z",
        "name": "secret1",
        "namespace": "tenant-1",
        "resourceVersion": "1126",
        "uid": "b5bc1ce0-8299-45c8-9a3e-a9cb65c8398d"
    },
    "type": "Opaque"
}

Perform a rollback to confirm that the secret reverts to the original on a near-instant basis

{
    "apiVersion": "v1",
    "data": {
        "_raw": "eyJkYXRhIjp7InBhc3N3b3JkIjoiZGItc2VjcmV0LXBhc3N3b3JkIiwidXNlcm5hbWUiOiJkYi1yZWFkb25seS11c2VybmFtZSJ9LCJtZXRhZGF0YSI6eyJjcmVhdGVkX3RpbWUiOiIyMDI1LTEyLTAzVDA3OjAyOjQ3Ljc3NjcwNjMzNVoiLCJjdXN0b21fbWV0YWRhdGEiOm51bGwsImRlbGV0aW9uX3RpbWUiOiIiLCJkZXN0cm95ZWQiOmZhbHNlLCJ2ZXJzaW9uIjo0fX0=",
        "password": "ZGItc2VjcmV0LXBhc3N3b3Jk",
        "username": "ZGItcmVhZG9ubHktdXNlcm5hbWU="
    },
    "kind": "Secret",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{},\"name\":\"secret1\",\"namespace\":\"tenant-1\"},\"type\":\"Opaque\"}\n"
        },
        "creationTimestamp": "2025-12-03T06:57:32Z",
        "name": "secret1",
        "namespace": "tenant-1",
        "resourceVersion": "1426",
        "uid": "b5bc1ce0-8299-45c8-9a3e-a9cb65c8398d"
    },
    "type": "Opaque"
}

Perform deletion of the username field and confirm that it has been removed from k8s on a near-instant basis

{
    "apiVersion": "v1",
    "data": {
        "_raw": "eyJkYXRhIjp7InBhc3N3b3JkIjoiZGItc2VjcmV0LXBhc3N3b3JkIn0sIm1ldGFkYXRhIjp7ImNyZWF0ZWRfdGltZSI6IjIwMjUtMTItMDNUMDc6MDQ6NTQuNzc3Njc5ODhaIiwiY3VzdG9tX21ldGFkYXRhIjp7Im1ldGFkYXRhLWtleSI6InZhbHVlIn0sImRlbGV0aW9uX3RpbWUiOiIiLCJkZXN0cm95ZWQiOmZhbHNlLCJ2ZXJzaW9uIjo1fX0=",
        "password": "ZGItc2VjcmV0LXBhc3N3b3Jk"
    },
    "kind": "Secret",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{},\"name\":\"secret1\",\"namespace\":\"tenant-1\"},\"type\":\"Opaque\"}\n"
        },
        "creationTimestamp": "2025-12-03T06:57:32Z",
        "name": "secret1",
        "namespace": "tenant-1",
        "resourceVersion": "1658",
        "uid": "b5bc1ce0-8299-45c8-9a3e-a9cb65c8398d"
    },
    "type": "Opaque"
}

PCI review checklist

• I have documented a clear reason for, and description of, the change I am making.

• If applicable, I've documented a plan to revert these changes if they require more than reverting the pull request.

• If applicable, I've documented the impact of any changes to security controls.

  Examples of changes to security controls include using new access control methods, adding or removing logging pipelines, etc.

[tvoran]

From the example logs it looks like this is responding to credential generation and rotate-root. I think for dynamic secrets, this should really only care about lease revocation? Like if the lease for the dynamic credentials was revoked on the Vault side.

The credentials were generated by VSO, and IIRC it would've already re-queued the VDS object based on the ttl of the lease, so I don't think VSO should respond to that event. And I'm not following why VSO would want to react to the root credentials being rotated, since those are what Vault uses to connect to the database, and not a secret VSO would have access to.

[jaireddjawed]

From the example logs it looks like this is responding to credential generation and rotate-root. I think for dynamic secrets, this should really only care about lease revocation? Like if the lease for the dynamic credentials was revoked on the Vault side.

The credentials were generated by VSO, and IIRC it would've already re-queued the VDS object based on the ttl of the lease, so I don't think VSO should respond to that event. And I'm not following why VSO would want to react to the root credentials being rotated, since those are what Vault uses to connect to the database, and not a secret VSO would have access to.

The original code I had here was a simple skeleton that would log out all of the events that occur on a secret. The actual instant updates on dynamic secrets were included in #1159. However, I believe I found a better way to re-use the code from the VaultStaticSecret eventing system for both static and dynamic secrets. See my updated PR description for more info.
Also, I don't believe that it is possible at the moment for us to respond to lease revocations since there aren't any lease events being sent from Vault yet :/ (Vault event notification list). In #1159, I do respond to any events that are only relevant to the secret and it's role.