Skip to content

Commit 9dc2874

Browse files
slntoppslntopp
authored
🧪 Test suites for Vault and Mondoo Integration (#1424)
* fix: generate crds * test-suites for external scanning with Vault and self-scanning with connected Mondoo integration * feat: implement Vault credential refresh logic and update user manual
1 parent e84720d commit 9dc2874

16 files changed

+783
-18
lines changed

‎charts/mondoo-operator/crds/k8s.mondoo.com_mondooauditconfigs.yaml‎

Lines changed: 86 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -393,7 +393,7 @@ spec:
393393
description: |-
394394
KubeconfigSecretRef references a Secret containing kubeconfig for the remote cluster.
395395
The Secret must have a key "kubeconfig" with the kubeconfig content.
396-
Mutually exclusive with ServiceAccountAuth and WorkloadIdentity.
396+
Mutually exclusive with ServiceAccountAuth, WorkloadIdentity, SPIFFEAuth, and VaultAuth.
397397
properties:
398398
name:
399399
default: ""
@@ -436,7 +436,7 @@ spec:
436436
serviceAccountAuth:
437437
description: |-
438438
ServiceAccountAuth configures authentication using a service account token.
439-
Mutually exclusive with KubeconfigSecretRef and WorkloadIdentity.
439+
Mutually exclusive with KubeconfigSecretRef, WorkloadIdentity, SPIFFEAuth, and VaultAuth.
440440
properties:
441441
credentialsSecretRef:
442442
description: |-
@@ -473,7 +473,7 @@ spec:
473473
spiffeAuth:
474474
description: |-
475475
SPIFFEAuth configures SPIFFE/SPIRE-based authentication using X.509 SVIDs.
476-
Mutually exclusive with KubeconfigSecretRef, ServiceAccountAuth, and WorkloadIdentity.
476+
Mutually exclusive with KubeconfigSecretRef, ServiceAccountAuth, WorkloadIdentity, and VaultAuth.
477477
properties:
478478
audience:
479479
description: |-
@@ -512,10 +512,92 @@ spec:
512512
- server
513513
- trustBundleSecretRef
514514
type: object
515+
vaultAuth:
516+
description: |-
517+
VaultAuth configures HashiCorp Vault Kubernetes secrets engine for dynamic credential generation.
518+
The operator authenticates to Vault during reconciliation using its own service account token,
519+
fetches short-lived credentials, and writes a kubeconfig Secret that the CronJob mounts.
520+
Mutually exclusive with KubeconfigSecretRef, ServiceAccountAuth, WorkloadIdentity, and SPIFFEAuth.
521+
properties:
522+
authPath:
523+
default: auth/kubernetes
524+
description: AuthPath is the Vault Kubernetes auth method
525+
mount path.
526+
type: string
527+
authRole:
528+
description: AuthRole is the Vault role for authenticating
529+
the pod's service account.
530+
type: string
531+
caCertSecretRef:
532+
description: |-
533+
CACertSecretRef references a Secret containing Vault's CA certificate
534+
for TLS verification. The Secret must have a key "ca.crt".
535+
properties:
536+
name:
537+
default: ""
538+
description: |-
539+
Name of the referent.
540+
This field is effectively required, but due to backwards compatibility is
541+
allowed to be empty. Instances of this type with an empty value here are
542+
almost certainly wrong.
543+
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
544+
type: string
545+
type: object
546+
x-kubernetes-map-type: atomic
547+
credsRole:
548+
description: CredsRole is the Vault role for generating
549+
target cluster credentials.
550+
type: string
551+
kubernetesNamespace:
552+
description: KubernetesNamespace is the target namespace
553+
for the generated service account token.
554+
type: string
555+
secretsPath:
556+
default: kubernetes
557+
description: SecretsPath is the Vault Kubernetes secrets
558+
engine mount path.
559+
type: string
560+
server:
561+
description: |-
562+
Server is the URL of the target Kubernetes API server.
563+
Example: "https://target-cluster.example.com:6443"
564+
pattern: ^https://.*
565+
type: string
566+
targetCACertSecretRef:
567+
description: |-
568+
TargetCACertSecretRef references a Secret containing the target cluster's
569+
CA certificate for TLS verification. The Secret must have a key "ca.crt".
570+
properties:
571+
name:
572+
default: ""
573+
description: |-
574+
Name of the referent.
575+
This field is effectively required, but due to backwards compatibility is
576+
allowed to be empty. Instances of this type with an empty value here are
577+
almost certainly wrong.
578+
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
579+
type: string
580+
type: object
581+
x-kubernetes-map-type: atomic
582+
ttl:
583+
description: TTL is the requested TTL for the generated
584+
credentials (e.g. "1h", "30m").
585+
type: string
586+
vaultAddr:
587+
description: |-
588+
VaultAddr is the address of the Vault server.
589+
Example: "https://vault.example.com:8200"
590+
type: string
591+
required:
592+
- authRole
593+
- credsRole
594+
- server
595+
- vaultAddr
596+
type: object
515597
workloadIdentity:
516598
description: |-
517599
WorkloadIdentity configures cloud-native Workload Identity Federation authentication.
518-
Mutually exclusive with KubeconfigSecretRef, ServiceAccountAuth, and SPIFFEAuth.
600+
Mutually exclusive with KubeconfigSecretRef, ServiceAccountAuth, SPIFFEAuth, and VaultAuth.
519601
properties:
520602
aks:
521603
description: |-

‎config/crd/bases/k8s.mondoo.com_mondooauditconfigs.yaml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -393,7 +393,7 @@ spec:
393393
description: |-
394394
KubeconfigSecretRef references a Secret containing kubeconfig for the remote cluster.
395395
The Secret must have a key "kubeconfig" with the kubeconfig content.
396-
Mutually exclusive with ServiceAccountAuth and WorkloadIdentity.
396+
Mutually exclusive with ServiceAccountAuth, WorkloadIdentity, SPIFFEAuth, and VaultAuth.
397397
properties:
398398
name:
399399
default: ""
@@ -436,7 +436,7 @@ spec:
436436
serviceAccountAuth:
437437
description: |-
438438
ServiceAccountAuth configures authentication using a service account token.
439-
Mutually exclusive with KubeconfigSecretRef and WorkloadIdentity.
439+
Mutually exclusive with KubeconfigSecretRef, WorkloadIdentity, SPIFFEAuth, and VaultAuth.
440440
properties:
441441
credentialsSecretRef:
442442
description: |-
@@ -597,7 +597,7 @@ spec:
597597
workloadIdentity:
598598
description: |-
599599
WorkloadIdentity configures cloud-native Workload Identity Federation authentication.
600-
Mutually exclusive with KubeconfigSecretRef, ServiceAccountAuth, and SPIFFEAuth.
600+
Mutually exclusive with KubeconfigSecretRef, ServiceAccountAuth, SPIFFEAuth, and VaultAuth.
601601
properties:
602602
aks:
603603
description: |-

‎controllers/k8s_scan/deployment_handler.go‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -178,7 +178,21 @@ func (n *DeploymentHandler) Reconcile(ctx context.Context) (ctrl.Result, error)
178178
}
179179
}
180180

181-
return ctrl.Result{}, nil
181+
// If any external cluster uses VaultAuth, request an earlier requeue so the
182+
// operator refreshes Vault credentials before they expire. Default to half
183+
// the requested TTL, with a floor of 10 minutes and a ceiling of 1 hour.
184+
result := ctrl.Result{}
185+
for i := range n.Mondoo.Spec.KubernetesResources.ExternalClusters {
186+
cluster := &n.Mondoo.Spec.KubernetesResources.ExternalClusters[i]
187+
if cluster.VaultAuth != nil {
188+
requeue := vaultRequeueInterval(cluster.VaultAuth.TTL)
189+
if result.RequeueAfter == 0 || requeue < result.RequeueAfter {
190+
result.RequeueAfter = requeue
191+
}
192+
}
193+
}
194+
195+
return result, nil
182196
}
183197

184198
func (n *DeploymentHandler) syncCronJob(ctx context.Context) error {

‎controllers/k8s_scan/deployment_handler_test.go‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -500,7 +500,7 @@ func (s *DeploymentHandlerSuite) TestReconcile_ExternalCluster_VaultAuth() {
500500

501501
result, err := d.Reconcile(s.ctx)
502502
s.NoError(err)
503-
s.True(result.IsZero())
503+
s.Equal(30*time.Minute, result.RequeueAfter, "should requeue at default interval when no TTL is set")
504504

505505
// Verify kubeconfig Secret was created
506506
kubeconfigSecret := &corev1.Secret{}
@@ -599,7 +599,7 @@ func (s *DeploymentHandlerSuite) TestReconcile_ExternalCluster_VaultAuth_WithCAC
599599

600600
result, err := d.Reconcile(s.ctx)
601601
s.NoError(err)
602-
s.True(result.IsZero())
602+
s.Equal(30*time.Minute, result.RequeueAfter, "should requeue at default interval when no TTL is set")
603603

604604
// Verify mock received the Vault CA cert bytes
605605
s.Equal([]byte("-----BEGIN CERTIFICATE-----\nvault-ca\n-----END CERTIFICATE-----"), receivedVaultCA)
@@ -758,7 +758,7 @@ func (s *DeploymentHandlerSuite) createDeploymentHandlerWithVaultMock(vaultFetch
758758
MondooOperatorConfig: &mondoov1alpha2.MondooOperatorConfig{},
759759
MondooClientBuilder: mondooclient.NewClient,
760760
VaultTokenFetcher: vaultFetcher,
761-
SATokenPath: tmpFile.Name(),
761+
SATokenPath: tmpName,
762762
}
763763
}
764764

‎controllers/k8s_scan/vault.go‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import (
88
"encoding/base64"
99
"fmt"
1010
"strings"
11+
"time"
1112

1213
vault "github.com/hashicorp/vault-client-go"
1314
"github.com/hashicorp/vault-client-go/schema"
@@ -93,6 +94,35 @@ func DefaultVaultTokenFetcher(ctx context.Context, saToken string, config v1alph
9394
return token, nil
9495
}
9596

97+
// vaultRequeueInterval calculates a reconcile requeue interval based on the Vault TTL.
98+
// It targets half the TTL so credentials are refreshed well before expiry, with a
99+
// floor of 10 minutes and a ceiling of 1 hour.
100+
func vaultRequeueInterval(ttl string) time.Duration {
101+
const (
102+
minInterval = 10 * time.Minute
103+
maxInterval = 1 * time.Hour
104+
defaultInterval = 30 * time.Minute
105+
)
106+
107+
if ttl == "" {
108+
return defaultInterval
109+
}
110+
111+
d, err := time.ParseDuration(ttl)
112+
if err != nil {
113+
return defaultInterval
114+
}
115+
116+
half := d / 2
117+
if half < minInterval {
118+
return minInterval
119+
}
120+
if half > maxInterval {
121+
return maxInterval
122+
}
123+
return half
124+
}
125+
96126
// VaultKubeconfigSecretName returns the name for the Vault kubeconfig Secret.
97127
func VaultKubeconfigSecretName(prefix, clusterName string) string {
98128
return fmt.Sprintf("%s-vault-kubeconfig-%s", prefix, clusterName)
@@ -126,7 +156,7 @@ current-context: default
126156
users:
127157
- name: vault
128158
user:
129-
token: %s
159+
token: "%s"
130160
`, clusterConfig, token)
131161
}
132162

‎docs/user-manual.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -453,7 +453,7 @@ externalClusters:
453453

454454
> **Note: Credential refresh timing**
455455
>
456-
> Vault credentials are refreshed each time the operator reconciles (typically every 7 days, or when configuration changes). Ensure the requested TTL is longer than the operator's reconcile interval. If you need more frequent credential rotation, consider adjusting the operator's reconcile schedule.
456+
> When VaultAuth is configured, the operator automatically reconciles at half the requested TTL (clamped between 10 minutes and 1 hour) to refresh credentials before they expire. For example, a `ttl: "1h"` results in a 30-minute refresh interval. If no TTL is specified, the operator defaults to refreshing every 30 minutes.
457457

458458
## Container Image Scanning
459459

‎tests/e2e/README.md‎

Lines changed: 56 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ End-to-end tests that deploy the Mondoo operator to a real GKE cluster and verif
66

77
- **Fresh Deploy** (`run-fresh-deploy.sh`): Builds the operator from the current branch, deploys to a GKE cluster, configures scanning, and verifies everything works.
88
- **Upgrade** (`run-upgrade.sh`): Installs a released baseline version first, verifies it, then upgrades to the current branch and verifies again.
9+
- **External Cluster** (`run-external-cluster.sh`): Deploys the operator and configures external cluster scanning against a target GKE cluster using a static kubeconfig Secret.
10+
- **Vault External Cluster** (`run-vault-external-cluster.sh`): Like External Cluster, but uses HashiCorp Vault's Kubernetes secrets engine to dynamically generate short-lived service account tokens instead of a static kubeconfig.
911
- **Registry Mirroring & Proxy** (`run-registry-mirroring.sh`): Deploys with an Artifact Registry mirror repo and optional Squid proxy. Verifies image references are rewritten, `imagePullSecrets` are propagated, and proxy env vars are set.
1012

1113
All tests pause for manual verification at each verify step (check Mondoo console for assets/scan results). Press Enter to continue or Ctrl+C to abort.
@@ -61,6 +63,7 @@ terraform apply \
6163
| `region` | no | `europe-west3` | GCP region |
6264
| `autopilot` | no | `true` | `true` for Autopilot, `false` for Standard cluster |
6365
| `enable_mirror_test` | no | `false` | Create a mirror AR repo for registry mirroring/imagePullSecrets tests |
66+
| `enable_target_cluster` | no | `false` | Create a second GKE cluster for external cluster / Vault scanning tests |
6467
| `enable_proxy_test` | no | `false` | Provision a Squid proxy VM for proxy tests (requires `enable_mirror_test`) |
6568

6669
You can also set these in a `terraform.tfvars` file.
@@ -99,6 +102,45 @@ What it does:
99102
6. Upgrades to the current branch image via local Helm chart
100103
7. Waits, verifies again, pauses for manual check
101104

105+
### External Cluster (Static Kubeconfig)
106+
107+
```bash
108+
# Provision infrastructure with a target cluster
109+
cd tests/e2e/terraform
110+
terraform apply -var="project_id=MY_PROJECT" -var="mondoo_org_id=MY_ORG" \
111+
-var="enable_target_cluster=true"
112+
113+
# Run the test
114+
cd tests/e2e
115+
./run-external-cluster.sh
116+
```
117+
118+
### Vault External Cluster
119+
120+
```bash
121+
# Provision infrastructure with a target cluster
122+
cd tests/e2e/terraform
123+
terraform apply -var="project_id=MY_PROJECT" -var="mondoo_org_id=MY_ORG" \
124+
-var="enable_target_cluster=true"
125+
126+
# Run the test
127+
cd tests/e2e
128+
./run-vault-external-cluster.sh
129+
```
130+
131+
What it does:
132+
1. Loads Terraform outputs (requires `enable_target_cluster=true`)
133+
2. Builds the operator image and pushes to Artifact Registry
134+
3. Deploys nginx test workloads to both clusters
135+
4. Installs the operator via local Helm chart
136+
5. Deploys HashiCorp Vault in dev mode to the scanner cluster
137+
6. Configures Vault: Kubernetes auth (scanner cluster) + Kubernetes secrets engine (target cluster)
138+
7. Creates target cluster CA cert Secret for TLS verification
139+
8. Applies MondooAuditConfig with `vaultAuth` external cluster config
140+
9. Waits 90s for operator to reconcile and fetch Vault credentials
141+
10. Verifies: vault-kubeconfig Secret created, CronJob has no init containers, correct volume mounts
142+
11. Pauses for manual verification in the Mondoo console
143+
102144
### Registry Mirroring & Proxy
103145

104146
```bash
@@ -148,6 +190,8 @@ tests/e2e/
148190
├── README.md
149191
├── run-fresh-deploy.sh # Fresh deploy test orchestrator
150192
├── run-upgrade.sh # Upgrade test orchestrator
193+
├── run-external-cluster.sh # External cluster scanning test orchestrator
194+
├── run-vault-external-cluster.sh # Vault external cluster test orchestrator
151195
├── run-registry-mirroring.sh # Registry mirroring & proxy test orchestrator
152196
├── terraform/
153197
│ ├── versions.tf # Provider requirements
@@ -163,16 +207,25 @@ tests/e2e/
163207
│ ├── deploy-operator-mirroring.sh # Helm install with mirror/proxy values
164208
│ ├── deploy-baseline.sh # Helm install released version
165209
│ ├── deploy-test-workload.sh # Deploy nginx for scanning
210+
│ ├── deploy-target-workload.sh # Deploy nginx + kubeconfig Secret for external cluster
211+
│ ├── deploy-target-workload-only.sh # Deploy nginx to target (no kubeconfig Secret)
212+
│ ├── deploy-vault.sh # Deploy + configure Vault for external cluster auth
166213
│ ├── apply-mondoo-config.sh # Create secret + apply MondooAuditConfig
167214
│ ├── setup-mirror-registry.sh # Create imagePullSecret for mirror AR repo
168215
│ ├── populate-mirror-registry.sh # Copy cnspec image into mirror AR repo via crane
169216
│ ├── verify.sh # Automated checks + manual verification pause
217+
│ ├── verify-external.sh # External cluster verification
218+
│ ├── verify-vault-external.sh # Vault external cluster verification
170219
│ ├── verify-mirroring.sh # Mirroring/proxy-specific verification
171220
│ └── cleanup.sh # Remove all test resources from cluster
172221
└── manifests/
173-
├── mondoo-audit-config.yaml.tpl # Standard cluster config (nodes enabled)
174-
├── mondoo-audit-config-autopilot.yaml.tpl # Autopilot config (nodes disabled)
175-
└── nginx-workload.yaml # Test workload
222+
├── mondoo-audit-config.yaml.tpl # Standard config (nodes enabled)
223+
├── mondoo-audit-config-autopilot.yaml.tpl # Autopilot config (nodes disabled)
224+
├── mondoo-audit-config-external.yaml.tpl # External cluster (static kubeconfig)
225+
├── mondoo-audit-config-external-autopilot.yaml.tpl # External cluster + Autopilot
226+
├── mondoo-audit-config-vault-external.yaml.tpl # Vault external cluster
227+
├── mondoo-audit-config-vault-external-autopilot.yaml.tpl # Vault external + Autopilot
228+
└── nginx-workload.yaml # Test workload
176229
```
177230

178231
## Notes

0 commit comments

Comments
 (0)