Skip to content

Commit 69a00cd

Browse files
slntoppslntopp
committed
test-suites for external scanning with Vault and self-scanning with connected Mondoo integration
1 parent bffc8a7 commit 69a00cd

10 files changed

+640
-5
lines changed

‎tests/e2e/README.md‎

Lines changed: 56 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ End-to-end tests that deploy the Mondoo operator to a real GKE cluster and verif
66

77
- **Fresh Deploy** (`run-fresh-deploy.sh`): Builds the operator from the current branch, deploys to a GKE cluster, configures scanning, and verifies everything works.
88
- **Upgrade** (`run-upgrade.sh`): Installs a released baseline version first, verifies it, then upgrades to the current branch and verifies again.
9+
- **External Cluster** (`run-external-cluster.sh`): Deploys the operator and configures external cluster scanning against a target GKE cluster using a static kubeconfig Secret.
10+
- **Vault External Cluster** (`run-vault-external-cluster.sh`): Like External Cluster, but uses HashiCorp Vault's Kubernetes secrets engine to dynamically generate short-lived service account tokens instead of a static kubeconfig.
911
- **Registry Mirroring & Proxy** (`run-registry-mirroring.sh`): Deploys with an Artifact Registry mirror repo and optional Squid proxy. Verifies image references are rewritten, `imagePullSecrets` are propagated, and proxy env vars are set.
1012

1113
All tests pause for manual verification at each verify step (check Mondoo console for assets/scan results). Press Enter to continue or Ctrl+C to abort.
@@ -61,6 +63,7 @@ terraform apply \
6163
| `region` | no | `europe-west3` | GCP region |
6264
| `autopilot` | no | `true` | `true` for Autopilot, `false` for Standard cluster |
6365
| `enable_mirror_test` | no | `false` | Create a mirror AR repo for registry mirroring/imagePullSecrets tests |
66+
| `enable_target_cluster` | no | `false` | Create a second GKE cluster for external cluster / Vault scanning tests |
6467
| `enable_proxy_test` | no | `false` | Provision a Squid proxy VM for proxy tests (requires `enable_mirror_test`) |
6568

6669
You can also set these in a `terraform.tfvars` file.
@@ -99,6 +102,45 @@ What it does:
99102
6. Upgrades to the current branch image via local Helm chart
100103
7. Waits, verifies again, pauses for manual check
101104

105+
### External Cluster (Static Kubeconfig)
106+
107+
```bash
108+
# Provision infrastructure with a target cluster
109+
cd tests/e2e/terraform
110+
terraform apply -var="project_id=MY_PROJECT" -var="mondoo_org_id=MY_ORG" \
111+
-var="enable_target_cluster=true"
112+
113+
# Run the test
114+
cd tests/e2e
115+
./run-external-cluster.sh
116+
```
117+
118+
### Vault External Cluster
119+
120+
```bash
121+
# Provision infrastructure with a target cluster
122+
cd tests/e2e/terraform
123+
terraform apply -var="project_id=MY_PROJECT" -var="mondoo_org_id=MY_ORG" \
124+
-var="enable_target_cluster=true"
125+
126+
# Run the test
127+
cd tests/e2e
128+
./run-vault-external-cluster.sh
129+
```
130+
131+
What it does:
132+
1. Loads Terraform outputs (requires `enable_target_cluster=true`)
133+
2. Builds the operator image and pushes to Artifact Registry
134+
3. Deploys nginx test workloads to both clusters
135+
4. Installs the operator via local Helm chart
136+
5. Deploys HashiCorp Vault in dev mode to the scanner cluster
137+
6. Configures Vault: Kubernetes auth (scanner cluster) + Kubernetes secrets engine (target cluster)
138+
7. Creates target cluster CA cert Secret for TLS verification
139+
8. Applies MondooAuditConfig with `vaultAuth` external cluster config
140+
9. Waits 90s for operator to reconcile and fetch Vault credentials
141+
10. Verifies: vault-kubeconfig Secret created, CronJob has no init containers, correct volume mounts
142+
11. Pauses for manual verification in the Mondoo console
143+
102144
### Registry Mirroring & Proxy
103145

104146
```bash
@@ -148,6 +190,8 @@ tests/e2e/
148190
├── README.md
149191
├── run-fresh-deploy.sh # Fresh deploy test orchestrator
150192
├── run-upgrade.sh # Upgrade test orchestrator
193+
├── run-external-cluster.sh # External cluster scanning test orchestrator
194+
├── run-vault-external-cluster.sh # Vault external cluster test orchestrator
151195
├── run-registry-mirroring.sh # Registry mirroring & proxy test orchestrator
152196
├── terraform/
153197
│ ├── versions.tf # Provider requirements
@@ -163,16 +207,25 @@ tests/e2e/
163207
│ ├── deploy-operator-mirroring.sh # Helm install with mirror/proxy values
164208
│ ├── deploy-baseline.sh # Helm install released version
165209
│ ├── deploy-test-workload.sh # Deploy nginx for scanning
210+
│ ├── deploy-target-workload.sh # Deploy nginx + kubeconfig Secret for external cluster
211+
│ ├── deploy-target-workload-only.sh # Deploy nginx to target (no kubeconfig Secret)
212+
│ ├── deploy-vault.sh # Deploy + configure Vault for external cluster auth
166213
│ ├── apply-mondoo-config.sh # Create secret + apply MondooAuditConfig
167214
│ ├── setup-mirror-registry.sh # Create imagePullSecret for mirror AR repo
168215
│ ├── populate-mirror-registry.sh # Copy cnspec image into mirror AR repo via crane
169216
│ ├── verify.sh # Automated checks + manual verification pause
217+
│ ├── verify-external.sh # External cluster verification
218+
│ ├── verify-vault-external.sh # Vault external cluster verification
170219
│ ├── verify-mirroring.sh # Mirroring/proxy-specific verification
171220
│ └── cleanup.sh # Remove all test resources from cluster
172221
└── manifests/
173-
├── mondoo-audit-config.yaml.tpl # Standard cluster config (nodes enabled)
174-
├── mondoo-audit-config-autopilot.yaml.tpl # Autopilot config (nodes disabled)
175-
└── nginx-workload.yaml # Test workload
222+
├── mondoo-audit-config.yaml.tpl # Standard config (nodes enabled)
223+
├── mondoo-audit-config-autopilot.yaml.tpl # Autopilot config (nodes disabled)
224+
├── mondoo-audit-config-external.yaml.tpl # External cluster (static kubeconfig)
225+
├── mondoo-audit-config-external-autopilot.yaml.tpl # External cluster + Autopilot
226+
├── mondoo-audit-config-vault-external.yaml.tpl # Vault external cluster
227+
├── mondoo-audit-config-vault-external-autopilot.yaml.tpl # Vault external + Autopilot
228+
└── nginx-workload.yaml # Test workload
176229
```
177230

178231
## Notes

‎tests/e2e/manifests/mondoo-audit-config-vault-external-autopilot.yaml.tpl‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Copyright Mondoo, Inc. 2026
2+
# SPDX-License-Identifier: BUSL-1.1
3+
4+
apiVersion: k8s.mondoo.com/v1alpha2
5+
kind: MondooAuditConfig
6+
metadata:
7+
name: mondoo-client
8+
namespace: ${NAMESPACE}
9+
spec:
10+
mondooCredsSecretRef:
11+
name: mondoo-client
12+
filtering:
13+
namespaces:
14+
exclude:
15+
- kube-system
16+
- gke-managed-system
17+
- gke-managed-cim
18+
kubernetesResources:
19+
enable: true
20+
schedule: "*/5 * * * *"
21+
externalClusters:
22+
- name: target-cluster
23+
vaultAuth:
24+
server: "${VAULT_TARGET_SERVER}"
25+
vaultAddr: "${VAULT_ADDR_INTERNAL}"
26+
authRole: "mondoo-operator"
27+
credsRole: "target-cluster-scanner"
28+
ttl: "1h"
29+
targetCACertSecretRef:
30+
name: vault-target-ca-cert
31+
containers:
32+
enable: true
33+
schedule: "*/5 * * * *"
34+
nodes:
35+
# Disabled: GKE Autopilot does not allow hostPath volumes on /
36+
enable: false

‎tests/e2e/manifests/mondoo-audit-config-vault-external.yaml.tpl‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Copyright Mondoo, Inc. 2026
2+
# SPDX-License-Identifier: BUSL-1.1
3+
4+
apiVersion: k8s.mondoo.com/v1alpha2
5+
kind: MondooAuditConfig
6+
metadata:
7+
name: mondoo-client
8+
namespace: ${NAMESPACE}
9+
spec:
10+
mondooCredsSecretRef:
11+
name: mondoo-client
12+
filtering:
13+
namespaces:
14+
exclude:
15+
- kube-system
16+
- gke-managed-system
17+
- gke-managed-cim
18+
kubernetesResources:
19+
enable: true
20+
schedule: "*/5 * * * *"
21+
externalClusters:
22+
- name: target-cluster
23+
vaultAuth:
24+
server: "${VAULT_TARGET_SERVER}"
25+
vaultAddr: "${VAULT_ADDR_INTERNAL}"
26+
authRole: "mondoo-operator"
27+
credsRole: "target-cluster-scanner"
28+
ttl: "1h"
29+
targetCACertSecretRef:
30+
name: vault-target-ca-cert
31+
containers:
32+
enable: true
33+
schedule: "*/5 * * * *"
34+
nodes:
35+
enable: true
36+
style: cronjob
37+
schedule: "*/5 * * * *"

‎tests/e2e/run-operator-only.sh‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
#!/usr/bin/env bash
2+
# Copyright Mondoo, Inc. 2026
3+
# SPDX-License-Identifier: BUSL-1.1
4+
5+
# Test: Operator-Only Deploy
6+
# Builds operator from current branch and deploys to GKE Autopilot.
7+
# Does NOT create a MondooAuditConfig — apply one manually from the UI.
8+
#
9+
# Prerequisites:
10+
# - Terraform infrastructure provisioned (cd terraform && terraform apply)
11+
# - gcloud authenticated, docker, helm, kubectl available
12+
#
13+
# Usage:
14+
# ./run-operator-only.sh
15+
16+
set -euo pipefail
17+
18+
E2E_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
19+
source "${E2E_ROOT}/scripts/common.sh"
20+
21+
info "=========================================="
22+
info " Test: Operator-Only Deploy"
23+
info "=========================================="
24+
25+
# Step 1: Load Terraform outputs
26+
load_tf_outputs
27+
28+
# Step 2: Build and push operator image
29+
info "--- Step: Build and Push ---"
30+
source "${E2E_ROOT}/scripts/build-and-push.sh"
31+
32+
# Step 3: Deploy test workload
33+
info "--- Step: Deploy Test Workload ---"
34+
source "${E2E_ROOT}/scripts/deploy-test-workload.sh"
35+
36+
# Step 4: Deploy operator from local chart
37+
info "--- Step: Deploy Operator ---"
38+
source "${E2E_ROOT}/scripts/deploy-operator.sh"
39+
40+
info ""
41+
info "=========================================="
42+
info " Operator deployed. Apply AuditConfig from the UI."
43+
info "=========================================="

‎tests/e2e/run-vault-external-cluster.sh‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
#!/usr/bin/env bash
2+
# Copyright Mondoo, Inc. 2026
3+
# SPDX-License-Identifier: BUSL-1.1
4+
5+
# Test: Vault-Authenticated External Cluster Scanning
6+
# Builds operator from current branch, deploys Vault with Kubernetes auth + secrets
7+
# engine, configures external cluster scanning via Vault, and verifies scanning.
8+
#
9+
# Prerequisites:
10+
# - Terraform infrastructure provisioned with enable_target_cluster=true
11+
# - gcloud authenticated, docker, helm, kubectl available
12+
#
13+
# Usage:
14+
# ./run-vault-external-cluster.sh
15+
16+
set -euo pipefail
17+
18+
E2E_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
19+
source "${E2E_ROOT}/scripts/common.sh"
20+
21+
info "=========================================="
22+
info " Test: Vault External Cluster Scanning"
23+
info "=========================================="
24+
25+
# Step 1: Load Terraform outputs
26+
load_tf_outputs
27+
28+
if [[ "${ENABLE_TARGET_CLUSTER}" != "true" ]]; then
29+
die "Target cluster is not enabled. Run: terraform apply -var='enable_target_cluster=true'"
30+
fi
31+
32+
# Step 2: Build and push operator image
33+
info "--- Step: Build and Push ---"
34+
source "${E2E_ROOT}/scripts/build-and-push.sh"
35+
36+
# Step 3: Deploy test workload to scanner cluster
37+
info "--- Step: Deploy Test Workload (scanner cluster) ---"
38+
source "${E2E_ROOT}/scripts/deploy-test-workload.sh"
39+
40+
# Step 4: Deploy operator from local chart
41+
info "--- Step: Deploy Operator ---"
42+
source "${E2E_ROOT}/scripts/deploy-operator.sh"
43+
44+
# Step 5: Ensure CRDs include vaultAuth field (Helm doesn't upgrade CRDs)
45+
info "--- Step: Update CRDs ---"
46+
kubectl apply --server-side --force-conflicts -f "${REPO_ROOT}/config/crd/bases/"
47+
48+
# Step 6: Deploy test workload to target cluster (no kubeconfig Secret — Vault handles auth)
49+
info "--- Step: Deploy Target Workload ---"
50+
source "${E2E_ROOT}/scripts/deploy-target-workload-only.sh"
51+
52+
# Step 7: Deploy and configure Vault
53+
info "--- Step: Deploy and Configure Vault ---"
54+
source "${E2E_ROOT}/scripts/deploy-vault.sh"
55+
56+
# Step 8: Apply MondooAuditConfig with Vault auth
57+
info "--- Step: Apply Mondoo Config (with Vault auth) ---"
58+
export ENABLE_VAULT_TEST="true"
59+
source "${E2E_ROOT}/scripts/apply-mondoo-config.sh"
60+
61+
# Step 9: Wait for operator to reconcile
62+
info "Waiting 90s for operator to reconcile and Vault token fetch..."
63+
sleep 90
64+
65+
# Step 10: Verify local scanning
66+
info "--- Step: Verify (local) ---"
67+
source "${E2E_ROOT}/scripts/verify.sh"
68+
69+
# Step 11: Verify Vault-based external cluster scanning
70+
info "--- Step: Verify (Vault external cluster) ---"
71+
source "${E2E_ROOT}/scripts/verify-vault-external.sh"
72+
73+
info ""
74+
info "=========================================="
75+
info " Test: Vault External Cluster Scanning - COMPLETE"
76+
info "=========================================="

‎tests/e2e/scripts/apply-mondoo-config.sh‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,13 @@ rm -f /tmp/mondoo-creds.json
2323

2424
info "Applying MondooAuditConfig..."
2525
export NAMESPACE
26-
if [[ "${ENABLE_TARGET_CLUSTER:-false}" == "true" ]]; then
26+
if [[ "${ENABLE_VAULT_TEST:-false}" == "true" ]]; then
27+
if [[ "${AUTOPILOT}" == "true" ]]; then
28+
MANIFEST="${E2E_DIR}/manifests/mondoo-audit-config-vault-external-autopilot.yaml.tpl"
29+
else
30+
MANIFEST="${E2E_DIR}/manifests/mondoo-audit-config-vault-external.yaml.tpl"
31+
fi
32+
elif [[ "${ENABLE_TARGET_CLUSTER:-false}" == "true" ]]; then
2733
if [[ "${AUTOPILOT}" == "true" ]]; then
2834
MANIFEST="${E2E_DIR}/manifests/mondoo-audit-config-external-autopilot.yaml.tpl"
2935
else

‎tests/e2e/scripts/cleanup.sh‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,14 @@ kubectl delete secret mondoo-client -n "${NAMESPACE}" --ignore-not-found
3030
info "Deleting target-kubeconfig secret..."
3131
kubectl delete secret target-kubeconfig -n "${NAMESPACE}" --ignore-not-found
3232

33+
# Vault-related secrets
34+
info "Deleting Vault-related secrets..."
35+
kubectl delete secret vault-target-ca-cert -n "${NAMESPACE}" --ignore-not-found
36+
# Delete any operator-created vault-kubeconfig secrets
37+
for secret in $(kubectl get secrets -n "${NAMESPACE}" -l app=mondoo-k8s-scan -o name 2>/dev/null | grep vault-kubeconfig || true); do
38+
kubectl delete "${secret}" -n "${NAMESPACE}" --ignore-not-found
39+
done
40+
3341
# Helm release
3442
info "Uninstalling mondoo-operator Helm release..."
3543
helm uninstall mondoo-operator -n "${NAMESPACE}" --wait --timeout 2m 2>/dev/null || true
@@ -62,9 +70,26 @@ if [[ -n "${TARGET_KUBECONFIG_PATH:-}" && -f "${TARGET_KUBECONFIG_PATH}" ]]; the
6270
-n default --ignore-not-found || true
6371
fi
6472

73+
# Vault (from vault external cluster tests)
74+
info "Uninstalling Vault Helm release..."
75+
helm uninstall vault -n vault --wait --timeout 2m 2>/dev/null || true
76+
kubectl delete namespace vault --ignore-not-found --timeout=30s 2>/dev/null || true
77+
78+
# Clean up Vault resources in target cluster
79+
if [[ -n "${TARGET_KUBECONFIG_PATH:-}" && -f "${TARGET_KUBECONFIG_PATH}" ]]; then
80+
info "Cleaning up Vault resources in target cluster..."
81+
kubectl --kubeconfig "${TARGET_KUBECONFIG_PATH}" delete clusterrolebinding vault-secrets-engine-admin --ignore-not-found 2>/dev/null || true
82+
kubectl --kubeconfig "${TARGET_KUBECONFIG_PATH}" delete namespace vault-secrets-engine --ignore-not-found 2>/dev/null || true
83+
# Clean up any temporary ClusterRoleBindings created by Vault's secrets engine
84+
for crb in $(kubectl --kubeconfig "${TARGET_KUBECONFIG_PATH}" get clusterrolebindings -o name 2>/dev/null | grep "v-token-" || true); do
85+
kubectl --kubeconfig "${TARGET_KUBECONFIG_PATH}" delete "${crb}" --ignore-not-found 2>/dev/null || true
86+
done
87+
fi
88+
6589
# Helm repo
66-
info "Removing mondoo Helm repo..."
90+
info "Removing Helm repos..."
6791
helm repo remove mondoo 2>/dev/null || true
92+
helm repo remove hashicorp 2>/dev/null || true
6893

6994
# Mirror registry resources (from registry mirroring tests)
7095
info "Deleting mirror-registry-creds secret..."

‎tests/e2e/scripts/deploy-target-workload-only.sh‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/usr/bin/env bash
2+
# Copyright Mondoo, Inc. 2026
3+
# SPDX-License-Identifier: BUSL-1.1
4+
5+
# Deploy a test nginx workload to the target cluster WITHOUT creating a kubeconfig
6+
# Secret. Used by the Vault test where auth is handled by Vault's Kubernetes
7+
# secrets engine instead of a static kubeconfig.
8+
9+
set -euo pipefail
10+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
11+
source "${SCRIPT_DIR}/common.sh"
12+
13+
: "${TARGET_CLUSTER_NAME:?TARGET_CLUSTER_NAME must be set}"
14+
: "${TARGET_KUBECONFIG_PATH:?TARGET_KUBECONFIG_PATH must be set}"
15+
: "${REGION:?REGION must be set}"
16+
: "${PROJECT_ID:?PROJECT_ID must be set}"
17+
18+
# Refresh target cluster credentials so the kubeconfig has a fresh token
19+
info "Refreshing target cluster credentials..."
20+
KUBECONFIG="${TARGET_KUBECONFIG_PATH}" \
21+
gcloud container clusters get-credentials "${TARGET_CLUSTER_NAME}" \
22+
--region "${REGION}" --project "${PROJECT_ID}" --quiet
23+
24+
info "Deploying nginx test workload to target cluster..."
25+
kubectl --kubeconfig "${TARGET_KUBECONFIG_PATH}" apply -f "${E2E_DIR}/manifests/nginx-workload.yaml"
26+
27+
info "Waiting for nginx deployment on target cluster..."
28+
kubectl --kubeconfig "${TARGET_KUBECONFIG_PATH}" rollout status deployment/nginx-test-workload \
29+
-n default --timeout=300s
30+
31+
info "Target cluster workload deployed."

0 commit comments

Comments
 (0)