Merge branch 'main' into feat/vm-brownfield-overrides

Author: HiranAdikari

docs/architecture.md

@@ -107,6 +107,20 @@ This phase uses four modules, typically applied together:
 
 **Provider dependencies**: `rancher/rancher2 ~> 3.0`
 
+#### 2e. namespace-credential-provisioner (`modules/management/namespace-credential-provisioner`)
+
+- Deploys a long-running reconciler on the Harvester cluster that watches for tenant namespaces.
+- For each namespace, automatically creates a scoped ServiceAccount, RoleBindings, and a
+  `harvester-vm-kubeconfig` Secret that consumer teams use to authenticate the `harvester`
+  Terraform provider — no admin involvement, no file handover.
+- Backfills existing namespaces on startup (safe to deploy to running clusters).
+- Cleans up cross-namespace RoleBindings when a namespace is deleted.
+
+**Must be deployed before `tenant-space` creates namespaces** so that credentials are
+ready by the time consumer teams run `terraform apply`.
+
+**Provider dependencies**: `hashicorp/kubernetes >= 2.0`
+
 ---
 
 ### Phase 3 — Identity & Monitoring
@@ -203,7 +217,13 @@ This phase configures the `asgardeo` provider (or equivalent OIDC configuration
            │
            │ projects/namespaces ready
            ▼
-    ┌───────────────────┐ 
+    ┌─────────────────────────┐
+    │ namespace-credential-   │
+    │ provisioner (Phase 2e)  │
+    └──────────┬──────────────┘
+               │ harvesterconfig + harvester-vm-kubeconfig per namespace
+               ▼
+    ┌───────────────────┐
     │   identity        │
     │   (Phase 3a)      │
     └────────┬──────────┘

modules/management/namespace-credential-provisioner/README.md

@@ -0,0 +1,93 @@
+# Module: management/namespace-credential-provisioner
+
+Deploys a long-running reconciler on the Harvester cluster that automatically provisions
+credentials in every tenant namespace. This is a required part of the management phase —
+deploy it after `harvester-integration` and before creating tenant workloads.
+
+## What it does
+
+For every namespace labelled as a tenant namespace, the provisioner creates:
+
+1. `harvester-vm-access-<ns>` ServiceAccount with scoped RoleBindings:
+   - `harvesterhci.io:edit` in the tenant namespace (VM lifecycle)
+   - `edit` in the tenant namespace (generic Kubernetes resources)
+   - `harvesterhci.io:view` in `harvester-public` (read shared OS images)
+2. A long-lived SA token Secret
+3. `harvester-vm-kubeconfig` Secret in the namespace — a namespace-scoped kubeconfig
+   consumers use to authenticate the `harvester` Terraform provider
+
+On startup the provisioner backfills any existing namespaces that are missing the
+`harvester-vm-kubeconfig` Secret (upgrade path).
+
+On namespace deletion it cleans up the cross-namespace `harvester-public` RoleBinding.
+
+## Why this matters
+
+Without the provisioner, consumer teams cannot authenticate to Harvester to create VMs.
+The alternative — handing out admin kubeconfigs or running per-team credential setup
+manually — does not scale and creates security exposure. This provisioner eliminates
+both problems: credentials are created automatically, scoped per namespace, and revoked
+automatically when the namespace is deleted.
+
+## Deployment sequence
+
+```text
+Phase 2a  harvester-integration   — registers Harvester with Rancher
+Phase 2e  namespace-credential-provisioner  ← deploy here
+          tenant-space            — creates namespaces; provisioner reacts immediately
+```
+
+The provisioner must be running before `tenant-space` creates namespaces so that
+`harvester-vm-kubeconfig` is ready by the time consumer teams run `terraform apply`.
+
+## Usage
+
+```hcl
+module "provisioner" {
+  source = "github.com/wso2/open-cloud-datacenter//modules/management/namespace-credential-provisioner?ref=v0.8.0"
+
+  providers = {
+    kubernetes = kubernetes.harvester
+  }
+
+  harvester_api_server = "https://192.168.10.6:6443"
+  rancher_kubeconfig   = file(var.rancher_kubeconfig_path)
+
+  depends_on = [module.harvester_integration]
+}
+```
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|----------|
+| `harvester_api_server` | Harvester Kubernetes API server URL (e.g. `https://192.168.10.6:6443`) | `string` | — | yes |
+| `rancher_kubeconfig` | Kubeconfig for the Rancher cluster. Used to write `harvesterconfig` secrets into `fleet-default`. | `string` | — | yes |
+| `namespace` | Namespace to deploy the provisioner into | `string` | `"kube-system"` | no |
+| `image` | Container image for the provisioner (needs `kubectl`, `bash`, `jq`) | `string` | `"alpine/k8s:1.32.3"` | no |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| `deployment_name` | Name of the provisioner Deployment |
+| `service_account_name` | ServiceAccount used by the provisioner pod |
+
+## Security
+
+The provisioner SA has cluster-wide namespace watch and cross-namespace write access for
+ServiceAccounts, Secrets, and RoleBindings — this is the minimum required to manage
+credentials across all tenant namespaces. The credentials it creates are namespace-scoped:
+each `harvester-vm-access-<ns>` SA can only act within its own namespace (plus read-only
+access to `harvester-public` for shared images).
+
+One project per team is strongly recommended. Within a shared project, namespace isolation
+is enforced by the SA RoleBindings — not Rancher project RBAC — so consumers cannot
+cross namespace boundaries even if they share a project.
+
+## Relation to `harvester-cloud-credential`
+
+`workloads/harvester-cloud-credential` is deprecated. It served the same purpose
+(creating per-cluster Harvester credentials) but required manual invocation per cluster.
+The provisioner replaces it for all greenfield deployments. Retain the module only for
+brownfield clusters that have existing credentials that cannot be migrated.

modules/management/namespace-credential-provisioner/scripts/reconcile.sh

@@ -6,12 +6,26 @@
 # 1. Namespace watch (Harvester)
 #    Watches tenant namespaces on the Harvester cluster. For each new namespace
 #    that belongs to a Rancher project it creates:
+#
+#    Cloud-provider credentials (for RKE2 Harvester cloud provider):
 #      - ServiceAccount harvester-cloud-provider-<ns>
 #      - RoleBinding to harvesterhci.io:cloudprovider in the tenant namespace
 #      - Optional RoleBinding to view in the project's network namespace
 #      - Long-lived SA token secret
+#
+#    Consumer VM-access credentials (for tenant teams provisioning VMs/clusters):
+#      - ServiceAccount harvester-vm-access-<ns>
+#      - RoleBinding to harvesterhci.io:edit in the tenant namespace
+#      - RoleBinding to edit (k8s built-in) in the tenant namespace
+#      - RoleBinding to view in harvester-public (for shared OS images)
+#      - Long-lived SA token secret
+#      - Secret "harvester-vm-kubeconfig" in the tenant namespace containing a
+#        namespace-scoped kubeconfig. The platform team retrieves this once at
+#        onboarding and hands it to the tenant team.
+#
 #    On namespace deletion it deletes any harvesterconfig-* secrets on Rancher
-#    whose kubeconfig was built from that namespace's SA token.
+#    whose kubeconfig was built from that namespace's SA token, and cleans up
+#    the harvester-public RoleBinding for the VM-access SA.
 #
 # 2. Cluster watch (Rancher)
 #    Watches clusters.provisioning.cattle.io on the Rancher cluster. For each
@@ -22,9 +36,6 @@
 #        v2prov-secret-authorized-for-cluster already set at creation time
 #    On cluster deletion it removes harvesterconfig-<cluster-name>.
 #
-# Consumers (tenant teams) only need the rancher2 provider. No Harvester or
-# Rancher kubeconfig required on their side.
-#
 # Environment variables (injected by the Deployment):
 #   HARVESTER_API_SERVER  — Harvester Kubernetes API server URL
 #   RANCHER_KUBECONFIG    — Path to kubeconfig for Rancher's local cluster
@@ -131,6 +142,114 @@ $(echo "$kubeconfig" | sed 's/^/    /')
 EOF
 }
 
+# Build a namespace-scoped VM-access kubeconfig for tenant teams and write it
+# as the well-known "harvester-vm-kubeconfig" Secret in the tenant namespace.
+# Consumers retrieve this secret once at onboarding to provision VMs and RKE2
+# clusters using the workloads/vm and workloads/k8s-cluster OCD modules.
+# Args: ns
+write_vm_kubeconfig() {
+  local ns="$1"
+  local sa_name="harvester-vm-access-${ns}"
+  local secret_name="harvester-vm-kubeconfig"
+
+  # ServiceAccount in tenant namespace.
+  kubectl create serviceaccount "$sa_name" -n "$ns" \
+    --dry-run=client -o yaml | kubectl apply -f -
+
+  # RoleBinding — Harvester VM lifecycle (VMs, keypairs, images, NADs, backups).
+  kubectl create rolebinding "${sa_name}" \
+    --clusterrole=harvesterhci.io:edit \
+    --serviceaccount="${ns}:${sa_name}" \
+    -n "$ns" --dry-run=client -o yaml | kubectl apply -f -
+
+  # RoleBinding — Kubernetes resource edit (Secrets, PVCs, ConfigMaps).
+  kubectl create rolebinding "${sa_name}-k8s-edit" \
+    --clusterrole=edit \
+    --serviceaccount="${ns}:${sa_name}" \
+    -n "$ns" --dry-run=client -o yaml | kubectl apply -f -
+
+  # RoleBinding — read shared OS images in default namespace.
+  kubectl create rolebinding "${ns}-${sa_name}-default-view" \
+    --clusterrole=view \
+    --serviceaccount="${ns}:${sa_name}" \
+    -n "default" --dry-run=client -o yaml | kubectl apply -f -
+
+  # RoleBinding — read shared OS images in harvester-public.
+  kubectl create rolebinding "${ns}-${sa_name}-public-view" \
+    --clusterrole=view \
+    --serviceaccount="${ns}:${sa_name}" \
+    -n "harvester-public" --dry-run=client -o yaml | kubectl apply -f -
+
+  # Long-lived token secret.
+  kubectl apply -f - <<EOF
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ${sa_name}-token
+  namespace: ${ns}
+  annotations:
+    kubernetes.io/service-account.name: ${sa_name}
+type: kubernetes.io/service-account-token
+EOF
+
+  # Wait for the token to be populated by the token controller.
+  local token=""
+  for _ in $(seq 1 20); do
+    token=$(kubectl get secret "${sa_name}-token" -n "$ns" \
+      -o jsonpath='{.data.token}' 2>/dev/null || true)
+    [[ -n "$token" ]] && break
+    sleep 1
+  done
+  if [[ -z "$token" ]]; then
+    log "  ERROR: VM access token not populated for ${sa_name} in ${ns}"
+    return 1
+  fi
+
+  local token_decoded ca_cert_b64
+  token_decoded=$(echo "$token" | base64 -d)
+  ca_cert_b64=$(kubectl get configmap kube-root-ca.crt -n kube-system \
+    -o jsonpath='{.data.ca\.crt}' | base64 | tr -d '\n')
+
+  local kubeconfig
+  kubeconfig=$(cat <<EOF
+apiVersion: v1
+kind: Config
+clusters:
+- name: harvester
+  cluster:
+    certificate-authority-data: ${ca_cert_b64}
+    server: ${HARVESTER_API_SERVER}
+users:
+- name: ${ns}
+  user:
+    token: ${token_decoded}
+contexts:
+- name: ${ns}@harvester
+  context:
+    cluster: harvester
+    namespace: ${ns}
+    user: ${ns}
+current-context: ${ns}@harvester
+EOF
+)
+
+  kubectl apply -f - <<EOF
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ${secret_name}
+  namespace: ${ns}
+  annotations:
+    platform.wso2.com/vm-access-sa: "${sa_name}"
+type: Opaque
+stringData:
+  kubeconfig: |
+$(echo "$kubeconfig" | sed 's/^/    /')
+EOF
+
+  log "  [ns] VM access kubeconfig ready: ${secret_name} in ${ns}"
+}
+
 # ── Namespace watch handlers ───────────────────────────────────────────────────
 
 on_added_namespace() {
@@ -179,6 +298,11 @@ type: kubernetes.io/service-account-token
 EOF
 
   log "  [ns] SA ready: ${sa_name} in ${ns}"
+
+  # Consumer VM-access kubeconfig — separate SA with broader permissions.
+  # Explicit return propagates failure to the caller so the namespace is NOT
+  # marked processed; the watch loop will retry on the next event.
+  write_vm_kubeconfig "$ns" || return 1
 }
 
 on_deleted_namespace() {
@@ -216,6 +340,14 @@ on_deleted_namespace() {
         kubectl delete rolebinding "$rb_name_found" -n "$rb_ns" 2>/dev/null \
           && log "  [ns] deleted rolebinding ${rb_name_found} from ${rb_ns}"
       done || true
+
+  # Delete the VM-access SA's cross-namespace RoleBindings.
+  # (Resources inside the deleted namespace are cleaned up by Kubernetes.)
+  local vm_sa_name="harvester-vm-access-${ns}"
+  kubectl delete rolebinding "${ns}-${vm_sa_name}-default-view" -n "default" \
+    2>/dev/null && log "  [ns] deleted default RoleBinding for ${vm_sa_name}" || true
+  kubectl delete rolebinding "${ns}-${vm_sa_name}-public-view" -n "harvester-public" \
+    2>/dev/null && log "  [ns] deleted harvester-public RoleBinding for ${vm_sa_name}" || true
 }
 
 # ── Cluster watch handlers ─────────────────────────────────────────────────────
@@ -408,9 +540,26 @@ kubectl get namespaces -o json | jq -r '
   [[ -z "$project_id" ]] && continue
   is_system_namespace "$ns" && continue
   [[ "$role" == "network-namespace" ]] && continue
-  log "INIT namespace: ${ns} (project: ${project_id})"
-  if on_added_namespace "$ns" "$project_id"; then
-    echo "$ns" >> "$PROCESSED_NS_FILE"
+  sa_name="harvester-cloud-provider-${ns}"
+  if kubectl get secret "${sa_name}-token" -n "$ns" &>/dev/null; then
+    # Cloud-provider credentials already exist. Check for the VM-access kubeconfig
+    # separately — may be absent on pods that ran before this feature was added.
+    if kubectl get secret "harvester-vm-kubeconfig" -n "$ns" &>/dev/null; then
+      log "INIT namespace: ${ns} — already provisioned, skipping"
+      echo "$ns" >> "$PROCESSED_NS_FILE"
+    else
+      log "INIT namespace: ${ns} — backfilling VM access kubeconfig"
+      if write_vm_kubeconfig "$ns"; then
+        echo "$ns" >> "$PROCESSED_NS_FILE"
+      else
+        log "  WARN: VM access kubeconfig backfill failed for ${ns} — will retry on next watch event"
+      fi
+    fi
+  else
+    log "INIT namespace: ${ns} (project: ${project_id})"
+    if on_added_namespace "$ns" "$project_id"; then
+      echo "$ns" >> "$PROCESSED_NS_FILE"
+    fi
   fi
 done