Iris/CW: namespace-qualify cluster-scoped RBAC to support isolated lifecycles

ClusterRole and ClusterRoleBinding names were hardcoded to "iris-controller",
causing collisions when multiple Iris instances shared a CKS cluster. Key on
namespace instead (e.g. "iris-controller-iris", "iris-controller-iris-canary")
so teardown of one cluster doesn't break another.

Adds a dedicated coreweave-canary.yaml config with namespace/label_prefix
"iris-canary" and points the canary workflow at it, so its nightly teardown
no longer interferes with persistent workloads in the default "iris" namespace.

Closes #3698

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: yonromai

.github/workflows/marin-canary-ferry-cw.yaml

@@ -23,7 +23,7 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 180
     concurrency:
-      group: canary-ferry-cw
+      group: canary-ferry-cw-iris-canary
       cancel-in-progress: true
     env:
       RUN_ID: canary-gpu-${{ github.run_id }}-${{ github.run_attempt }}
@@ -35,7 +35,11 @@ jobs:
       CANARY_MAX_WALL_CLOCK: "7200"
       WANDB_ENTITY: marin-community
       WANDB_PROJECT: marin
-      IRIS_CONFIG: lib/iris/examples/coreweave.yaml
+      IRIS_CONFIG: lib/iris/examples/coreweave-canary.yaml
+      # Must match the label_prefix and namespace in IRIS_CONFIG so teardown
+      # targets only this cluster's resources.
+      IRIS_LABEL_PREFIX: iris-canary
+      IRIS_NAMESPACE: iris-canary
 
     steps:
       - name: Checkout code
@@ -156,11 +160,11 @@ jobs:
       - name: Capture failure diagnostics
         if: failure()
         run: |
-          kubectl --kubeconfig ~/.kube/coreweave-iris -n iris \
+          kubectl --kubeconfig ~/.kube/coreweave-iris -n ${{ env.IRIS_NAMESPACE }} \
             logs -l app=iris-controller --tail=500 || true
-          kubectl --kubeconfig ~/.kube/coreweave-iris -n iris \
+          kubectl --kubeconfig ~/.kube/coreweave-iris -n ${{ env.IRIS_NAMESPACE }} \
             describe pod -l app=iris-controller || true
-          kubectl --kubeconfig ~/.kube/coreweave-iris -n iris \
+          kubectl --kubeconfig ~/.kube/coreweave-iris -n ${{ env.IRIS_NAMESPACE }} \
             get events --sort-by='.lastTimestamp' --field-selector type!=Normal || true
 
       # `cluster stop` only deletes Pods; NodePools survive and rely on the
@@ -172,7 +176,7 @@ jobs:
           .venv/bin/iris -v --config=${{ env.IRIS_CONFIG }} cluster stop || true
           if [ "${{ inputs.keep_nodepool }}" != "true" ]; then
             kubectl --kubeconfig ~/.kube/coreweave-iris \
-              delete nodepool -l iris-iris-managed=true
+              delete nodepool -l iris-${{ env.IRIS_LABEL_PREFIX }}-managed=true
           else
             echo "Keeping node pool alive (keep_nodepool=true)"
           fi

lib/iris/docs/coreweave.md

@@ -173,8 +173,8 @@ in `CoreweavePlatform`):
 |----------|---------|
 | `iris` Namespace | Isolation for all Iris resources |
 | `iris-controller` ServiceAccount | In-cluster K8s API auth for controller and worker Pods |
-| `iris-controller` ClusterRole | API permissions (see below) |
-| `iris-controller` ClusterRoleBinding | Binds ServiceAccount to ClusterRole |
+| `iris-controller-{namespace}` ClusterRole | API permissions (see below). Namespace-qualified to support multiple Iris instances on the same CKS cluster. |
+| `iris-controller-{namespace}` ClusterRoleBinding | Binds ServiceAccount to ClusterRole. Namespace-qualified to avoid collisions. |
 
 **ClusterRole permissions**:
 
@@ -364,7 +364,7 @@ The platform detects fatal errors before the full timeout expires:
 `CoreweavePlatform.start_controller()` orchestrates the full startup sequence.
 See `lib/iris/src/iris/cluster/platform/coreweave.py`.
 
-1. Apply RBAC prerequisites (Namespace, ServiceAccount, ClusterRole, ClusterRoleBinding)
+1. Apply RBAC prerequisites (Namespace, ServiceAccount, ClusterRole `iris-controller-{ns}`, ClusterRoleBinding `iris-controller-{ns}`)
 2. Create S3 credentials Secret (if S3 storage configured)
 3. Apply ConfigMap with cluster config
 4. Create/reconcile all shared NodePools in parallel via `ensure_nodepools()`

lib/iris/examples/coreweave-canary.yaml

@@ -0,0 +1,83 @@
+# Iris configuration for the CoreWeave GPU canary ferry (CI).
+#
+# Minimal config: only the scale groups the canary actually uses (cpu-erapids
+# for the controller, h100-8x for training). Uses a dedicated namespace so
+# CI teardown doesn't interfere with persistent clusters in "iris".
+
+platform:
+  label_prefix: iris-canary
+  coreweave:
+    region: US-WEST-04A
+    namespace: iris-canary
+    kubeconfig_path: ~/.kube/coreweave-iris
+    object_storage_endpoint: https://74981a43be0de7712369306c7b19133d.r2.cloudflarestorage.com
+
+storage:
+  remote_state_dir: s3://marin-na/iris/state/canary
+
+controller:
+  image: ghcr.io/marin-community/iris-controller:latest
+  coreweave:
+    port: 10000
+    service_name: iris-controller-svc
+    scale_group: cpu-erapids
+
+defaults:
+  autoscaler:
+    evaluation_interval:
+      milliseconds: 10000
+    scale_up_delay:
+      milliseconds: 60000
+    scale_down_delay:
+      milliseconds: 300000
+    startup_grace_period:
+      milliseconds: 2400000    # 40 min — covers autoscaler node provisioning + Pod startup
+  worker:
+    docker_image: ghcr.io/marin-community/iris-worker:latest
+    port: 10001
+    cache_dir: /mnt/local/iris-cache
+    runtime: kubernetes
+    default_task_image: ghcr.io/marin-community/iris-task:latest
+
+scale_groups:
+  cpu-erapids:
+    num_vms: 1
+    resources:
+      cpu: 64
+      ram: 256GB
+      disk: 1TB
+      device_type: cpu
+    worker:
+      attributes:
+        region: US-WEST-04A
+        pool: cpu-erapids
+    min_slices: 0
+    max_slices: 1
+    priority: 50
+    slice_template:
+      num_vms: 1
+      coreweave:
+        region: US-WEST-04A
+        instance_type: cd-gp-i64-erapids
+
+  h100-8x:
+    num_vms: 1
+    resources:
+      cpu: 128
+      ram: 2048GB
+      disk: 1TB
+      device_type: gpu
+      device_variant: H100
+      device_count: 8
+    worker:
+      attributes:
+        region: US-WEST-04A
+        pool: h100-8x
+    min_slices: 0
+    max_slices: 1
+    priority: 100
+    slice_template:
+      num_vms: 1
+      coreweave:
+        region: US-WEST-04A
+        instance_type: gd-8xh100ib-i128

lib/iris/examples/coreweave.yaml

@@ -79,7 +79,7 @@ scale_groups:
         pool: cpu-erapids
     min_slices: 0
     max_slices: 1  # Cost-safe default; increase for production workloads
-    priority: 100
+    priority: 50
     slice_template:
       num_vms: 1
       coreweave:
@@ -102,7 +102,7 @@ scale_groups:
         pool: h100-8x
     min_slices: 0
     max_slices: 1  # Cost-safe default; increase for production workloads
-    priority: 50
+    priority: 100
     slice_template:
       num_vms: 1
       coreweave:
@@ -125,7 +125,7 @@ scale_groups:
         pool: h100-16x
     min_slices: 0
     max_slices: 1
-    priority: 50
+    priority: 100
     slice_template:
       num_vms: 2
       coreweave:

lib/iris/src/iris/cluster/platform/coreweave.py

@@ -361,13 +361,23 @@ def __init__(
 
     # -- RBAC / Namespace Prerequisites ----------------------------------------
 
+    def _rbac_cluster_role_name(self) -> str:
+        """Namespace-qualified ClusterRole name to avoid collisions across Iris instances."""
+        return f"iris-controller-{self._namespace}"
+
     def ensure_rbac(self) -> None:
         """Create the namespace, ServiceAccount, ClusterRole, and ClusterRoleBinding.
 
         Idempotent (kubectl apply). These were previously manual operator
         prerequisites; now they're auto-applied at cluster start so a single
         ``iris cluster start`` is sufficient.
+
+        ClusterRole and ClusterRoleBinding names are qualified with the namespace
+        (e.g. ``iris-controller-iris``) so multiple Iris instances on the same
+        CKS cluster don't collide on these cluster-scoped resources.
         """
+        cluster_role_name = self._rbac_cluster_role_name()
+
         namespace_manifest = {"apiVersion": "v1", "kind": "Namespace", "metadata": {"name": self._namespace}}
 
         sa_manifest = {
@@ -379,7 +389,7 @@ def ensure_rbac(self) -> None:
         role_manifest = {
             "apiVersion": "rbac.authorization.k8s.io/v1",
             "kind": "ClusterRole",
-            "metadata": {"name": "iris-controller"},
+            "metadata": {"name": cluster_role_name},
             "rules": [
                 {
                     "apiGroups": ["compute.coreweave.com"],
@@ -412,21 +422,21 @@ def ensure_rbac(self) -> None:
         binding_manifest = {
             "apiVersion": "rbac.authorization.k8s.io/v1",
             "kind": "ClusterRoleBinding",
-            "metadata": {"name": "iris-controller"},
+            "metadata": {"name": cluster_role_name},
             "subjects": [
                 {"kind": "ServiceAccount", "name": "iris-controller", "namespace": self._namespace},
             ],
             "roleRef": {
                 "kind": "ClusterRole",
-                "name": "iris-controller",
+                "name": cluster_role_name,
                 "apiGroup": "rbac.authorization.k8s.io",
             },
         }
 
         for manifest in [namespace_manifest, sa_manifest, role_manifest, binding_manifest]:
             self._kubectl.apply_json(manifest)
 
-        logger.info("RBAC prerequisites applied (namespace=%s)", self._namespace)
+        logger.info("RBAC prerequisites applied (namespace=%s, clusterRole=%s)", self._namespace, cluster_role_name)
 
     # -- Storage Detection ----------------------------------------------------
 
@@ -1151,7 +1161,7 @@ def restart_controller(self, config: config_pb2.IrisClusterConfig) -> str:
         return self.start_controller(config)
 
     def stop_controller(self, config: config_pb2.IrisClusterConfig) -> None:
-        """Stop the controller by deleting its K8s resources."""
+        """Stop the controller and clean up its RBAC resources."""
         cw = config.controller.coreweave
         service_name = cw.service_name or "iris-controller-svc"
 
@@ -1160,7 +1170,12 @@ def stop_controller(self, config: config_pb2.IrisClusterConfig) -> None:
         self._kubectl.delete("configmap", "iris-cluster-config")
         if self._uses_s3_storage(config):
             self._kubectl.delete("secret", _S3_SECRET_NAME)
-        logger.info("Controller resources deleted")
+
+        # Clean up cluster-scoped RBAC resources created by ensure_rbac().
+        cluster_role_name = self._rbac_cluster_role_name()
+        self._kubectl.delete("clusterrolebinding", cluster_role_name, cluster_scoped=True)
+        self._kubectl.delete("clusterrole", cluster_role_name, cluster_scoped=True)
+        logger.info("Controller resources deleted (including RBAC %s)", cluster_role_name)
 
     def stop_all(
         self,

lib/iris/tests/cluster/platform/test_coreweave_platform.py

@@ -92,6 +92,8 @@ def __init__(self):
         self._services: dict[str, dict] = {}
         self._configmaps: dict[str, dict] = {}
         self._secrets: dict[str, dict] = {}
+        self._cluster_roles: dict[str, dict] = {}
+        self._cluster_role_bindings: dict[str, dict] = {}
         self._failures: dict[str, str] = {}
         self._pod_logs: dict[str, str] = {}
         self._events: list[dict] = []
@@ -191,6 +193,10 @@ def __call__(self, cmd: list[str], **kwargs) -> subprocess.CompletedProcess:
             return self._handle_delete_generic(clean_args, "configmap", self._configmaps)
         if "delete" in clean_args and "secret" in clean_args:
             return self._handle_delete_generic(clean_args, "secret", self._secrets)
+        if "delete" in clean_args and "clusterrolebinding" in clean_args:
+            return self._handle_delete_generic(clean_args, "clusterrolebinding", self._cluster_role_bindings)
+        if "delete" in clean_args and "clusterrole" in clean_args:
+            return self._handle_delete_generic(clean_args, "clusterrole", self._cluster_roles)
         if "set" in clean_args and "image" in clean_args:
             return self._handle_set_image(clean_args)
         if "rollout" in clean_args and "restart" in clean_args:
@@ -264,6 +270,19 @@ def _handle_apply(self, input_data: str, namespace: str) -> subprocess.Completed
                 "data": data.get("data", {}),
             }
             return _completed()
+        elif kind == "ClusterRole":
+            self._cluster_roles[name] = {
+                "metadata": data.get("metadata", {}),
+                "rules": data.get("rules", []),
+            }
+            return _completed()
+        elif kind == "ClusterRoleBinding":
+            self._cluster_role_bindings[name] = {
+                "metadata": data.get("metadata", {}),
+                "subjects": data.get("subjects", []),
+                "roleRef": data.get("roleRef", {}),
+            }
+            return _completed()
 
         return _completed()
 
@@ -1150,7 +1169,7 @@ def test_start_controller_reconciles_when_already_available(fake_kubectl: FakeKu
 
 
 def test_stop_controller_deletes_resources_except_nodepool(fake_kubectl: FakeKubectl):
-    """stop_controller deletes Deployment, Service, ConfigMap, and S3 secret but not NodePool."""
+    """stop_controller deletes Deployment, Service, ConfigMap, S3 secret, and RBAC but not NodePool."""
     platform = _make_platform()
     cluster_config = _make_cluster_config(remote_state_dir="s3://test-bucket/bundles")
 
@@ -1190,6 +1209,35 @@ def test_stop_controller_idempotent(fake_kubectl: FakeKubectl):
     platform.shutdown()
 
 
+def test_rbac_isolation_across_namespaces(fake_kubectl: FakeKubectl):
+    """Two Iris instances with different namespaces get isolated RBAC; teardown of one doesn't affect the other."""
+    platform_a = _make_platform(namespace="alpha")
+    platform_b = _make_platform(namespace="beta")
+
+    platform_a.ensure_rbac()
+    platform_b.ensure_rbac()
+
+    # Each gets a namespace-qualified ClusterRole and ClusterRoleBinding
+    assert "iris-controller-alpha" in fake_kubectl._cluster_roles
+    assert "iris-controller-beta" in fake_kubectl._cluster_roles
+
+    # Binding references the correct ClusterRole and namespace
+    binding_a = fake_kubectl._cluster_role_bindings["iris-controller-alpha"]
+    assert binding_a["roleRef"]["name"] == "iris-controller-alpha"
+    assert binding_a["subjects"][0]["namespace"] == "alpha"
+
+    # Stopping alpha cleans up its RBAC without affecting beta
+    platform_a.stop_controller(_make_cluster_config())
+
+    assert "iris-controller-alpha" not in fake_kubectl._cluster_roles
+    assert "iris-controller-alpha" not in fake_kubectl._cluster_role_bindings
+    assert "iris-controller-beta" in fake_kubectl._cluster_roles
+    assert "iris-controller-beta" in fake_kubectl._cluster_role_bindings
+
+    platform_a.shutdown()
+    platform_b.shutdown()
+
+
 def test_tunnel_parses_address():
     """tunnel() extracts service name and port from the address string."""
     config = config_pb2.CoreweavePlatformConfig(region="LGA1", namespace="iris")