Iris/CW: namespace-qualify RBAC and isolate canary lifecycle

[yonromai]

Summary

Fixes #3698 — multiple Iris instances on the same CKS cluster no longer interfere with each other's lifecycle.

Problem: ClusterRole/ClusterRoleBinding were hardcoded to iris-controller (cluster-scoped), so tearing down one Iris namespace destroyed RBAC for all others. The canary workflow also ran in the default iris namespace, so its nightly teardown would kill any persistent workloads there.

Fix:

1. Qualify cluster-scoped RBAC names with namespace (iris-controller-{ns})
2. Move the canary to its own namespace (iris-canary) via a dedicated config, freeing iris for persistent use

What changed

• coreweave.py — ensure_rbac() creates iris-controller-{namespace} ClusterRole/Binding; stop_controller() cleans them up
• coreweave-canary.yaml (new) — canary-specific config with namespace: iris-canary, label_prefix: iris-canary, separate remote_state_dir
• marin-canary-ferry-cw.yaml — points at coreweave-canary.yaml; concurrency group, teardown label, and diagnostics namespace all scoped to iris-canary
• coreweave.md — RBAC table updated

Why key on namespace, not label_prefix?

RBAC binds a ServiceAccount (namespaced) to a ClusterRole. Keying on namespace keeps the naming aligned with K8s's own isolation boundary. Two configs with the same namespace but different label_prefix should share one ClusterRole, not create two that grant the same SA identical permissions.

What already worked (no changes needed)

• NodePool isolation — already uses label_prefix for names and labels
• Namespaced resources (SA, Deployment, Service, ConfigMap, Secret) — already isolated by namespace
• GCP platform — doesn't use K8s RBAC

Test plan

• Existing test_coreweave_platform.py suite passes (47 tests)
• New test_rbac_isolation_across_namespaces — two platforms create/teardown non-overlapping RBAC
• ./infra/pre-commit.py --all-files --fix passes
• Manual on CW — verified end-to-end (run 23119639900)

Manual test details

Started default cluster (namespace: iris) from laptop, then triggered the canary workflow on this branch with target_tokens=65536.

During canary run: both ClusterRoles coexisted:

iris-controller-iris          2026-03-15T20:52:22Z
iris-controller-iris-canary   2026-03-15T21:29:19Z

After canary teardown:

Check	Result
iris-controller-iris ClusterRole	Still exists
Default controller pod (iris namespace)	1/1 Running
iris-controller-iris-canary ClusterRole	Gone (cleaned up)
Canary NodePools (iris-canary-*)	Gone (cleaned up)
Default NodePools (iris-*)	Unaffected

The canary ferry itself failed at the training step (unrelated — likely 65k token edge case), but Start CoreWeave cluster, Submit canary ferry, and Tear down CoreWeave cluster all succeeded, which is what matters for the isolation test.

Note: a stale iris-controller ClusterRole from a pre-PR run (Feb 19) was found on the cluster — confirms the problem this PR fixes.

🤖 Generated with Claude Code

[yonromai]

@rjpower I have a hunch you'll have opinions about this one 😇

[rjpower]

nit: priorities are "backwards" in Iris (you and Rav can feel free to fix). groups are sorted in ascending order, so this means that we'll use GPU in preference to CPU.

[rjpower]

Seems reasonable to me!? If it's possible to use the label to infer the namespace, I think I'd prefer that, but that would mean we couldn't run multiple controllers in the same namespace (would we ever want to do that?)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 08224774d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".