osdc: rename arc-cbr-production → meta-prod-aws-ue2 (#815)

Rename the us-east-2 prod cluster `arc-cbr-production` →
`meta-prod-aws-ue2` to match the `meta-prod-aws-*` convention (mirrors
`meta-prod-aws-ue1`). Renames cluster ID, `cluster_name`, and
`github_secret_name`; keeps the legacy `state_bucket`
(`ciforge-tfstate-arc-cbr-prod`) like ue1 did.

`cluster_name` is a ForceNew property of the EKS cluster, so this is a
destroy + recreate, not an in-place change.

## ⚠️ Runner-group change — needs confirmation before rollout

Unlike the uw1 rename, this cluster currently uses the **shared
`default` / `release-runners`** groups (it sets no `runner_group`). This
PR moves it to a **dedicated `meta-prod-aws-ue2`** group (full ue1
mirror). Consequences to confirm:
- A new `meta-prod-aws-ue2` runner group must be created at the pytorch
org with `pytorch/pytorch` access.
- **Nothing else may rely on prod runners being in
`default`/`release-runners`** — in particular release workflows that
target `release-runners`. The Grafana mapping for
`default`/`release-runners` is replaced by `meta-prod-aws-ue2`, so those
groups drop off the dashboard.

If keeping `default`/`release-runners` is preferred, say so and I'll
drop the `runner_group` line + revert the grafana mapping change.

## Rollout

0. Rebase to latest `main`.
1. Drain traffic: `just drain-runners arc-cbr-production` (or the `OSDC:
Drain cluster` workflow). Old id, since the rename hasn't merged.
2. Destroy: `aws s3 rm s3://pytorch-arc-cbr-production-harbor-registry
--recursive --region us-east-2` (also delete versions per
`cluster-recreation.md` §4), then `./scripts/destroy-cluster.sh
arc-cbr-production`. Leave the state bucket.
3. Deploy the new cluster **with `pause_runners`** (local edit, do NOT
commit): `just deploy-base meta-prod-aws-ue2` → plant the
`meta-prod-aws-ue2` App secret → `just deploy meta-prod-aws-ue2`. No
`just bootstrap` (state bucket persists).
4. Smoke test: `just smoke meta-prod-aws-ue2`.
5. Merge and deploy again **without `pause_runners`**: `git checkout
osdc/clusters.yaml`, merge, then dispatch `OSDC: Deploy production` with
`target: meta-prod-aws-ue2`.

## Notes
- Full propagation: observability docs
(loki/mimir/observability/estimates), operations, pypi-package-cache,
and test fixtures had `arc-cbr-production` as the canonical example —
all updated to `meta-prod-aws-ue2`.
- Branches off `main`, so it overlaps the uw1 rename PR (#814) on a few
shared lines (deploy-prod rollout comment, grafana IN-lists); whichever
lands second needs a trivial rebase.

`just lint` (13/13) and `just test` (98.70% cov) pass.

## Also: drop unused modules

Removes `pypi-cache` and `cache-enforcer` from this cluster's `modules`
list (the `lf-prod-aws-*` clusters run neither). Just stops deploying
them here; module teardown is a separate cleanup, and the `pypi_cache:`
config block is left in place for now.

---------

Signed-off-by: Huy Do <huydo@meta.com>

Author: huydhn

.github/workflows/_osdc-deploy.yml

@@ -4,7 +4,7 @@ on:
   workflow_call:
     inputs:
       cluster:
-        description: "Cluster id from clusters.yaml (e.g. meta-staging-aws-uw1, arc-cbr-production)"
+        description: "Cluster id from clusters.yaml (e.g. meta-staging-aws-uw1, meta-prod-aws-ue2)"
         required: true
         type: string
       environment:

.github/workflows/_osdc-slow-tests.yml

@@ -12,7 +12,7 @@ on:
   workflow_call:
     inputs:
       cluster:
-        description: "Cluster id from clusters.yaml (e.g. meta-staging-aws-uw1, arc-cbr-production)"
+        description: "Cluster id from clusters.yaml (e.g. meta-staging-aws-uw1, meta-prod-aws-ue2)"
         required: true
         type: string
       environment:

.github/workflows/osdc-deploy-prod.yml

@@ -5,7 +5,7 @@ name: "OSDC: Deploy production"
 # the reusable workflow, which selects LF_AWS_DEPLOY_ROLE_ARN over the Meta role.
 #
 # Default behavior (target = all): sequential rollout of all clusters.
-#   Meta: meta-prod-aws-uw1 → arc-cbr-production → meta-prod-aws-ue1
+#   Meta: meta-prod-aws-uw1 → meta-prod-aws-ue2 → meta-prod-aws-ue1
 #   LF:   lf-prod-aws-ue1 → lf-prod-aws-ue2 (runs in parallel with Meta)
 #
 # Use `all-meta` or `all-lf` to deploy only one account's clusters.
@@ -30,7 +30,7 @@ on:
           - all-meta
           - all-lf
           - meta-prod-aws-uw1
-          - arc-cbr-production
+          - meta-prod-aws-ue2
           - meta-prod-aws-ue1
           - lf-prod-aws-ue1
           - lf-prod-aws-ue2
@@ -77,12 +77,12 @@ jobs:
     needs: deploy_uw1
     if: |
       !cancelled() && (
-        inputs.target == 'arc-cbr-production' ||
+        inputs.target == 'meta-prod-aws-ue2' ||
         ((inputs.target == 'all' || inputs.target == 'all-meta') && needs.deploy_uw1.result == 'success')
       )
     uses: ./.github/workflows/_osdc-deploy.yml
     with:
-      cluster: arc-cbr-production
+      cluster: meta-prod-aws-ue2
       environment: osdc-production
       taint_nodes: ${{ inputs.taint_nodes }}
       restart_listeners: ${{ inputs.restart_listeners }}

.github/workflows/osdc-drain.yml

@@ -21,7 +21,7 @@ on:
           - meta-staging-aws-ue1
           - meta-staging-aws-ue2
           - meta-prod-aws-uw1
-          - arc-cbr-production
+          - meta-prod-aws-ue2
           - meta-prod-aws-ue1
           - lf-prod-aws-ue1
           - lf-prod-aws-ue2

.github/workflows/osdc-plan-prod.yml

@@ -22,7 +22,7 @@ jobs:
     if: github.event.pull_request.head.repo.full_name == github.repository
     uses: ./.github/workflows/_osdc-plan.yml
     with:
-      cluster: arc-cbr-production
+      cluster: meta-prod-aws-ue2
       pr_number: ${{ github.event.pull_request.number }}
     secrets: inherit
 

.github/workflows/osdc-undrain.yml

@@ -22,7 +22,7 @@ on:
           - meta-staging-aws-ue1
           - meta-staging-aws-ue2
           - meta-prod-aws-uw1
-          - arc-cbr-production
+          - meta-prod-aws-ue2
           - meta-prod-aws-ue1
           - lf-prod-aws-ue1
           - lf-prod-aws-ue2

grafana/AGENTS.md

@@ -33,7 +33,7 @@ mise run push --folder "..."
 
   | `runner_group_name` | `${cluster}` value |
   |---|---|
-  | `default`, `release-runners` | `pytorch-arc-cbr-production` |
+  | `meta-prod-aws-ue2` | `meta-prod-aws-ue2` |
   | `meta-prod-aws-uw1` | `meta-prod-aws-uw1` |
   | `meta-prod-aws-ue1` | `meta-prod-aws-ue1` |
   | `lf-prod-aws-ue1` | `lf-prod-aws-ue1` |

grafana/osdc.json

@@ -964,7 +964,7 @@
                       },
                       "pluginVersion": "4.17.0",
                       "queryType": "timeseries",
-                      "rawSql": "SELECT\n  toStartOfInterval(created_at, INTERVAL 5 MINUTE) AS time,\n  uniqExact(id) AS jobs\nFROM default.workflow_job\nWHERE match(repository_full_name, '^(${repository:regex})$')\n  AND arrayExists(label -> match(label, '^(mt|lf)-(rel-)?l-'), labels)\n  AND (\n    (runner_group_name IN ('default', 'release-runners') AND match('pytorch-arc-cbr-production', '^(${cluster:regex})$'))\n    OR (runner_group_name = 'meta-prod-aws-uw1' AND match('meta-prod-aws-uw1', '^(${cluster:regex})$'))\n    OR (runner_group_name = 'meta-prod-aws-ue1' AND match('meta-prod-aws-ue1', '^(${cluster:regex})$'))\n    OR (runner_group_name = 'lf-prod-aws-ue1' AND match('lf-prod-aws-ue1', '^(${cluster:regex})$'))\n    OR (runner_group_name = 'lf-prod-aws-ue2' AND match('lf-prod-aws-ue2', '^(${cluster:regex})$'))\n  )\n  AND $__timeFilter(created_at)\nGROUP BY time\nORDER BY time ASC"
+                      "rawSql": "SELECT\n  toStartOfInterval(created_at, INTERVAL 5 MINUTE) AS time,\n  uniqExact(id) AS jobs\nFROM default.workflow_job\nWHERE match(repository_full_name, '^(${repository:regex})$')\n  AND arrayExists(label -> match(label, '^(mt|lf)-(rel-)?l-'), labels)\n  AND (\n    (runner_group_name = 'meta-prod-aws-ue2' AND match('meta-prod-aws-ue2', '^(${cluster:regex})$'))\n    OR (runner_group_name = 'meta-prod-aws-uw1' AND match('meta-prod-aws-uw1', '^(${cluster:regex})$'))\n    OR (runner_group_name = 'meta-prod-aws-ue1' AND match('meta-prod-aws-ue1', '^(${cluster:regex})$'))\n    OR (runner_group_name = 'lf-prod-aws-ue1' AND match('lf-prod-aws-ue1', '^(${cluster:regex})$'))\n    OR (runner_group_name = 'lf-prod-aws-ue2' AND match('lf-prod-aws-ue2', '^(${cluster:regex})$'))\n  )\n  AND $__timeFilter(created_at)\nGROUP BY time\nORDER BY time ASC"
                     },
                     "version": "v0"
                   },
@@ -1215,7 +1215,7 @@
                       },
                       "pluginVersion": "4.17.0",
                       "queryType": "timeseries",
-                      "rawSql": "SELECT\n    bucket_time AS time,\n    dateDiff('second', j_created_at, bucket_time) / 60.0 AS queue_minutes\nFROM (\n    SELECT\n        argMax(created_at, started_at) AS j_created_at,\n        argMax(status, started_at) AS j_status,\n        argMax(runner_group_name, started_at) AS j_runner_group_name,\n        argMax(labels, started_at) AS j_labels,\n        max(_inserted_at) AS j_last_insert,\n        if(argMax(status, started_at) IN ('queued', 'waiting'), now64(9), argMax(started_at, started_at)) AS j_queue_end\n    FROM default.workflow_job\n    PREWHERE created_at <= $__toTime\n        AND created_at >= $__fromTime - INTERVAL 24 HOUR\n    WHERE arrayExists(x -> startsWith(x, 'mt-l-') OR startsWith(x, 'mt-rel-l-') OR startsWith(x, 'lf-l-') OR startsWith(x, 'lf-rel-l-'), labels)\n        AND match(repository_full_name, '^(${repository:regex})$')\n    GROUP BY id\n    HAVING NOT arrayExists(x -> position(x, 'h100') > 0 OR position(x, 'b200') > 0 OR position(x, 'gb200') > 0 OR position(x, 'a100') > 0, j_labels)\n        AND arrayExists(x -> match(x, '^(${runner_name_prefix:regex})$'), j_labels)\n        AND (\n          (j_runner_group_name IN ('default', 'release-runners') AND match('pytorch-arc-cbr-production', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'meta-prod-aws-uw1' AND match('meta-prod-aws-uw1', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'meta-prod-aws-ue1' AND match('meta-prod-aws-ue1', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'lf-prod-aws-ue1' AND match('lf-prod-aws-ue1', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'lf-prod-aws-ue2' AND match('lf-prod-aws-ue2', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = '' AND j_status IN ('queued', 'waiting') AND j_last_insert > now() - INTERVAL 3 HOUR)\n        )\n        AND j_queue_end >= $__fromTime\n) ARRAY JOIN\n    arrayDistinct(arrayConcat(\n        arrayFilter(\n            t -> t >= j_created_at AND t >= $__fromTime AND t <= $__toTime,\n            arrayMap(x -> toDateTime64(x, 9),\n                range(\n                    toUInt32(intDiv(toUInt32(greatest(j_created_at, $__fromTime)), 300) * 300),\n                    toUInt32(intDiv(toUInt32(least(j_queue_end, $__toTime)), 300) * 300) + 300,\n                    300\n                )\n            )\n        ),\n        if(j_queue_end >= $__fromTime AND j_queue_end <= $__toTime, [j_queue_end], [])\n    )) AS bucket_time\nWHERE bucket_time <= j_queue_end\nORDER BY bucket_time"
+                      "rawSql": "SELECT\n    bucket_time AS time,\n    dateDiff('second', j_created_at, bucket_time) / 60.0 AS queue_minutes\nFROM (\n    SELECT\n        argMax(created_at, started_at) AS j_created_at,\n        argMax(status, started_at) AS j_status,\n        argMax(runner_group_name, started_at) AS j_runner_group_name,\n        argMax(labels, started_at) AS j_labels,\n        max(_inserted_at) AS j_last_insert,\n        if(argMax(status, started_at) IN ('queued', 'waiting'), now64(9), argMax(started_at, started_at)) AS j_queue_end\n    FROM default.workflow_job\n    PREWHERE created_at <= $__toTime\n        AND created_at >= $__fromTime - INTERVAL 24 HOUR\n    WHERE arrayExists(x -> startsWith(x, 'mt-l-') OR startsWith(x, 'mt-rel-l-') OR startsWith(x, 'lf-l-') OR startsWith(x, 'lf-rel-l-'), labels)\n        AND match(repository_full_name, '^(${repository:regex})$')\n    GROUP BY id\n    HAVING NOT arrayExists(x -> position(x, 'h100') > 0 OR position(x, 'b200') > 0 OR position(x, 'gb200') > 0 OR position(x, 'a100') > 0, j_labels)\n        AND arrayExists(x -> match(x, '^(${runner_name_prefix:regex})$'), j_labels)\n        AND (\n          (j_runner_group_name = 'meta-prod-aws-ue2' AND match('meta-prod-aws-ue2', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'meta-prod-aws-uw1' AND match('meta-prod-aws-uw1', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'meta-prod-aws-ue1' AND match('meta-prod-aws-ue1', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'lf-prod-aws-ue1' AND match('lf-prod-aws-ue1', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'lf-prod-aws-ue2' AND match('lf-prod-aws-ue2', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = '' AND j_status IN ('queued', 'waiting') AND j_last_insert > now() - INTERVAL 3 HOUR)\n        )\n        AND j_queue_end >= $__fromTime\n) ARRAY JOIN\n    arrayDistinct(arrayConcat(\n        arrayFilter(\n            t -> t >= j_created_at AND t >= $__fromTime AND t <= $__toTime,\n            arrayMap(x -> toDateTime64(x, 9),\n                range(\n                    toUInt32(intDiv(toUInt32(greatest(j_created_at, $__fromTime)), 300) * 300),\n                    toUInt32(intDiv(toUInt32(least(j_queue_end, $__toTime)), 300) * 300) + 300,\n                    300\n                )\n            )\n        ),\n        if(j_queue_end >= $__fromTime AND j_queue_end <= $__toTime, [j_queue_end], [])\n    )) AS bucket_time\nWHERE bucket_time <= j_queue_end\nORDER BY bucket_time"
                     },
                     "version": "v0"
                   },
@@ -1342,7 +1342,7 @@
                       },
                       "pluginVersion": "4.17.0",
                       "queryType": "timeseries",
-                      "rawSql": "SELECT\n    bucket_time AS time,\n    dateDiff('second', j_created_at, bucket_time) / 60.0 AS queue_minutes\nFROM (\n    SELECT\n        argMax(created_at, started_at) AS j_created_at,\n        argMax(status, started_at) AS j_status,\n        argMax(runner_group_name, started_at) AS j_runner_group_name,\n        argMax(labels, started_at) AS j_labels,\n        max(_inserted_at) AS j_last_insert,\n        if(argMax(status, started_at) IN ('queued', 'waiting'), now64(9), argMax(started_at, started_at)) AS j_queue_end\n    FROM default.workflow_job\n    PREWHERE created_at <= $__toTime\n        AND created_at >= $__fromTime - INTERVAL 24 HOUR\n    WHERE arrayExists(x -> startsWith(x, 'mt-l-') OR startsWith(x, 'mt-rel-l-') OR startsWith(x, 'lf-l-') OR startsWith(x, 'lf-rel-l-'), labels)\n        AND match(repository_full_name, '^(${repository:regex})$')\n    GROUP BY id\n    HAVING arrayExists(x -> position(x, 'h100') > 0 OR position(x, 'b200') > 0 OR position(x, 'gb200') > 0 OR position(x, 'a100') > 0, j_labels)\n        AND arrayExists(x -> match(x, '^(${runner_name_prefix:regex})$'), j_labels)\n        AND (\n          (j_runner_group_name IN ('default', 'release-runners') AND match('pytorch-arc-cbr-production', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'meta-prod-aws-uw1' AND match('meta-prod-aws-uw1', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'meta-prod-aws-ue1' AND match('meta-prod-aws-ue1', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'lf-prod-aws-ue1' AND match('lf-prod-aws-ue1', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'lf-prod-aws-ue2' AND match('lf-prod-aws-ue2', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = '' AND j_status IN ('queued', 'waiting') AND j_last_insert > now() - INTERVAL 3 HOUR)\n        )\n        AND j_queue_end >= $__fromTime\n) ARRAY JOIN\n    arrayDistinct(arrayConcat(\n        arrayFilter(\n            t -> t >= j_created_at AND t >= $__fromTime AND t <= $__toTime,\n            arrayMap(x -> toDateTime64(x, 9),\n                range(\n                    toUInt32(intDiv(toUInt32(greatest(j_created_at, $__fromTime)), 300) * 300),\n                    toUInt32(intDiv(toUInt32(least(j_queue_end, $__toTime)), 300) * 300) + 300,\n                    300\n                )\n            )\n        ),\n        if(j_queue_end >= $__fromTime AND j_queue_end <= $__toTime, [j_queue_end], [])\n    )) AS bucket_time\nWHERE bucket_time <= j_queue_end\nORDER BY bucket_time"
+                      "rawSql": "SELECT\n    bucket_time AS time,\n    dateDiff('second', j_created_at, bucket_time) / 60.0 AS queue_minutes\nFROM (\n    SELECT\n        argMax(created_at, started_at) AS j_created_at,\n        argMax(status, started_at) AS j_status,\n        argMax(runner_group_name, started_at) AS j_runner_group_name,\n        argMax(labels, started_at) AS j_labels,\n        max(_inserted_at) AS j_last_insert,\n        if(argMax(status, started_at) IN ('queued', 'waiting'), now64(9), argMax(started_at, started_at)) AS j_queue_end\n    FROM default.workflow_job\n    PREWHERE created_at <= $__toTime\n        AND created_at >= $__fromTime - INTERVAL 24 HOUR\n    WHERE arrayExists(x -> startsWith(x, 'mt-l-') OR startsWith(x, 'mt-rel-l-') OR startsWith(x, 'lf-l-') OR startsWith(x, 'lf-rel-l-'), labels)\n        AND match(repository_full_name, '^(${repository:regex})$')\n    GROUP BY id\n    HAVING arrayExists(x -> position(x, 'h100') > 0 OR position(x, 'b200') > 0 OR position(x, 'gb200') > 0 OR position(x, 'a100') > 0, j_labels)\n        AND arrayExists(x -> match(x, '^(${runner_name_prefix:regex})$'), j_labels)\n        AND (\n          (j_runner_group_name = 'meta-prod-aws-ue2' AND match('meta-prod-aws-ue2', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'meta-prod-aws-uw1' AND match('meta-prod-aws-uw1', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'meta-prod-aws-ue1' AND match('meta-prod-aws-ue1', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'lf-prod-aws-ue1' AND match('lf-prod-aws-ue1', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = 'lf-prod-aws-ue2' AND match('lf-prod-aws-ue2', '^(${cluster:regex})$'))\n          OR (j_runner_group_name = '' AND j_status IN ('queued', 'waiting') AND j_last_insert > now() - INTERVAL 3 HOUR)\n        )\n        AND j_queue_end >= $__fromTime\n) ARRAY JOIN\n    arrayDistinct(arrayConcat(\n        arrayFilter(\n            t -> t >= j_created_at AND t >= $__fromTime AND t <= $__toTime,\n            arrayMap(x -> toDateTime64(x, 9),\n                range(\n                    toUInt32(intDiv(toUInt32(greatest(j_created_at, $__fromTime)), 300) * 300),\n                    toUInt32(intDiv(toUInt32(least(j_queue_end, $__toTime)), 300) * 300) + 300,\n                    300\n                )\n            )\n        ),\n        if(j_queue_end >= $__fromTime AND j_queue_end <= $__toTime, [j_queue_end], [])\n    )) AS bucket_time\nWHERE bucket_time <= j_queue_end\nORDER BY bucket_time"
                     },
                     "version": "v0"
                   },
@@ -1672,7 +1672,7 @@
             "$__all"
           ]
         },
-        "definition": "SELECT DISTINCT extract(runner_name, '^(.+)-[^-]+-runner-[^-]+$') AS prefix FROM default.workflow_job WHERE arrayExists(x -> startsWith(x, 'mt-l-') OR startsWith(x, 'mt-rel-l-') OR startsWith(x, 'lf-l-') OR startsWith(x, 'lf-rel-l-'), labels) AND started_at > now() - INTERVAL 7 DAY AND runner_group_name IN ('default', 'release-runners', 'meta-prod-aws-uw1', 'meta-prod-aws-ue1', 'lf-prod-aws-ue1', 'lf-prod-aws-ue2') AND extract(runner_name, '^(.+)-[^-]+-runner-[^-]+$') != '' ORDER BY prefix",
+        "definition": "SELECT DISTINCT extract(runner_name, '^(.+)-[^-]+-runner-[^-]+$') AS prefix FROM default.workflow_job WHERE arrayExists(x -> startsWith(x, 'mt-l-') OR startsWith(x, 'mt-rel-l-') OR startsWith(x, 'lf-l-') OR startsWith(x, 'lf-rel-l-'), labels) AND started_at > now() - INTERVAL 7 DAY AND runner_group_name IN ('meta-prod-aws-ue2', 'meta-prod-aws-uw1', 'meta-prod-aws-ue1', 'lf-prod-aws-ue1', 'lf-prod-aws-ue2') AND extract(runner_name, '^(.+)-[^-]+-runner-[^-]+$') != '' ORDER BY prefix",
         "hide": "dontHide",
         "includeAll": true,
         "label": "Runner name",
@@ -1689,7 +1689,7 @@
             "editorType": "sql",
             "format": 1,
             "queryType": "table",
-            "rawSql": "SELECT DISTINCT extract(runner_name, '^(.+)-[^-]+-runner-[^-]+$') AS prefix FROM default.workflow_job WHERE arrayExists(x -> startsWith(x, 'mt-l-') OR startsWith(x, 'mt-rel-l-') OR startsWith(x, 'lf-l-') OR startsWith(x, 'lf-rel-l-'), labels) AND started_at > now() - INTERVAL 7 DAY AND runner_group_name IN ('default', 'release-runners', 'meta-prod-aws-uw1', 'meta-prod-aws-ue1', 'lf-prod-aws-ue1', 'lf-prod-aws-ue2') AND extract(runner_name, '^(.+)-[^-]+-runner-[^-]+$') != '' ORDER BY prefix"
+            "rawSql": "SELECT DISTINCT extract(runner_name, '^(.+)-[^-]+-runner-[^-]+$') AS prefix FROM default.workflow_job WHERE arrayExists(x -> startsWith(x, 'mt-l-') OR startsWith(x, 'mt-rel-l-') OR startsWith(x, 'lf-l-') OR startsWith(x, 'lf-rel-l-'), labels) AND started_at > now() - INTERVAL 7 DAY AND runner_group_name IN ('meta-prod-aws-ue2', 'meta-prod-aws-uw1', 'meta-prod-aws-ue1', 'lf-prod-aws-ue1', 'lf-prod-aws-ue2') AND extract(runner_name, '^(.+)-[^-]+-runner-[^-]+$') != '' ORDER BY prefix"
           },
           "version": "v0"
         },

osdc/clusters.yaml

@@ -421,9 +421,11 @@ clusters:
       - monitoring
       - logging
 
-  arc-cbr-production:
-    cluster_name: pytorch-arc-cbr-production
+  meta-prod-aws-ue2:
+    cluster_name: meta-prod-aws-ue2
     region: us-east-2
+    # state_bucket kept on the legacy arc-cbr name to avoid an S3 migration
+    # (mirrors meta-prod-aws-ue1, which also kept its arc-cbr bucket).
     state_bucket: ciforge-tfstate-arc-cbr-prod
     access_config:
       authentication_mode: API_AND_CONFIG_MAP
@@ -448,8 +450,9 @@ clusters:
         arm64_fallback: 8
     arc-runners:
       github_config_url: "https://github.com/pytorch"
-      github_secret_name: pytorch-arc-cbr-production
+      github_secret_name: meta-prod-aws-ue2
       runner_name_prefix: "mt-"
+      runner_group: "meta-prod-aws-ue2"   # moved off the shared "default"/"release-runners" groups
     nodepools-h100:
       # us-east-2 H100 capacity reservation (p5.48xlarge, 8× H100).
       capacity_reservation_ids:
@@ -473,8 +476,6 @@ clusters:
       - arc-runners-h100
       - keda
       - buildkit
-      - pypi-cache
-      - cache-enforcer
       - zombie-cleanup
       - harbor-cache-recovery
       - monitoring