[CRCR] Move Buildkite repo mappings to ci_providers.yml (#8468)

## Summary

Moves Buildkite pipeline-to-repo identity mappings into
`ci_providers.yml` so adding new downstream repos only requires a config
change — no Lambda redeployment.

**Key security design decision**: OIDC trust anchors (issuer URLs, JWKS
endpoints) remain **compiled into** `jwt_helper.py`. They cannot be
overridden from the runtime-fetched config file. Only mutable
*authorization* data (which pipeline maps to which repo) is loaded from
config.

### What's in config (runtime-mutable)
- `providers.buildkite.repo_map` — pipeline-to-repo mappings with
optional `required_claims`

### What's compiled in (immutable without redeployment)
- GitHub issuer URL + JWKS endpoint
- Buildkite issuer URL + JWKS endpoint
- Repo extractor functions per provider

### Deploy ordering
The legacy flat `buildkite:` section is still supported as a fallback —
if the `providers` section has no `repo_map`, the loader falls back to
the flat section. This means old code + new config still works.

### Adding a new downstream repo
1. Add entry to `providers.buildkite.repo_map` in `ci_providers.yml`
2. No Lambda redeployment needed

### Adding a new CI provider
1. Add issuer/JWKS constants and `_extract_repo_<provider>` function in
`jwt_helper.py`
2. Deploy Lambda
3. Add `repo_map` entries to config

| File | What |
| --- | --- |
| `config/ci_providers.yml` | Buildkite repo_map only (issuer/JWKS
removed) |
| `utils/jwt_helper.py` | Trust anchors compiled in; config loads only
repo maps |
| `tests/test_jwt_helper.py` | Tests verify JWKS clients are not
expandable via config |

> **Depends on**: #8453

## Test plan

- All unit tests pass (including new `TestTrustAnchorsCompiledIn` class)
- CI passes (lintrunner, python-tests)
- Verify a rogue issuer in config does NOT create a JWKS client
- Verify backward compat with legacy flat config format

Author: subinz1

aws/lambda/cross_repo_ci_relay/config/ci_providers.yml

@@ -1,53 +1,45 @@
-# External CI Provider → GitHub Repo Mapping for CRCR OIDC
+# CI Provider Configuration for CRCR
 #
-# CI platforms that don't carry a native "repository" OIDC claim need an
-# explicit mapping from their pipeline identity to a GitHub owner/repo.
+# This file supplies MUTABLE authorization data only — specifically,
+# Buildkite pipeline-to-repo identity mappings.  Trust anchors (OIDC
+# issuer URLs and JWKS endpoints) are compiled into jwt_helper.py and
+# cannot be overridden from this file.
 #
 # IMPORTANT: Buildkite mappings use immutable organization_id/pipeline_id
 # (UUIDs) rather than slugs.  Slugs are renameable and a released slug
 # can be claimed by a different organization; IDs are permanent.
 # Find IDs via: buildkite-agent oidc request-token --audience <aud>
 #
-# SECURITY: By default, any build in a mapped pipeline can authenticate.
-# If a pipeline runs fork-PR builds, use required_claims to restrict
-# which builds are authorized (e.g. only builds on specific branches).
-# Pipelines that do NOT expose OIDC to fork builds are safe without
-# required_claims.
-#
 # The Lambda fetches this file at runtime (via CI_PROVIDERS_URL) and
-# caches it in Redis, so changes here take effect without redeployment.
-
-buildkite:
-  # Simple format (pipeline does not run fork builds):
-  #   organization_id/pipeline_id: owner/repo
-  #
-  # Constrained format (pipeline may run fork builds):
-  #   organization_id/pipeline_id:
-  #     repo: owner/repo
-  #     required_claims:
-  #       build_branch: [main, nightly]  # only these branches can auth
-  #       cluster_id: specific-cluster-uuid
+# caches it in Redis, so mapping changes take effect without redeployment.
 
-  # Example entries:
-  # 018e4f2a-1b2c-3d4e/018e5a6b-7c8d-9e0f: vllm-project/vllm
-  #
-  # 018e4f2a-1b2c-3d4e/018e9a0b-1c2d-3e4f:
-  #   repo: vllm-project/vllm
-  #   required_claims:
-  #     build_branch: [main, release]
+providers:
+  buildkite:
+    # Pipeline-to-repo mapping (Buildkite OIDC has no "repository" claim).
+    # Keys are immutable organization_id/pipeline_id UUIDs.
+    #
+    # Simple format:
+    #   org_id/pipeline_id: owner/repo
+    #
+    # Constrained format (for pipelines that run fork builds):
+    #   org_id/pipeline_id:
+    #     repo: owner/repo
+    #     required_claims:
+    #       build_branch: [main, nightly]
+    repo_map:
+      # vllm-project/vllm -- the "CI" pipeline in the vllm organization.
+      # Reports torch-nightly results to CRCR (vllm-project/vllm#51830).
+      # This pipeline builds fork PRs and any job in it can mint an OIDC
+      # token, so it is constrained to main: a fork PR must not be able
+      # to report as vllm-project/vllm.
+      018b5ec5-c236-4037-ba08-d8a5d29bcdae/018cdabc-d930-49f6-9085-634c4cb582ed:
+        repo: vllm-project/vllm
+        required_claims:
+          build_branch: [main]
 
-  # vllm-project/vllm -- the "CI" pipeline in the vllm organization.
-  # Reports torch-nightly results to CRCR (vllm-project/vllm#51830).
-  # This pipeline builds fork PRs and any job in it can mint an OIDC token, so
-  # it is constrained to main: a fork PR must not be able to report as
-  # vllm-project/vllm.
+# Legacy flat section kept for deploy-ordering safety (old code reads this).
+buildkite:
   018b5ec5-c236-4037-ba08-d8a5d29bcdae/018cdabc-d930-49f6-9085-634c4cb582ed:
     repo: vllm-project/vllm
     required_claims:
       build_branch: [main]
-
-# gitlab:
-#   group/project: owner/repo
-
-# jenkins:
-#   job-name: owner/repo

aws/lambda/cross_repo_ci_relay/tests/test_jwt_helper.py

@@ -2,6 +2,7 @@
 from unittest.mock import MagicMock, patch
 
 from utils.jwt_helper import (
+    _jwks_clients,
     AUDIENCE,
     BUILDKITE_ISSUER,
     BUILDKITE_REPO_MAP,
@@ -259,7 +260,7 @@ def tearDown(self):
         BUILDKITE_REPO_MAP.clear()
         BUILDKITE_REPO_MAP.update(self._orig_map)
 
-    def test_loads_valid_buildkite_entries(self):
+    def test_loads_valid_buildkite_entries_legacy(self):
         raw = {
             "buildkite": {
                 "org-id-1/pipe-id-1": "vllm-project/vllm",
@@ -275,7 +276,7 @@ def test_loads_valid_buildkite_entries(self):
             BUILDKITE_REPO_MAP[("org-id-2", "pipe-id-2")]["repo"], "acme/repo"
         )
 
-    def test_loads_constrained_entries(self):
+    def test_loads_constrained_entries_legacy(self):
         raw = {
             "buildkite": {
                 "org-id/pipe-id": {
@@ -293,6 +294,59 @@ def test_loads_constrained_entries(self):
         self.assertEqual(entry["required_claims"]["build_branch"], ["main", "nightly"])
         self.assertEqual(entry["required_claims"]["cluster_id"], ["cluster-uuid"])
 
+    def test_providers_section_loads_repo_map(self):
+        raw = {
+            "providers": {
+                "buildkite": {
+                    "repo_map": {
+                        "org-id/pipe-id": "vllm-project/vllm",
+                    },
+                },
+            }
+        }
+        load_ci_provider_mappings(raw)
+        self.assertEqual(
+            BUILDKITE_REPO_MAP[("org-id", "pipe-id")]["repo"], "vllm-project/vllm"
+        )
+
+    def test_providers_section_loads_constrained_repo_map(self):
+        raw = {
+            "providers": {
+                "buildkite": {
+                    "repo_map": {
+                        "org-id/pipe-id": {
+                            "repo": "myorg/myrepo",
+                            "required_claims": {"build_branch": ["main"]},
+                        },
+                    },
+                },
+            }
+        }
+        load_ci_provider_mappings(raw)
+        entry = BUILDKITE_REPO_MAP[("org-id", "pipe-id")]
+        self.assertEqual(entry["repo"], "myorg/myrepo")
+        self.assertEqual(entry["required_claims"]["build_branch"], ["main"])
+
+    def test_providers_ignores_issuer_jwks_fields(self):
+        """Config may still contain issuer/jwks_uri for documentation but they
+        are ignored — trust anchors are compiled in."""
+        raw = {
+            "providers": {
+                "buildkite": {
+                    "issuer": "https://evil.example.com",
+                    "jwks_uri": "https://evil.example.com/.well-known/jwks",
+                    "repo_map": {
+                        "org-id/pipe-id": "vllm-project/vllm",
+                    },
+                },
+            }
+        }
+        load_ci_provider_mappings(raw)
+        self.assertNotIn("https://evil.example.com", _jwks_clients)
+        self.assertEqual(
+            BUILDKITE_REPO_MAP[("org-id", "pipe-id")]["repo"], "vllm-project/vllm"
+        )
+
     def test_empty_config_clears_map(self):
         BUILDKITE_REPO_MAP[("old", "entry")] = {
             "repo": "old/repo",
@@ -317,6 +371,18 @@ def test_skips_invalid_entries(self):
         self.assertNotIn(("noslash", ""), BUILDKITE_REPO_MAP)
         self.assertEqual(BUILDKITE_REPO_MAP[("ok-id", "pipe-id")]["repo"], "ok/repo")
 
+    def test_legacy_fallback_when_providers_has_no_repo_map(self):
+        """If providers section exists but has no buildkite repo_map,
+        fall back to legacy flat buildkite section."""
+        raw = {
+            "providers": {"github": {}},
+            "buildkite": {"org-id/pipe-id": "legacy/repo"},
+        }
+        load_ci_provider_mappings(raw)
+        self.assertEqual(
+            BUILDKITE_REPO_MAP[("org-id", "pipe-id")]["repo"], "legacy/repo"
+        )
+
 
 class TestUnsupportedIssuer(unittest.TestCase):
     """Tests for tokens from unsupported issuers."""
@@ -342,5 +408,28 @@ def test_none_issuer_raises_401(self):
         self.assertEqual(ctx.exception.status_code, 401)
 
 
+class TestTrustAnchorsCompiledIn(unittest.TestCase):
+    """Verify that JWKS clients are hardcoded and not modifiable via config."""
+
+    def test_jwks_clients_contain_known_issuers(self):
+        self.assertIn(GITHUB_ISSUER, _jwks_clients)
+        self.assertIn(BUILDKITE_ISSUER, _jwks_clients)
+
+    def test_jwks_clients_not_expandable_via_config(self):
+        """Loading config with a rogue issuer must not add a JWKS client."""
+        fake_issuer = "https://rogue.example.com"
+        raw = {
+            "providers": {
+                "rogue": {
+                    "issuer": fake_issuer,
+                    "jwks_uri": f"{fake_issuer}/.well-known/jwks",
+                    "repo_map": {"org/pipe": "owner/repo"},
+                },
+            }
+        }
+        load_ci_provider_mappings(raw)
+        self.assertNotIn(fake_issuer, _jwks_clients)
+
+
 if __name__ == "__main__":
     unittest.main()