pytorch
diff --git a/‎.github/actions/cross-repo-ci-relay-callback/action.yml‎
Lines changed: 12 additions & 1 deletion b/‎.github/actions/cross-repo-ci-relay-callback/action.yml‎
Lines changed: 12 additions & 1 deletion
diff --git a/‎aws/lambda/cross_repo_ci_relay/README.md‎
Lines changed: 65 additions & 6 deletions b/‎aws/lambda/cross_repo_ci_relay/README.md‎
Lines changed: 65 additions & 6 deletions
diff --git a/‎aws/lambda/cross_repo_ci_relay/callback/callback_handler.py‎
Lines changed: 115 additions & 1 deletion b/‎aws/lambda/cross_repo_ci_relay/callback/callback_handler.py‎
Lines changed: 115 additions & 1 deletion
@@ -47,6 +47,17 @@ inputs:
       Maximum time in seconds to wait for the callback HTTP request to complete.
     required: false
     default: 10
+  job-name:
+    description: >
+      Identifier for this job, used to name the upstream check run
+      (crcr/<repo>/<workflow>/<job-name>). Defaults to github.job, the job's key
+      in the workflow file, which is fine for a normal single job. For a MATRIX
+      job every leg shares the same github.job, so you MUST pass a value that
+      includes the matrix values (for example a job-name of "build-cpu" /
+      "build-cuda" for a matrix over cpu/cuda) to give each leg its own check
+      run; otherwise the legs collide on the same name.
+    required: false
+    default: ${{ github.job }}
 
 runs:
   using: composite
@@ -73,7 +84,7 @@ runs:
         OIDC_TOKEN: ${{ steps.oidc.outputs.token }}
         CALLBACK_URL: ${{ inputs.callback-url }}
         ARTIFACT_URL: ${{ inputs.artifact-url }}
-        JOB_NAME: ${{ github.job }}
+        JOB_NAME: ${{ inputs.job-name }}
         CHECK_RUN_ID: ${{ job.check_run_id }}
         RUN_ID: ${{ github.run_id }}
         RUN_ATTEMPT: ${{ github.run_attempt }}
 
@@ -20,7 +20,10 @@ L1:
 L2:
   - org3/repo3
 L3:
-  - org4/repo4
+  device1:
+    org41/device1-repo: [oncall1, oncall2]
+  device2:
+    org42/device2-repo: [oncall3]
 L4:
   - org5/repo5: oncall1, oncall2
 ```
@@ -31,6 +34,22 @@ Each entry is either a plain `owner/repo` string or a `owner/repo: oncall1, onca
 
 The allowlist is cached in Redis under the key `crcr:allowlist_yaml` with a TTL controlled by `ALLOWLIST_TTL_SECONDS`. On a Redis error the function falls back to fetching directly from GitHub.
 
+### Repository levels
+
+Every level is dispatched to. Higher levels add capabilities on top, and each level includes everything the lower ones grant:
+
+| Level | Dispatch | Report results to HUD | Upstream check run on the PR |
+|---|---|---|---|
+| **L1** | ✅ | — | — |
+| **L2** | ✅ | ✅ | — |
+| **L3** | ✅ | ✅ | ✅ — only when the PR carries the matching `ciflow/crcr/<device>` label |
+| **L4** | ✅ | ✅ | ✅ — always |
+
+- **L3** entries are grouped by *device*. A repo registered under `device1` gets an upstream check run only when the PR has the `ciflow/crcr/device1` label, letting maintainers opt specific PRs into a backend's CI. The `[oncall, ...]` list is the oncalls associated with that repo.
+- **L4** repos always get an upstream check run, with no label required.
+
+The dispatch/HUD-reporting path (L1/L2) is covered below; the upstream check run path (L3/L4) is covered in [Upstream Check Runs](#upstream-check-runs-l3-and-l4).
+
 ## Reporting Results from Downstream CI
 
 L2+ downstream repositories can report the status of their CI workflows back to the relay server using the [`cross-repo-ci-relay-callback`](../../../.github/actions/cross-repo-ci-relay-callback/action.yml) composite action.
@@ -44,12 +63,12 @@ The callback endpoint validates incoming callbacks and forwards them to HUD for
 - **Identity**: the `Authorization: Bearer <oidc-token>` header is verified against GitHub's JWKS.  The OIDC `repository` claim is a trusted identity for the caller and is used for the L2+ allowlist check. Relay forwards this trusted value to HUD as a top-level `verified_repo` field; HUD should prefer it over anything self-reported in `callback_payload`.
 - **Repo level**: Relay determines the downstream repository's allowlist level (L1–L4) and forwards it to HUD as `downstream_repo_level`. This authoritative level information is determined once by the relay, ensuring HUD doesn't need to recompute it and avoiding synchronization/timing issues if tiering information becomes dynamic.
 - **Schema validation**: Relay validates that required fields (`delivery_id` and `workflow.status`) are present in the callback body.  Missing fields result in a `400` error to signal contract violations to the caller.  HUD receives validated data and does not need to perform schema checks.
-- **State machine**: Relay maintains a **unified state machine** in Redis to validate callback lifecycles, compute timing metrics, and support per-workflow tracking:
-  - **Unified structure**: Single enum `CallbackState` with states `DISPATCHED` (webhook side, keyed by sentinel `run_id=0, run_attempt=0`), `IN_PROGRESS`, and `COMPLETED` (callback side, per-workflow). State records stored as JSON: `{"state": "...", "timestamp": 1234.56}`.
+- **State machine**: Relay maintains a **unified state machine** in Redis to validate callback lifecycles, compute timing metrics, and support per-job tracking:
+  - **Unified structure**: Single enum `CallbackState` with states `DISPATCHED` (webhook side, keyed by sentinel `run_id=0, run_attempt=0`), `IN_PROGRESS`, and `COMPLETED` (callback side, per-job). State records stored as JSON: `{"state": "...", "timestamp": 1234.56}`.
   - **Dispatch validation**: `DISPATCHED` state proves valid webhook origin. Callbacks without this state are rejected (no prior dispatch).
-  - **Workflow-level tracking**: Each workflow has independent state and timestamps keyed by `{run_id}:{run_attempt}` (`crcr:state:{delivery_id}:{repo}:{run_id}:{run_attempt}`). Supports multiple workflows per webhook.
+  - **Job-level tracking**: Each job has independent state and timestamps keyed by `{run_id}:{run_attempt}:{job_name}` (`crcr:state:{delivery_id}:{repo}:{run_id}:{run_attempt}:{job_name}`). Supports multiple jobs (and workflows) per webhook; the `job_name` suffix keeps concurrent jobs of one run from colliding.
   - **Timing metrics**: `queue_time = dispatch_timestamp → in_progress_timestamp`, `execution_time = in_progress_timestamp → completed_timestamp`. Timestamps extracted from state records.
-  - **State transitions**: Rejects invalid flows (`COMPLETED` without prior `IN_PROGRESS`, duplicate `IN_PROGRESS` for the same `{run_id}:{run_attempt}`, duplicate `COMPLETED`, callbacks without a prior `DISPATCHED` record).
+  - **State transitions**: Rejects invalid flows (`COMPLETED` without prior `IN_PROGRESS`, duplicate `IN_PROGRESS` for the same `{run_id}:{run_attempt}:{job_name}`, duplicate `COMPLETED`, callbacks without a prior `DISPATCHED` record).
     Note that the direction graph below is for a single check run, reruns have different `run_attempt` and are treated as separate workflows, so they won't violate the state machine since they won't have a prior `IN_PROGRESS` or `COMPLETED` record.
     ```mermaid
     stateDiagram-v2
@@ -87,7 +106,10 @@ The HUD request looks like (two top-level namespaces: `trusted` and `untrusted`)
         "conclusion": "success",
         "name": "CI",
         "url": "https://github.com/org/repo/actions/runs/123",
-        "job_name": "my-ci-job",
+        "run_id": "123",        // stable across re-runs of the same run
+        "run_attempt": "2",     // increments each re-run; (run_id, run_attempt) distinguishes attempts
+        "job_name": "build-cuda",  // from the action's job-name input (defaults to github.job)
+        "check_run_id": "456",  // unique per attempt
         "started_at": "2026-05-04T20:48:28Z", // when status == in_progress, else None
         "completed_at": "2026-05-04T21:23:45Z", // when status == completed, else None
         "test_results": { "passed": 42, "failed": 3, "skipped": 5 },
@@ -133,6 +155,8 @@ All three attacks are **scoped to the attacker's own OIDC-authenticated repo ide
 
 - The downstream repository must be listed at level **L2 or higher** in the allowlist.
 - The **calling job** must declare `permissions: id-token: write` so that the action can mint a GitHub OIDC token for authentication.
+- The upstream check run is named after the calling job. By default it uses `github.job` (the job's workflow-file key), which is fine for a normal single job. A **matrix** job shares one `github.job` across all legs, so pass the `job-name` input with the matrix values (e.g. `job-name: build-${{ matrix.config }}`) to give each leg its own check run; otherwise the legs collide on the same name.
+- Each downstream **job** that should surface as its own upstream check run must invoke the action itself (one `in_progress`, one `completed`), so multi-job / matrix workflows report per job with no extra configuration.
 
 ### Usage
 
@@ -175,6 +199,41 @@ jobs:
 | `callback-url` | **yes** | — | Callback endpoint URL (production Lambda URL; set once at the workflow level) |
 | `artifact-url` | no | `''` | URL to downstream-hosted artifacts (logs, reports, results) |
 
+## Upstream Check Runs (L3 and L4)
+
+For L3 and L4 repositories the relay creates a **check run on the upstream PR** that mirrors the downstream workflow's status, so the upstream sees downstream CI as a normal PR check. This is built on top of the same callback used for HUD reporting (L2+), so an L3/L4 repo still reports results to HUD exactly as described above.
+
+### How it works
+
+Check runs are **per job**, named `crcr/<downstream_repo>/<workflow_name>/<job_name>`, and created from the **callback**, not at dispatch time. `job_name` comes from the action's `job-name` input, which defaults to `github.job`. Because every leg of a matrix shares one `github.job`, a matrix workflow must pass a distinct `job-name` per leg (e.g. `build-${{ matrix.config }}`) so the legs get separate check runs instead of colliding. Each downstream job reports its own callbacks (one `in_progress`, one `completed`), so a multi-job or matrix workflow surfaces one check run per job on the upstream PR instead of a single collapsed one:
+
+- On an `in_progress` callback the relay creates an in-progress check run linking to the downstream run.
+- On a `completed` callback it creates a completed check run carrying the downstream `conclusion`.
+
+The relay always **creates** a new check run rather than editing an existing one. GitHub only surfaces the latest check run of a given name on a commit, so each new one naturally supersedes the previous. Scoping the name by `job_name` keeps distinct jobs from overwriting each other while an in_progress → completed pair for the *same* job still self-supersedes. This keeps the logic stateless and makes reruns and reopens self-correcting.
+
+### L3 label timing
+
+For L3 the upstream check run is gated on the `ciflow/crcr/<device>` label, which a maintainer can add at any point relative to the workflow. The relay covers all three orderings:
+
+1. **Label present before dispatch** — the callback sees the label and creates the check run directly.
+2. **Label added while the workflow is running** — the `pull_request.labeled` handler reads the cached downstream job state (`crcr:dispatch_job:<head_sha>:<repo>`, a Redis hash whose fields are `<workflow_name>:<job_name>`) and backfills an in-progress check run for **every** job that has already reported.
+3. **Label added after the workflow finished** — the `labeled` handler creates a completed check run for each cached job directly from that state.
+
+Because the downstream echoes back the *dispatch-time* payload — whose labels can be stale, e.g. on **reopen** where no fresh `labeled` event fires — the relay records a per-commit "check run wanted" flag (`crcr:check_run_wanted:<head_sha>:<repo>`) at dispatch time (when the label is already present) and in the `labeled` handler. The callback consults this flag so it still creates the check run when the echoed labels don't reflect the PR's current state.
+
+### Re-running checks
+
+A developer can re-run downstream CI directly from the upstream PR's checks UI. Re-runs happen at the **workflow-run** level via `rerun-failed-jobs`: the downstream `run_id` is stored as each check run's `external_id`, and GitHub rejects re-running individual jobs of a run that is already running, so one run-level call re-runs all failed jobs together without conflict. The GitHub App subscribes to the `check_run` and `check_suite` events, and the relay handles their `rerequested` action:
+
+- **Re-run a single check** (`check_run` `rerequested`) — the downstream repo is parsed from the check run name (`crcr/<owner>/<repo>/<workflow_name>/<job_name>`) and the `run_id` from its `external_id`. The relay verifies the repo is L3+ and re-runs the failed jobs of that run (`POST /repos/<repo>/actions/runs/<run_id>/rerun-failed-jobs`). A 403 "run already running" is treated as a benign no-op.
+- **Re-run all checks** (`check_suite` `rerequested`) — the CRCR app owns a single check suite per commit, so the relay lists every check run in that suite, dedupes them by `(repo, run_id)`, and re-runs the failed jobs of each distinct L3+ run. Check runs with no `external_id` are skipped.
+
+Because the re-run carries the **original `delivery_id`** but a **new `check_run_id`** (GitHub mints fresh check runs per attempt), the two CI timing metrics behave differently:
+
+- **`execution_time`** (in_progress → completed) is correct as-is — the new `check_run_id` gets its own fresh state records.
+- **`queue_time`** (dispatch → in_progress) would otherwise be measured against the *original* dispatch timestamp (possibly days old). Thus, using `run_attempt` keyword in payload to identify whether it is a trustworthy value.
+
 ## Build, Deploy, and Test
 
 ### Deployment layout
 
@@ -5,6 +5,7 @@
 
 import utils.redis_helper as redis_helper
 from redis.exceptions import RedisError
+from utils import gh_helper
 from utils.allowlist import AllowlistLevel, AllowlistMap, load_allowlist
 from utils.config import RelayConfig
 from utils.hud import forward_to_hud
@@ -13,6 +14,7 @@
     CallbackStateRecord,
     DISPATCH_RUN_ATTEMPT,
     DISPATCH_RUN_ID,
+    extract_pr_labels,
     HTTPException,
 )
 from utils.redis_helper import check_rate_limit
@@ -107,6 +109,10 @@ def _update_state_and_compute_metrics(
 
     Both metrics default to None when the required prior state is unavailable
     (e.g. Redis cache miss or rerun without matching prior record).
+
+    For a re-run, ``queue_time`` is measured against the original dispatch (the
+    re-run reuses its delivery_id), so it is not a meaningful queue interval —
+    HUD distinguishes re-runs via ``workflow.run_attempt`` in the forwarded body.
     """
     if status not in ("in_progress", "completed"):
         raise HTTPException(400, f"unknown callback status: {status!r}")
@@ -168,6 +174,66 @@ def _update_state_and_compute_metrics(
     return ci_metrics
 
 
+def _create_upstream_check_run(
+    *,
+    config: RelayConfig,
+    verified_repo: str,
+    delivery_id: str,
+    status: str,
+    conclusion: str | None,
+    run_id: int,
+    head_sha: str,
+    workflow_name: str,
+    job_name: str | None,
+    details_url: str,
+) -> None:
+    """Create a new upstream check run mirroring the downstream job's status.
+
+    Called for L3+ repos (only when a check run is wanted and head_sha is known)
+    before HUD forwarding, so a HUD error cannot block the upstream PR check.
+
+    Every callback always *creates* a new check run (never edits an existing
+    one): GitHub only surfaces the latest check run of a given name on a commit,
+    so each new one supersedes the previous, keeping the logic stateless.
+    Best-effort: a GitHub failure must not fail the callback.
+    """
+    output = gh_helper.build_check_run_output(
+        status, conclusion, details_url, verified_repo
+    )
+    try:
+        upstream_token = gh_helper.get_repo_access_token(
+            config.github_app_id,
+            config.github_app_private_key,
+            config.upstream_repo,
+        )
+        cr_id = gh_helper.create_check_run(
+            token=upstream_token,
+            repo_full_name=config.upstream_repo,
+            name=gh_helper.check_run_name(verified_repo, workflow_name, job_name),
+            head_sha=head_sha,
+            status=status,
+            conclusion=conclusion,
+            details_url=details_url,
+            # Store the downstream run_id so a check-run rerequest can re-run
+            # the failed jobs of that workflow run.
+            external_id=str(run_id),
+            output=output,
+        )
+        logger.info(
+            "upstream check run created delivery_id=%s repo=%s status=%s cr_id=%s",
+            delivery_id,
+            verified_repo,
+            status,
+            cr_id,
+        )
+    except Exception:
+        logger.exception(
+            "failed to create upstream check run delivery_id=%s repo=%s",
+            delivery_id,
+            verified_repo,
+        )
+
+
 def handle(config: RelayConfig, body: dict, verified_repo: str) -> dict:
     """Forward a downstream callback to HUD.
 
@@ -188,7 +254,7 @@ def handle(config: RelayConfig, body: dict, verified_repo: str) -> dict:
     result = _verify_access(config, verified_repo)
     if result is None:
         return {"ok": True, "status": "ignored"}
-    _, repo_level = result
+    allowlist, repo_level = result
 
     delivery_id, status, run_id, run_attempt, workflow_name, job_name = (
         _parse_callback_body(body)
@@ -237,6 +303,54 @@ def handle(config: RelayConfig, body: dict, verified_repo: str) -> dict:
         job_name=job_name,
     )
 
+    # L3+: cache the job and create its upstream check run. Runs after state
+    # validation (above) but before HUD forwarding, so a HUD error cannot block
+    # the PR check. Without a head_sha there is neither a check run to create nor
+    # a cache entry the label handler could ever look up.
+    if repo_level.value >= AllowlistLevel.L3.value:
+        pr_field = (body.get("payload") or {}).get("pull_request") or {}
+        head_sha = (pr_field.get("head") or {}).get("sha", "")
+        if head_sha:
+            conclusion = (body.get("workflow") or {}).get("conclusion")
+            details_url = f"https://github.com/{verified_repo}/actions/runs/{run_id}"
+
+            # Always cache the job so a later label event can backfill its check
+            # run, even when we are not creating one now.
+            redis_helper.set_dispatch_job(
+                config,
+                head_sha,
+                verified_repo,
+                status,
+                conclusion,
+                details_url,
+                run_id=str(run_id),
+                workflow_name=workflow_name,
+                job_name=job_name,
+            )
+
+            needs_cr = allowlist.needs_check_run(verified_repo, extract_pr_labels(body))
+            if not needs_cr and repo_level == AllowlistLevel.L3:
+                # The downstream's echoed payload labels may not reflect the PR's
+                # current state (e.g. on reopen). Fall back to the per-commit flag
+                # recorded at dispatch / label time for this (head_sha, repo).
+                needs_cr = redis_helper.is_check_run_wanted(
+                    config, head_sha, verified_repo
+                )
+
+            if needs_cr:
+                _create_upstream_check_run(
+                    config=config,
+                    verified_repo=verified_repo,
+                    delivery_id=delivery_id,
+                    status=status,
+                    conclusion=conclusion,
+                    run_id=run_id,
+                    head_sha=head_sha,
+                    workflow_name=workflow_name,
+                    job_name=job_name,
+                    details_url=details_url,
+                )
+
     if status == "in_progress":
         redis_helper.add_in_progress_tracker(
             config, delivery_id, verified_repo, run_id, run_attempt, job_name=job_name