Skip to content

Commit 5f8dd50

Browse files
KarhouTamcan-gaa-houfffrog
authored
[CRCR] Initial implementation of L2 (#7967)
## Author - @KarhouTam - @can-gaa-hou - @fffrog ## Summary - This PR implements the L2 levels of the cross-repository CI relay described in pytorch/rfcs#90. - For the previous L1 implementation, please refer to #7847. - Please refer to pytorch/rfcs#90 (comment) for the overall implementation. - Please refer to pytorch/rfcs#96 for the design of HUD side. - Please refer to #8069 for the implementation of HUD side. Higher-level behaviors for `L3` and `L4` are intentionally left for follow-up work. ## Architecture The relay is split into two AWS Lambda functions: - `webhook` lambda function (Updated) - [x] receives GitHub webhook PR and push events from the upstream repo - [x] validates webhook signatures and authenticates with AWS Secret Manager - [x] reads the downstream whitelist from the URL and stores it in Redis - [x] for `opened`/`reopened`/`synchronized`/`closed` actions, forwards repository_dispatch events to downstream repos - `callback` lambda function (Added) - [x] receives downstream callback payload through a public lambda function URL - [x] validates callback payload with OIDC - [x] reads the downstream whitelist from the URL and stores it in Redis - [x] extracts CI result information from the payload and uploads to PyTorch HUD - [x] records `queue time` and `execute time` for evolution to `L3` repo ## Changes ```md ..github/ ├── workflows/ │ └── _lambda-do-release-runners.yml # Updates the Lambda release workflow to include cross-repo-ci-relay packaging/release │ └── actions/ └── cross-repo-ci-relay-callback/ └── action.yml # Composite action used by downstream workflows to report status back to the relay/result endpoint aws/lambda/cross_repo_ci_relay/ ├── tests/ # Unit tests for allowlist/config/webhook/result/redis behavior ├── README.md # Project overview, local development, callback flow, and result-side validation steps ├── Makefile # Top-level local developer entrypoint for test / deploy / clean ├── local_server.py # FastAPI wrapper for local end-to-end testing of both webhook and result endpoints ├── requirements.txt # Python dependencies required by the relay Lambdas │ ├── utils/ │ ├── allowlist.py # Loads, parses, and queries the downstream allowlist by rollout level │ ├── config.py # Shared runtime config loading and cached get_config() helper │ ├── gh_helper.py # GitHub App, repository_dispatch, and GitHub file access helpers │ ├── hud.py # HUD write helpers for downstream result reporting │ ├── jwt_helper.py # Helpers for minting/verifying relay callback tokens │ ├── redis_helper.py # Redis helpers for allowlist cache, OOT state, and timing data │ └── misc.py # Shared TypedDict definitions and HTTPException │ ├── webhook/ │ ├── Makefile # Build/package/deploy commands for the webhook Lambda │ ├── lambda_function.py # Webhook Lambda entrypoint: verifies GitHub webhook requests and routes events │ └── event_handler.py # Handles PR/push events, resolves allowlist targets, and dispatches to downstream repos │ └── callback/ ├── Makefile # Build/package/deploy commands for the result Lambda ├── lambda_function.py # Result Lambda entrypoint: verifies callback token and GitHub OIDC token └── callback_handler.py # Validates callback payloads, checks L2+ eligibility, stores state, and writes to HUD ``` ## Usage See [README.md](https://github.com/KarhouTam/test-infra/blob/crcr-L2/aws/lambda/cross_repo_ci_relay/README.md) for more details. ## Verification We performed the following scenario verification on our AWS Lambda instance: - [x] Test with Upstream PR create/reopen/synchronize and push events triggering webhook, then redispatching to the Downstream CI (different organization) workflow. - [x] Test with Downstream workflow send callback payload through the added action to the result lambda, then extract CI result information and send to PyTorch HUD. ## Terraform configuration - pytorch/ci-infra#446 ## Unit Tests - [x] Unit Tests (Mock) ## Security - **Callback payload carries full upstream webhook data back to HUD** — `action.yml` builds the callback body by mutating `github.event.client_payload` (which contains the entire original webhook payload: PR metadata, commits, author info) and adding `status`/`conclusion`/`workflow_name`/`workflow_url` on top. This full blob is forwarded verbatim by `hud.py` to HUD with no relay-side filtering. HUD receives both relay-trusted `verified_repo` and an unvalidated body — if HUD trusts self-reported fields inside the body over `verified_repo`, a manipulated dispatch payload could tamper with HUD records. - **Lambda callback URL is public and hardcoded** — The endpoint is hardcoded in `action.yml and exposed in a public action, making it trivially discoverable. OIDC verification blocks unauthorized HUD writes, but the endpoint has no rate limiting; request flooding can cause Lambda concurrency exhaustion or Redis connection saturation. - **Only OIDC is used for verification** — The callback lambda relies solely on GitHub OIDC token verification for authentication, without additional application-level secrets or signatures. If an attacker compromises a downstream repo's GitHub Actions permissions, they could forge authenticated requests to the callback endpoint. Besides, OIDC has its own limitations (e.g., token expiration, potential misconfigurations) that could lead to unauthorized access if not carefully managed. ## HUD Interaction - **Design Principle: Transparent Relay & Decoupling** The Relay Server acts as a **lightweight data passthrough layer**. It does not define or parse specific CI data formats; instead, it offloads data interpretation and validation to the HUD. This ensures complete decoupling between the relay infrastructure and business-specific data. - **Security & Risk Mitigation** The relay uses **OIDC authentication** to guarantee the authenticity of the data source (**Verified Repo**). Its core responsibility is to ensure the data originates from the claimed repository, while security filtering and content compliance are enforced at the HUD level. --------- Co-authored-by: can-gaa-hou <jiahaochen535@gmail.com> Co-authored-by: fffrog <ljw1101.vip@gmail.com>
1 parent a966a81 commit 5f8dd50

30 files changed

Lines changed: 2209 additions & 243 deletions

‎.github/actions/cross-repo-ci-relay-callback/action.yml‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
name: Cross-Repo CI Relay Callback
2+
3+
description: >
4+
Report the status of a downstream CI workflow back to the Cross-Repo CI
5+
Relay server. The job must have `id-token: write` permission so that a
6+
GitHub OIDC token can be minted and used to authenticate the callback.
7+
8+
This action is meant to run in a workflow triggered by a `repository_dispatch`
9+
event from the relay. It reads the dispatch payload (`github.event.client_payload`)
10+
and the ambient `github` context directly, so workflow authors only need to
11+
supply the relay URL, the status/conclusion, and optional structured test
12+
results.
13+
14+
inputs:
15+
status:
16+
description: >
17+
Workflow status to report. Must be either "in_progress" or "completed".
18+
required: true
19+
conclusion:
20+
description: >
21+
Conclusion of the workflow run. Required (and must be "success" or
22+
"failure") when status is "completed". Ignored when status is
23+
"in_progress".
24+
required: false
25+
default: ''
26+
test-results:
27+
description: >
28+
Optional JSON string with test result summary (counts: passed/failed/skipped).
29+
Note: This should be a summary only, not a full enumeration of all test cases.
30+
Full test results should be uploaded as artifacts and referenced via `artifact-url`.
31+
required: false
32+
default: ''
33+
callback-url:
34+
description: >
35+
Base URL of the result callback server.
36+
required: true
37+
artifact-url:
38+
description: >
39+
URL to downstream-hosted artifacts (logs, reports, results),
40+
any publicly accessible URL.
41+
required: false
42+
default: ''
43+
max-time:
44+
description: >
45+
Maximum time in seconds to wait for the callback HTTP request to complete.
46+
required: false
47+
default: 10
48+
max-retries:
49+
description: >
50+
Maximum number of retries for the callback HTTP request in case of failure.
51+
required: false
52+
default: 3
53+
retry-delay:
54+
description: >
55+
Delay in seconds between retries for the callback HTTP request.
56+
required: false
57+
default: 2
58+
59+
runs:
60+
using: composite
61+
steps:
62+
- name: Mint OIDC token
63+
id: oidc
64+
uses: actions/github-script@v7
65+
with:
66+
script: |
67+
const token = await core.getIDToken("pytorch-cross-repo-ci-relay");
68+
core.setSecret(token);
69+
core.setOutput('token', token);
70+
71+
- name: Send callback to relay server
72+
shell: bash
73+
env:
74+
SCHEMA_VERSION: 1
75+
STATUS: ${{ inputs.status }}
76+
CONCLUSION: ${{ inputs.conclusion }}
77+
WORKFLOW_NAME: ${{ github.workflow }}
78+
WORKFLOW_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
79+
TEST_RESULTS: ${{ inputs.test-results }}
80+
CLIENT_PAYLOAD: ${{ toJson(github.event.client_payload) }}
81+
OIDC_TOKEN: ${{ steps.oidc.outputs.token }}
82+
CALLBACK_URL: ${{ inputs.callback-url }}
83+
ARTIFACT_URL: ${{ inputs.artifact-url }}
84+
JOB_NAME: ${{ github.job }}
85+
CHECK_RUN_ID: ${{ job.check_run_id }}
86+
RUN_ID: ${{ github.run_id }}
87+
RUN_ATTEMPT: ${{ github.run_attempt }}
88+
MAX_TIME: ${{ inputs.max-time }}
89+
MAX_RETRIES: ${{ inputs.max-retries }}
90+
RETRY_DELAY: ${{ inputs.retry-delay }}
91+
run: |
92+
set -euo pipefail
93+
94+
PAYLOAD=$(python3 - <<'PYEOF'
95+
import json, os, sys
96+
from datetime import datetime, timezone
97+
98+
status = os.environ["STATUS"]
99+
if status not in ("in_progress", "completed"):
100+
sys.exit(f"::error::status must be 'in_progress' or 'completed', got {status!r}")
101+
102+
conclusion = os.environ.get("CONCLUSION", "").strip() or None
103+
if status == "completed" and conclusion not in ("success", "failure"):
104+
sys.exit("::error::conclusion must be 'success' or 'failure' when status is 'completed'")
105+
if status == "in_progress":
106+
conclusion = None
107+
108+
try:
109+
client_payload = json.loads(os.environ["CLIENT_PAYLOAD"])
110+
except json.JSONDecodeError as exc:
111+
sys.exit(f"::error::github.event.client_payload is not valid JSON: {exc}")
112+
113+
current_time = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
114+
115+
# In case check_run_id is not exist (edge case), replace it
116+
# with {run_id}-{run_attempt}, which is also unique for each job run.
117+
check_run_id = os.environ.get("CHECK_RUN_ID", "").strip()
118+
if not check_run_id:
119+
check_run_id = f"{os.environ['RUN_ID']}-{os.environ['RUN_ATTEMPT']}"
120+
121+
# Relay's original dispatch payload (event_type, delivery_id, payload) is
122+
# forwarded verbatim. Downstream-reported fields live in a sibling
123+
# `workflow` dict so the two sources stay clearly separated on the wire.
124+
workflow: dict = {
125+
"schema_version": str(os.environ["SCHEMA_VERSION"]),
126+
"status": status,
127+
"conclusion": conclusion,
128+
"name": os.environ["WORKFLOW_NAME"],
129+
"url": os.environ["WORKFLOW_URL"],
130+
"run_attempt": os.environ["RUN_ATTEMPT"],
131+
"job_name": os.environ["JOB_NAME"],
132+
"check_run_id": check_run_id,
133+
"run_id": str(os.environ["RUN_ID"]),
134+
"started_at": None if status == "completed" else current_time,
135+
"completed_at": None if status == "in_progress" else current_time,
136+
}
137+
138+
test_results = os.environ.get("TEST_RESULTS", "").strip()
139+
if test_results:
140+
try:
141+
workflow["test_results"] = json.loads(test_results)
142+
except json.JSONDecodeError as exc:
143+
sys.exit(f"::error::test-results input is not valid JSON: {exc}")
144+
145+
artifact_url = os.environ.get("ARTIFACT_URL", "").strip()
146+
if artifact_url:
147+
workflow["artifact_url"] = artifact_url
148+
149+
client_payload["workflow"] = workflow
150+
print(json.dumps(client_payload))
151+
PYEOF
152+
)
153+
154+
set +e
155+
HTTP_CODE=$(
156+
curl --silent --show-error --fail-with-body --output /tmp/relay_response.json \
157+
--write-out "%{http_code}" \
158+
-X POST \
159+
--max-time ${MAX_TIME} \
160+
--retry ${MAX_RETRIES} \
161+
--retry-delay ${RETRY_DELAY} \
162+
-H "Content-Type: application/json" \
163+
-H "Authorization: Bearer ${OIDC_TOKEN}" \
164+
--data "${PAYLOAD}" \
165+
"${CALLBACK_URL%/}"
166+
)
167+
CURL_EXIT_CODE=$?
168+
set -e
169+
170+
if [[ "${CURL_EXIT_CODE}" -ne 0 ]]; then
171+
echo "::error::Callback server returned HTTP ${HTTP_CODE}."
172+
if [[ -s /tmp/relay_response.json ]]; then
173+
echo "Relay server error response body:"
174+
cat /tmp/relay_response.json
175+
fi
176+
exit "${CURL_EXIT_CODE}"
177+
fi
178+
179+
echo "Relay server response HTTP: ${HTTP_CODE}"

‎.github/workflows/_lambda-do-release-runners.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,8 @@ jobs:
9393
{ dir-name: 'keep-going-call-log-classifier', zip-name: 'keep-going-call-log-classifier' },
9494
{ dir-name: 'buildkite-webhook-handler', zip-name: 'buildkite-webhook-handler' },
9595
{ dir-name: 'benchmark_regression_summary_report', zip-name: 'benchmark-regression-summary-report' },
96-
{ dir-name: 'cross_repo_ci_relay', zip-name: 'cross-repo-ci-webhook' },
96+
{ dir-name: 'cross_repo_ci_relay/callback', zip-name: 'cross-repo-ci-callback' },
97+
{ dir-name: 'cross_repo_ci_relay/webhook', zip-name: 'cross-repo-ci-webhook' },
9798
]
9899
name: Upload Release for ${{ matrix.dir-name }} lambda
99100
runs-on: ubuntu-latest

‎aws/lambda/cross_repo_ci_relay/Makefile‎

Lines changed: 18 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,24 @@
1-
SHARED := config.py utils.py redis_helper.py allowlist.py gh_helper.py event_handler.py
2-
PIP_FLAGS := --platform manylinux2014_x86_64 --only-binary=:all: --implementation cp --python-version 3.13
3-
AWS_REGION := us-east-1
4-
FUNCTION_NAME := cross_repo_ci_webhook
1+
AWS_REGION ?= us-east-1
2+
CALLBACK_FUNCTION_NAME ?= cross_repo_ci_callback
3+
WEBHOOK_FUNCTION_NAME ?= cross_repo_ci_webhook
54

6-
deployment.zip: clean
7-
mkdir -p ./deployment
8-
cp $(SHARED) lambda_function.py ./deployment/
9-
pip3 install --target ./deployment -r requirements.txt $(PIP_FLAGS)
10-
cd deployment && zip -r ../deployment.zip .
11-
12-
deploy: deployment.zip
13-
aws lambda update-function-code --region $(AWS_REGION) --function-name $(FUNCTION_NAME) --zip-file fileb://deployment.zip
5+
.PHONY: test deploy deploy-callback deploy-webhook clean
146

157
test:
168
python3 -m pytest tests -v
179

18-
clean:
19-
rm -rf deployment deployment.zip
10+
# Deploy both; keep going on failure so a broken half doesn't block the other.
11+
deploy:
12+
@rc=0; \
13+
for t in deploy-callback deploy-webhook; do $(MAKE) $$t || rc=$$?; done; \
14+
exit $$rc
15+
16+
deploy-callback:
17+
$(MAKE) -C callback deploy AWS_REGION=$(AWS_REGION) FUNCTION_NAME=$(CALLBACK_FUNCTION_NAME)
2018

21-
.PHONY: prepare deploy test clean
19+
deploy-webhook:
20+
$(MAKE) -C webhook deploy AWS_REGION=$(AWS_REGION) FUNCTION_NAME=$(WEBHOOK_FUNCTION_NAME)
21+
22+
clean:
23+
$(MAKE) -C callback clean
24+
$(MAKE) -C webhook clean

0 commit comments

Comments
 (0)