Skip to content

Commit ae595e5

Browse files
huydhnhuydhn
committed
Add the gha-log-uploader lambda
**Impact:** none yet -- nothing invokes this function **Risk:** low ## What A new lambda that downloads a completed GitHub Actions job log and archives it to `s3://ossci-raw-job-status/log/[<repo>/]<job_id>`. It takes a direct invoke payload (`{repo, job_id, conclusion}`) and has no API Gateway integration and no function URL, so `lambda:InvokeFunction` is the only way in. The log-download logic is copied from `github-status-test`: App installation token minting, the PAT pool fallback, the per-repo cool-off cache, gzip, the S3 key scheme, and the classifier call over its function URL -- the last with a bounded wait for the reply, so a hung classifier cannot burn the whole function timeout and get the invocation replayed. Not copied: the raw event archive, the API Gateway event parsing, and the synthetic `backfill` action branch. ghstack-source-id: ed018ec Pull-Request: #8591
1 parent a6cd5e7 commit ae595e5

5 files changed

Lines changed: 872 additions & 0 deletions

File tree

‎aws/lambda/gha-log-uploader/Makefile‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
ZIP := gha-log-uploader-deployment.zip
2+
FUNCTION := gha-log-uploader
3+
# Must match the deployed runtime. The package contains version-specific
4+
# compiled wheels (cffi), so a package built for one python on a function
5+
# running another fails at import on every single invocation. `deploy` checks
6+
# this against the live function rather than trusting the two to stay in sync.
7+
PYTHON_VERSION := 3.12
8+
# Third-party modules lambda_function.py imports. The built zip is checked for
9+
# these before it can be deployed: a package missing a dependency fails at
10+
# import, which drops every log upload until someone notices.
11+
VENDORED := boto3 requests github
12+
13+
# The lambda runs on x86_64. cryptography ships compiled wheels, so pin the
14+
# target platform rather than inheriting whatever python the CI runner happens
15+
# to default to -- otherwise the zip gets wheels the runtime can't load.
16+
# Starts from clean so a stale packages/ or zip can't leak into the artifact.
17+
prepare: clean
18+
mkdir -p ./packages
19+
pip install --target ./packages \
20+
--platform manylinux2014_x86_64 --python-version $(PYTHON_VERSION) \
21+
--implementation cp --only-binary=:all: --no-compile \
22+
-r requirements.txt
23+
cd packages && zip -r ../$(ZIP) .
24+
zip -g $(ZIP) lambda_function.py
25+
$(MAKE) verify
26+
27+
verify:
28+
@for m in $(VENDORED); do \
29+
unzip -l $(ZIP) | grep -qE " $$m/__init__\.py$$" \
30+
|| { echo "ERROR: '$$m' missing from $(ZIP), refusing to deploy"; exit 1; }; \
31+
done
32+
@echo "verified: $(ZIP) contains $(VENDORED)"
33+
34+
# Refuse to publish a package built for a different python than the function
35+
# actually runs. Without this the two can drift silently and the first symptom
36+
# is Runtime.ImportModuleError on every invocation.
37+
check-runtime:
38+
@live=$$(aws lambda get-function-configuration --function-name $(FUNCTION) \
39+
--query Runtime --output text); \
40+
if [ "$$live" != "python$(PYTHON_VERSION)" ]; then \
41+
echo "ERROR: $(FUNCTION) runs $$live but this package targets python$(PYTHON_VERSION)."; \
42+
echo " Change the function runtime first, or set PYTHON_VERSION to $${live#python}."; \
43+
exit 1; \
44+
fi; \
45+
echo "runtime check: $(FUNCTION) runs $$live, package targets python$(PYTHON_VERSION)"
46+
47+
deploy: check-runtime prepare
48+
aws lambda update-function-code --function-name $(FUNCTION) --zip-file fileb://$(ZIP)
49+
50+
clean:
51+
rm -rf $(ZIP) packages

‎aws/lambda/gha-log-uploader/README.md‎

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
# gha-log-uploader
2+
3+
Downloads a completed GitHub Actions job log and archives it to
4+
`s3://ossci-raw-job-status/log/`. This is the log-download half of the old
5+
`github-status-test` lambda, moved behind the PyTorch bot so onboarding a repo to
6+
HUD no longer needs an admin to add a repo webhook. See
7+
https://github.com/pytorch/test-infra/issues/7549.
8+
9+
`github-status-test` still exists and is untouched. It is deleted after the
10+
cutover, not edited into this shape.
11+
12+
## How it is invoked
13+
14+
Only through `lambda:InvokeFunction`. **There is no API Gateway integration and no
15+
function URL, and neither should be added** — the function must not be reachable
16+
from the internet.
17+
18+
Two callers, both in torchci, both using `InvocationType: "Event"`:
19+
20+
- `lib/bot/logUploader.ts`, on a `workflow_job` webhook with `action == completed`.
21+
- `lib/jobUtils.ts`'s `backfillMissingLog`, when Dr.CI notices a log is missing.
22+
External callers reach the same path through the authenticated
23+
`POST /api/log-uploader/backfill` route.
24+
25+
Payload:
26+
27+
```json
28+
{ "repo": "pytorch/executorch", "job_id": 12345, "conclusion": "failure" }
29+
```
30+
31+
`conclusion` is optional. A malformed payload raises, which means Lambda retries
32+
twice and then DLQs it.
33+
34+
## What fails how
35+
36+
Callers invoke asynchronously, so *raising* is what reaches Lambda's retries and
37+
then the dead-letter queue; *returning* records the invocation as a success no
38+
matter what the return value says. Which failures do which:
39+
40+
| Failure | Behaviour |
41+
| --- | --- |
42+
| Network error reaching GitHub | raises — retried, then DLQ |
43+
| GitHub 5xx or 429 | raises — retried, then DLQ |
44+
| No usable credential | raises — retried, then DLQ |
45+
| Malformed payload | raises — retried, then DLQ |
46+
| GitHub 404 (log has aged out), 401, 403 | returns `stored: false`, no retry |
47+
| Classifier call fails or times out | returns `classified: false`, no retry |
48+
49+
The bottom two are deliberate. A log GitHub has already dropped does not come
50+
back on the third attempt, and re-running a whole download to retry a classifier
51+
handoff would re-fetch megabytes to redo something that takes milliseconds.
52+
`stored: false` is therefore not visible on the DLQ; if you want to alarm on it,
53+
match the terminal `ERROR <status> downloading log` line specifically. Do **not**
54+
alarm on `ERROR` generally: `installation_token` logs one every time it falls
55+
back to the PAT pool, which is a path that then usually succeeds.
56+
57+
## Classification
58+
59+
After a log is stored, `log_classifier` is called through its function URL —
60+
byte for byte the call `github-status-test` makes today.
61+
62+
Function URLs only support the `RequestResponse` invocation type, so there is no
63+
way to ask for fire-and-forget. `CLASSIFIER_TIMEOUT` gets close enough: after 30s
64+
this stops waiting for the reply. Disconnecting does not cancel the classifier —
65+
it runs to completion regardless — so nothing is lost by hanging up, and
66+
`github-status-test`'s 274s/344s/400s/900s duration maxima do not carry over.
67+
68+
That bound is load-bearing, not tidiness. `urlopen` with no `timeout` has none at
69+
all, so a connection that is accepted and never answered raises nothing and burns
70+
the entire function timeout. Since callers invoke asynchronously, Lambda counts
71+
that as a failure and replays the whole invocation twice more, re-downloading the
72+
same log each time and eventually DLQ-ing a job whose log was archived fine on
73+
the first attempt. `github-status-test` does hit its 900s ceiling, so this is an
74+
observed tail, not a theoretical one.
75+
76+
With the wait bounded, every step has an explicit ceiling — two 30s log fetches
77+
at most, then a 30s classifier call — so a **300s function timeout** is
78+
comfortable, rather than the 900s `github-status-test` needs.
79+
80+
The way out is `lambda:InvokeFunction` with `InvocationType: "Event"`, which
81+
needs `log_classifier` to accept a plain `{"job_id", "repo"}` payload — it
82+
currently only parses the API Gateway request its `lambda_http` handler expects.
83+
That change also lets its `AuthType: NONE` function URL be retired, once
84+
`backfillJobs.mjs`, `keep-going-call-log-classifier` and `github-status-test`
85+
move off it. Worth doing on its own, not as a rider on this migration.
86+
87+
A failed handoff is logged and reported as `classified: false`, not raised.
88+
Raising would make Lambda retry the whole function, re-downloading a
89+
multi-megabyte log from GitHub to retry something that takes milliseconds; the
90+
log itself is already safe in S3.
91+
92+
## What it does not do
93+
94+
It does not archive raw webhook payloads. Nothing read them —
95+
`clickhouse-replicator-s3` has no `SUPPORTED_PATHS` entry for `workflow_job/`,
96+
`workflow_run/`, or `full_workflow_*/`, and ClickHouse gets jobs from DynamoDB via
97+
`clickhouse-replicator-dynamo`.
98+
99+
## S3 key scheme
100+
101+
`log/<job_id>` for `pytorch/pytorch`, `log/<owner>/<repo>/<job_id>` for everything
102+
else. The asymmetry is historical but load-bearing: the `log_url` ALIAS in
103+
`clickhouse_db_schema/default.workflow_job/schema.sql` derives URLs from exactly
104+
this shape, so changing it silently breaks every log link in the HUD.
105+
106+
## GitHub credentials
107+
108+
Job logs are downloaded with a GitHub App installation token, falling back to the
109+
`GITHUB_TOKENS` PAT pool when the app is rate limited, rejected, or not installed
110+
on the repo.
111+
112+
| Env var | Required | Purpose |
113+
| --- | --- | --- |
114+
| `GITHUB_APP_ID` | no | Numeric app id used to mint installation tokens, e.g. `4550824` — the id of the `pytorch-bot-preview` app. Must be the number: it goes through `int()`, so an app slug fails |
115+
| `GITHUB_APP_PRIVATE_KEY` | no | The app's private key, base64-encoded PEM (same encoding torchci uses) |
116+
| `GITHUB_TOKENS` | yes | Comma-separated PAT pool, used as the fallback and when no app is configured |
117+
118+
With both app vars unset the function only uses `GITHUB_TOKENS`, so the app can be
119+
rolled back by clearing the env vars — no code change or redeploy needed.
120+
121+
Notes on the app path:
122+
123+
- Installation tokens last an hour and are cached per repo in module scope, so a
124+
warm invocation reuses one rather than minting a token per job.
125+
- The app's rate limit is per installation. `pytorch` is enterprise-owned, so its
126+
installation gets 15,000 requests/hour, independent of any other app's quota.
127+
Use a dedicated app rather than the shared `pytorch-bot` installation, whose
128+
quota Dr. CI and the HUD already draw on.
129+
- Repos outside the installation (e.g. `vllm-project/vllm`) resolve to no
130+
installation and go straight to the PAT pool; that negative result is cached
131+
briefly to avoid a lookup per job.
132+
- Downloading job logs is documented as needing `actions: read`. It currently
133+
works without it because pytorch repos are public, but the permission should be
134+
granted before any private repo is onboarded.
135+
136+
## One-time AWS setup
137+
138+
Not done by CI. Needed before the deploy workflow can run.
139+
140+
1. Create the function: python3.12, x86_64, handler `lambda_function.lambda_handler`,
141+
512 MB, **300s timeout** — every step is individually bounded, so this does
142+
not need `github-status-test`'s 900s. See Classification above.
143+
2. Give its execution role `s3:PutObject` on `arn:aws:s3:::ossci-raw-job-status/log/*`
144+
plus the usual CloudWatch Logs permissions. No `lambda:InvokeFunction` is
145+
needed while the classifier is reached over its function URL.
146+
3. Set the env vars above. Prefer fresh credentials over copying
147+
`github-status-test`'s, whose PATs sit in plaintext env vars and are due for
148+
rotation.
149+
4. Configure an on-failure destination or DLQ, and alarm on it. For a trunk-only
150+
job that is the only signal its log went missing — Dr.CI's self-heal only
151+
covers PR jobs. See "What fails how" for what does and does not land there.
152+
5. Add the invoke grant for torchci, and nothing else:
153+
```
154+
aws lambda add-permission --function-name gha-log-uploader \
155+
--statement-id torchci-invoke --action lambda:InvokeFunction \
156+
--principal arn:aws:iam::308535385114:user/pytorch_hud_bot
157+
```
158+
Confirm that user really is the principal behind torchci's
159+
`OUR_AWS_ACCESS_KEY_ID` before granting.
160+
6. Create the `gha_workflow_gha-log-uploader-lambda` IAM role the deploy workflow
161+
assumes, mirroring `gha_workflow_github-status-test-lambda`.
162+
7. Nothing to wire for classification: the classifier is called over its existing
163+
function URL, so there is no notification or extra permission to add.
164+
165+
## Deployment
166+
167+
`make deploy` publishes to `$LATEST` and is live immediately; the deploy job in
168+
`.github/workflows/gha-log-uploader-lambda.yml` runs it on every push to main that
169+
touches this directory. `make prepare` verifies the zip contains every vendored
170+
module and `make deploy` refuses to publish a package built for a different python
171+
than the function runs, but there is no staged rollout behind either.
172+
173+
`PYTHON_VERSION` in the Makefile must match the function's runtime. Changing one
174+
without the other breaks every invocation.

0 commit comments

Comments
 (0)