Skip to content

Commit e1e53d7

Browse files
committed
Update (base update)
[ghstack-poisoned]
1 parent 936e72e commit e1e53d7

20 files changed

Lines changed: 1795 additions & 74 deletions
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
name: Test and deploy gha-log-uploader
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
paths:
8+
- ".github/workflows/gha-log-uploader-lambda.yml"
9+
- "aws/lambda/gha-log-uploader/**"
10+
- "aws/lambda/call-log-classifier/**"
11+
pull_request:
12+
paths:
13+
- ".github/workflows/gha-log-uploader-lambda.yml"
14+
- "aws/lambda/gha-log-uploader/**"
15+
- "aws/lambda/call-log-classifier/**"
16+
17+
jobs:
18+
test:
19+
runs-on: ubuntu-latest
20+
steps:
21+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
22+
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
23+
with:
24+
python-version: '3.12'
25+
cache: pip
26+
- name: Test gha-log-uploader
27+
working-directory: aws/lambda/gha-log-uploader/
28+
run: |
29+
pip3 install -r requirements.txt pytest
30+
pytest -v test_lambda_function.py
31+
- name: Test call-log-classifier
32+
working-directory: aws/lambda/call-log-classifier/
33+
run: |
34+
pip3 install boto3 pytest
35+
pytest -v test_lambda_function.py
36+
37+
deploy:
38+
needs: test
39+
runs-on: ubuntu-latest
40+
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
41+
permissions:
42+
id-token: write
43+
contents: read
44+
steps:
45+
- name: configure aws credentials
46+
uses: aws-actions/configure-aws-credentials@67fbcbb121271f7775d2e7715933280b06314838 # v1.7.0
47+
with:
48+
role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_gha-log-uploader-lambda
49+
aws-region: us-east-1
50+
51+
- name: Check out test infra
52+
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
53+
54+
- name: Setup python
55+
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
56+
57+
- uses: nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 # v2.8.2
58+
name: Setup dependencies
59+
with:
60+
shell: bash
61+
timeout_minutes: 10
62+
max_attempts: 3
63+
retry_wait_seconds: 30
64+
command: |
65+
set -eux
66+
python3 -m pip install awscli
67+
68+
- name: Deploy gha-log-uploader
69+
working-directory: aws/lambda/gha-log-uploader/
70+
run: make deploy
71+
72+
- name: Deploy call-log-classifier
73+
working-directory: aws/lambda/call-log-classifier/
74+
run: make deploy
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
ZIP := deployment.zip
2+
FUNCTION := call-log-classifier
3+
4+
# boto3 ships in the Lambda python runtime and nothing else is imported, so the
5+
# package is just the handler.
6+
prepare: clean
7+
mkdir -p deployment
8+
cp lambda_function.py ./deployment/.
9+
cd ./deployment && zip -q -r ../$(ZIP) .
10+
11+
deploy: prepare
12+
aws lambda update-function-code --function-name $(FUNCTION) --zip-file fileb://$(ZIP)
13+
14+
clean:
15+
rm -rf deployment $(ZIP)
16+
17+
.PHONY: prepare deploy clean
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# call-log-classifier
2+
3+
Triggers log classification when a job log lands in
4+
`s3://ossci-raw-job-status/log/`. Wired to `s3:ObjectCreated:*` on that prefix, so
5+
whatever wrote the log — `gha-log-uploader`, torchci's backfill route, or a person
6+
running `aws s3 cp` — gets it classified.
7+
8+
This exists so uploading and classifying are decoupled. `github-status-test`
9+
called the classifier inline with an untimed `urlopen` and blocked until it
10+
finished, which is what produced its 274s/344s/400s/900s duration tails.
11+
12+
## Key shapes
13+
14+
| Key | Repo |
15+
| --- | --- |
16+
| `log/<job_id>` | `pytorch/pytorch` (unprefixed for historical reasons) |
17+
| `log/<owner>/<repo>/<job_id>` | `<owner>/<repo>` |
18+
19+
Anything else is skipped and logged. `log/<owner>/<job_id>` in particular cannot be
20+
attributed to a repo, so it is ignored rather than guessed at.
21+
22+
## Why it does not use the classifier's function URL
23+
24+
`log_classifier` has a Lambda function URL with `AuthType: NONE`. This function
25+
reaches it through `lambda:InvokeFunction` instead, so the path from S3 to
26+
classification never crosses a public endpoint.
27+
28+
`log_classifier` is built on `lambda_http` with only the `apigw_http` feature, so
29+
it expects an API Gateway HTTP API v2.0 request. `classifier_payload()` reproduces
30+
that shape. Verified against the deployed function: a payload with no `job_id`
31+
returns its 400 `no job id provided` branch, and one with a non-numeric `job_id`
32+
fails inside its `parse::<usize>()` — together showing that both the envelope and
33+
the query string are read from a direct invoke.
34+
35+
The invoke is asynchronous. Classification can take minutes and nothing here reads
36+
the result.
37+
38+
## Relationship to keep-going-call-log-classifier
39+
40+
`keep-going-call-log-classifier` does the same job for the `temp_logs/` prefix on
41+
`gha-artifacts`, and still calls the classifier over its public function URL. It
42+
is deliberately left alone during the `github-status-test` cutover. Folding the
43+
two together, and then removing the public function URL, is cleanup for
44+
afterwards.
45+
46+
## One-time AWS setup
47+
48+
1. Create the function: python3.12, handler `lambda_function.lambda_handler`. Only
49+
boto3 is imported, so the package is just the handler.
50+
2. Give its execution role `lambda:InvokeFunction` on
51+
`arn:aws:lambda:us-east-1:308535385114:function:log_classifier`, plus the usual
52+
CloudWatch Logs permissions.
53+
3. Let S3 invoke it:
54+
```
55+
aws lambda add-permission --function-name call-log-classifier \
56+
--statement-id s3-ossci-raw-job-status --action lambda:InvokeFunction \
57+
--principal s3.amazonaws.com \
58+
--source-arn arn:aws:s3:::ossci-raw-job-status \
59+
--source-account 308535385114
60+
```
61+
4. Add an `s3:ObjectCreated:*` notification on `ossci-raw-job-status` filtered to
62+
prefix `log/`. **Read the existing configuration and add to it**
63+
`put-bucket-notification-configuration` replaces the whole document, and that
64+
bucket already carries twelve `clickhouse-replicator-s3` rules that must
65+
survive. None of them overlap `log/`.
66+
67+
Do not enable the notification before `gha-log-uploader` is live, or every log the
68+
old lambda writes gets classified twice.
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
"""Kick off log classification when a job log lands in S3.
2+
3+
Wired to `s3:ObjectCreated:*` on the `log/` prefix of `ossci-raw-job-status`, so
4+
whatever put the log there -- gha-log-uploader, torchci's backfill route, or a
5+
person running `aws s3 cp` -- gets it classified. Decoupling this from the
6+
uploader is what keeps a slow classification from showing up as uploader latency.
7+
8+
`log_classifier` is reached through `lambda:InvokeFunction` rather than its public
9+
function URL. It is built on lambda_http with only the `apigw_http` feature, so it
10+
expects an API Gateway HTTP API v2.0 request; PAYLOAD_TEMPLATE reproduces that
11+
shape. Verified against the deployed function: a payload with no `job_id` returns
12+
its 400 "no job id provided" branch, and a non-numeric one fails in its
13+
`parse::<usize>()`, which together show both the envelope and the query string
14+
are read.
15+
16+
Sibling `keep-going-call-log-classifier` does the same job for the `temp_logs/`
17+
prefix on `gha-artifacts`. The two should be folded together once the
18+
github-status-test cutover is finished.
19+
"""
20+
21+
import json
22+
from typing import Any, Optional, Tuple
23+
24+
import boto3
25+
26+
27+
LOG_CLASSIFIER_FUNCTION = "log_classifier"
28+
DEFAULT_REPO = "pytorch/pytorch"
29+
LOG_PREFIX = "log/"
30+
31+
lambda_client = boto3.client("lambda")
32+
33+
34+
def parse_key(key: str) -> Optional[Tuple[str, int]]:
35+
"""Map an S3 key under `log/` to (repo, job_id), or None if it isn't one.
36+
37+
pytorch/pytorch is unprefixed for historical reasons, so `log/<id>` means
38+
pytorch/pytorch and `log/<owner>/<repo>/<id>` names its repo explicitly.
39+
"""
40+
if not key.startswith(LOG_PREFIX):
41+
return None
42+
43+
parts = key[len(LOG_PREFIX) :].split("/")
44+
if len(parts) == 1:
45+
repo = DEFAULT_REPO
46+
elif len(parts) == 3:
47+
repo = f"{parts[0]}/{parts[1]}"
48+
else:
49+
# Neither shape. Includes `log/<owner>/<id>` and anything deeper, which
50+
# we have no way to attribute to a repo.
51+
return None
52+
53+
try:
54+
return repo, int(parts[-1])
55+
except ValueError:
56+
return None
57+
58+
59+
def classifier_payload(repo: str, job_id: int) -> dict:
60+
"""An API Gateway HTTP API v2.0 request, which is what lambda_http parses."""
61+
query = {"job_id": str(job_id), "repo": repo}
62+
return {
63+
"version": "2.0",
64+
"routeKey": "$default",
65+
"rawPath": "/",
66+
"rawQueryString": f"job_id={job_id}&repo={repo}",
67+
"headers": {},
68+
"queryStringParameters": query,
69+
"requestContext": {
70+
"accountId": "308535385114",
71+
"apiId": "call-log-classifier",
72+
"domainName": "lambda-invoke",
73+
"domainPrefix": "lambda-invoke",
74+
"http": {
75+
"method": "GET",
76+
"path": "/",
77+
"protocol": "HTTP/1.1",
78+
"sourceIp": "127.0.0.1",
79+
"userAgent": "call-log-classifier",
80+
},
81+
"requestId": f"call-log-classifier-{job_id}",
82+
"routeKey": "$default",
83+
"stage": "$default",
84+
"time": "01/Jan/1970:00:00:00 +0000",
85+
"timeEpoch": 0,
86+
},
87+
"isBase64Encoded": False,
88+
}
89+
90+
91+
def lambda_handler(event: Any, context: Any) -> None:
92+
for record in event.get("Records", []):
93+
key = record.get("s3", {}).get("object", {}).get("key", "")
94+
95+
parsed = parse_key(key)
96+
if parsed is None:
97+
print(f"Skipping key that isn't a job log: key={key}")
98+
continue
99+
100+
repo, job_id = parsed
101+
try:
102+
# Async: classification can take minutes and nothing here reads the
103+
# result, so blocking on it would only burn this function's runtime.
104+
lambda_client.invoke(
105+
FunctionName=LOG_CLASSIFIER_FUNCTION,
106+
InvocationType="Event",
107+
Payload=json.dumps(classifier_payload(repo, job_id)).encode(),
108+
)
109+
except Exception as error:
110+
# One bad key must not strand the rest of the batch.
111+
print(
112+
f"Failed to call log classifier for job_id={job_id}, "
113+
f"repo={repo}, key={key}, error={error}"
114+
)
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
import json
2+
import unittest
3+
from unittest.mock import patch
4+
5+
from lambda_function import classifier_payload, lambda_handler, parse_key
6+
7+
8+
def s3_event(*keys: str) -> dict:
9+
return {
10+
"Records": [
11+
{
12+
"s3": {
13+
"bucket": {"name": "ossci-raw-job-status"},
14+
"object": {"key": key},
15+
}
16+
}
17+
for key in keys
18+
]
19+
}
20+
21+
22+
class TestParseKey(unittest.TestCase):
23+
def test_bare_id_is_pytorch_pytorch(self):
24+
self.assertEqual(parse_key("log/123345"), ("pytorch/pytorch", 123345))
25+
26+
def test_prefixed_key_names_its_repo(self):
27+
self.assertEqual(
28+
parse_key("log/pytorch/executorch/999"), ("pytorch/executorch", 999)
29+
)
30+
31+
def test_meta_pytorch_repo(self):
32+
self.assertEqual(
33+
parse_key("log/meta-pytorch/torchcomms/42"), ("meta-pytorch/torchcomms", 42)
34+
)
35+
36+
def test_ignores_other_prefixes(self):
37+
# The notification is filtered to `log/`, but `classification/` and
38+
# `logs_something/` must not be mistaken for it if that ever changes.
39+
self.assertIsNone(parse_key("classification/123"))
40+
self.assertIsNone(parse_key("log_archive/123"))
41+
42+
def test_ignores_a_non_numeric_id(self):
43+
self.assertIsNone(parse_key("log/not-a-number"))
44+
self.assertIsNone(parse_key("log/pytorch/executorch/not-a-number"))
45+
46+
def test_ignores_an_unattributable_depth(self):
47+
# `log/<owner>/<id>` gives no repo, and anything deeper is not ours.
48+
self.assertIsNone(parse_key("log/pytorch/123"))
49+
self.assertIsNone(parse_key("log/a/b/c/123"))
50+
51+
def test_ignores_a_directory_marker(self):
52+
self.assertIsNone(parse_key("log/"))
53+
54+
55+
class TestClassifierPayload(unittest.TestCase):
56+
def test_is_a_v2_request_the_classifier_can_parse(self):
57+
payload = classifier_payload("pytorch/executorch", 999)
58+
# log_classifier builds on lambda_http with only the apigw_http feature,
59+
# so version 2.0 and requestContext.http are what make it deserialize.
60+
self.assertEqual(payload["version"], "2.0")
61+
self.assertIn("http", payload["requestContext"])
62+
self.assertEqual(
63+
payload["queryStringParameters"],
64+
{"job_id": "999", "repo": "pytorch/executorch"},
65+
)
66+
self.assertEqual(
67+
payload["rawQueryString"], "job_id=999&repo=pytorch/executorch"
68+
)
69+
70+
def test_is_json_serializable(self):
71+
json.dumps(classifier_payload("pytorch/pytorch", 1))
72+
73+
74+
class TestLambdaHandler(unittest.TestCase):
75+
def test_invokes_the_classifier_asynchronously(self):
76+
with patch("lambda_function.lambda_client") as client:
77+
lambda_handler(s3_event("log/123345"), None)
78+
79+
kwargs = client.invoke.call_args.kwargs
80+
self.assertEqual(kwargs["FunctionName"], "log_classifier")
81+
# Event, not RequestResponse: nothing here reads the classification.
82+
self.assertEqual(kwargs["InvocationType"], "Event")
83+
self.assertEqual(
84+
json.loads(kwargs["Payload"])["queryStringParameters"],
85+
{"job_id": "123345", "repo": "pytorch/pytorch"},
86+
)
87+
88+
def test_handles_every_record_in_a_batch(self):
89+
with patch("lambda_function.lambda_client") as client:
90+
lambda_handler(s3_event("log/1", "log/pytorch/rl/2"), None)
91+
92+
self.assertEqual(client.invoke.call_count, 2)
93+
94+
def test_skips_a_key_it_cannot_attribute(self):
95+
with patch("lambda_function.lambda_client") as client:
96+
lambda_handler(s3_event("log/not-a-number"), None)
97+
98+
client.invoke.assert_not_called()
99+
100+
def test_one_failure_does_not_strand_the_batch(self):
101+
with patch("lambda_function.lambda_client") as client:
102+
client.invoke.side_effect = [RuntimeError("throttled"), None]
103+
lambda_handler(s3_event("log/1", "log/2"), None)
104+
105+
self.assertEqual(client.invoke.call_count, 2)
106+
107+
def test_an_empty_event_is_a_noop(self):
108+
with patch("lambda_function.lambda_client") as client:
109+
lambda_handler({}, None)
110+
111+
client.invoke.assert_not_called()
112+
113+
114+
if __name__ == "__main__":
115+
unittest.main()

0 commit comments

Comments
 (0)