Update the architecture doc for the log upload path

## What
Rewrites the "Logs and log classifications" section to describe the new path
(bot handler -> gha-log-uploader -> S3 -> call-log-classifier -> log_classifier)
and notes that the raw webhook payload archive goes away with
`github-status-test`.

## Why
The old text described logs as arriving through "a manual connection with the
`pytorch/pytorch` repo", and carried a note saying the logic "could be moved to
the `hud.pytorch.org` webhooks endpoint [...] Add suo as a reviewer on a PR if
you want to add it!" -- which is what these commits do.


ghstack-source-id: 4d9a6ad
Pull-Request: #8597

Author: huydhn

torchci/docs/architecture.md

@@ -63,17 +63,24 @@ These are uploaded to S3 directly by the GitHub workflows in `pytorch/pytorch`.
 
 ### Logs and log classifications
 
-Logs are downloaded by the [`github-status-test`] AWS lambda, which receives
-webhooks through a manual connection with the `pytorch/pytorch` repo.
-`github-status-test` then pings another lambda, [`log-classifier`], which
-performs log classification (more detail in the [README]).
+The PyTorch bot's `workflow_job` handler (`lib/bot/logUploader.ts`) asynchronously
+invokes the [`gha-log-uploader`] lambda when a job completes. That lambda
+downloads the log from GitHub and puts it in the [`ossci-raw-job-status`] bucket
+under `log/`. An S3 `ObjectCreated` notification on that prefix then invokes
+`call-log-classifier`, which invokes [`log-classifier`] to do the classification
+(more detail in the [README]).
 
 [readme]: https://github.com/pytorch/test-infra/blob/main/aws/lambda/log-classifier/README.md
 
-The logic in `github-status-test` could be moved to the `hud.pytorch.org`
-webhooks endpoint, and modified so that all repos that `torchci` is installed
-on benefits from log downloads and classifications, but this is not currently
-implemented. Add @suo as a reviewer on a PR if you want to add it!
+Because this hangs off the App webhook rather than a per-repo one, every repo the
+bot is installed on gets log downloads and classifications without an admin
+configuring anything. Which repos are enabled is controlled by the
+`LOG_UPLOADER_REPOS` env var while the cutover from [`github-status-test`] is in
+progress; see https://github.com/pytorch/test-infra/issues/7549.
+
+Missing logs are re-requested through `backfillMissingLog` in `lib/jobUtils.ts`,
+which Dr.CI calls when it finds a failed job with no log. Callers outside HUD use
+the authenticated `POST /api/log-uploader/backfill` route.
 
 ### Test statistics
 
@@ -91,11 +98,17 @@ an example.
 
 ### Raw webhook payloads
 
-For archival purposes, we save all raw webhook payloads to S3. This is done
-through the [`github-status-test`] lambda (the same one that downloads logs).
-The raw webhook payloads are uploaded to the [`ossci-raw-job-status`] S3 bucket.
+The [`github-status-test`] lambda archives raw webhook payloads to the
+[`ossci-raw-job-status`] S3 bucket, under a prefix per event type. Nothing reads
+them: `clickhouse-replicator-s3` has no `SUPPORTED_PATHS` entry for
+`workflow_job/`, `workflow_run/`, or `full_workflow_*/`, and ClickHouse gets jobs
+from DynamoDB through `clickhouse-replicator-dynamo`.
+
+This archive goes away with the lambda. It is not reproduced in
+[`gha-log-uploader`].
 
 [`github-status-test`]: https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/github-status-test?tab=code
+[`gha-log-uploader`]: https://github.com/pytorch/test-infra/blob/main/aws/lambda/gha-log-uploader/README.md
 [`ossci-raw-job-status`]: https://s3.console.aws.amazon.com/s3/buckets/ossci-raw-job-status?region=us-east-1&tab=overview
 
 ## Adding a new repo to `torchci`