Skip to content

architecture.md

Latest commit

 

History

History
125 lines (92 loc) · 5.83 KB

architecture.md

File metadata and controls

125 lines (92 loc) · 5.83 KB
Raw

torchci architecture notes

Primary write path

How does data get from GitHub to torchci?

---
title: torchci write path
---
flowchart LR
   pytorchbot[Github App
   PyTorchBot] --> vercel@{ shape: das, label: "vercel
   /api/github/webhooks" }
   vercel --> dynamodb[DynamoDB
   torchci-* tables]
   dynamodb --> replicatorlambda@{ shape: das, label: "AWS Lambda
   clickhouse-replicator-dynamo"}
   replicatorlambda --> ClickHouse
Loading

Whenever something happens on GitHub, a webhook event is created and sent to all subscribers. The webhook payload will contain all the necessary information about the event. For example, if a new issue is created, an issues webhook is generated, providing information about the issue title, who created the issue, etc.

GitHub Apps can subscribe to webhooks for the repos that they're installed on. torchci uses the PyTorch Bot app to keep track of what's going on in pytorch/pytorch and any other repos it's installed on.

These webhooks are delivered to an API endpoint on [hud.pytorch.org] (/api/github/webhooks), which writes the webhook payload to a DynamoDB table corresponding to the event type. For example, workflow_job payloads are written to torchci-workflow-job.

ClickHouse ingests from DynamoDB using an AWS Lambda to automatically pick up changes in the tables.

Secondary write paths

There are some parts to the system that don't use the write path described above, because GitHub doesn't generate webhooks for them.

Notably, this means that installing the PyTorch Bot app to repos will not enable these write paths. They must be manually wired up.

Build artifacts

Our build jobs create various artifacts that we want to save to persistent storage and use later. For example, most of our build jobs produce binary wheel distributions of PyTorch that are later downloaded by our test jobs.

These are uploaded to S3 directly by the GitHub workflows in pytorch/pytorch.

Logs and log classifications

The PyTorch bot's workflow_job handler (lib/bot/logUploader.ts) asynchronously invokes the gha-log-uploader lambda when a job completes. That lambda downloads the log from GitHub, puts it in the ossci-raw-job-status bucket under log/, and then asynchronously invokes [log-classifier] to do the classification (more detail in the README).

Because this hangs off the App webhook rather than a per-repo one, every repo the bot is installed on gets log downloads and classifications without an admin configuring anything. Which repos are enabled is controlled by the LOG_UPLOADER_REPOS env var while the cutover from github-status-test is in progress; see #7549.

Missing logs are re-requested through backfillMissingLog in lib/jobUtils.ts, which Dr.CI calls when it finds a failed job with no log. Callers outside HUD use the authenticated POST /api/log-uploader/backfill route.

Test statistics

See the README.

CircleCI results

CircleCI also has webhooks like GitHub, but they must be configured manually. The webhooks are delivered to the ossci-circleci-to-s3 lambda, which is synced to Rockset. See the example config for pytorch/vision webhooks for an example.

Raw webhook payloads

The github-status-test lambda archives raw webhook payloads to the ossci-raw-job-status S3 bucket, under a prefix per event type. Nothing reads them: clickhouse-replicator-s3 has no SUPPORTED_PATHS entry for workflow_job/, workflow_run/, or full_workflow_*/, and ClickHouse gets jobs from DynamoDB through clickhouse-replicator-dynamo.

This archive goes away with the lambda. It is not reproduced in gha-log-uploader.

Adding a new repo to torchci

To get the basic HUD working, you will need to:

  1. Install the PyTorch Bot app to the repo. This requires admin access to the pytorch org. To find someone with admin access, consult the Build/CI POCs.
  2. Configure CircleCI to send webhooks to torchci (see above). This requires admin access to CircleCI. (NOTE: Only required if running CircleCI workflows)

You won't get any of the other goodies listed in "Secondary write paths" above, but other than that, things should work!