[RFC] Automated Upstream CI Testing for ci-infra Changes

[tmuttaki]

Summary: This RFC implements automated upstream CI testing using GitHub Actions and Buildkite's official integration. It requires one-time setup and provides opt-in automated testing via PR labels.

Background

Currently, testing ci-infra changes requires manual steps:

1. Developer creates a feature branch in ci-infra
2. Developer manually creates a Buildkite build with VLLM_CI_BRANCH=<branch-name>
3. Developer must remember the correct vLLM branch to test against
4. No direct link between ci-infra PR and test results

This is inefficient, and provides poor visibility into test results.

Solution

This RFC introduces automated upstream CI testing triggered via GitHub Actions when ci-infra PRs are labeled.

Key principles:

• Opt-in by design (only triggers when labeled)
• Smart branch matching (tests against matching vLLM branch or falls back to main)

What Changed

New GitHub Actions Workflow
File: .github/workflows/trigger-vllm-buildkite.yml

Triggers:

• PR labeled/unlabeled/opened/synchronize events
• Only runs when ci-run-all or ci-fast-check-only label is present

Behavior:

1. Checks for matching vLLM branch (same name as ci-infra branch)
2. Triggers Buildkite build via official buildkite/trigger-pipeline-action@v2.4.1
3. Sets VLLM_CI_BRANCH=<ci-infra-pr-branch> environment variable
4. Posts build link as PR comment

Required Setup

Before this workflow can function, the following one-time setup is required by maintainers:

1. Add Buildkite API Token go GitHub Secret
2. Create GitHub Labels

Label	Pipeline	Use Case
ci-run-all	Full vLLM CI	Critical changes (bootstrap, templates, infrastructure)
ci-fast-check-only	Fastcheck	Quick validation, minor changes

Benefits

• Faster feedback (no manual Buildkite navigation)
• Less context switching (everything in GitHub PR)
• Can't forget to test before merging
• Consistent testing of ci-infra changes
• Resource efficient (opt-in prevents waste)
• Maintainable (uses official Buildkite action)
• Auditable (logged in GitHub Actions)

Future Enhancements

Potential follow-ups (not in this RFC):

• Status checks integration (Need to setup a new buildkite-github integration like we have for vllm)
• Accelerator specific triggers to run tests on target accelarator

[tmuttaki]

• fast-check is deprecated
• parameterize branch name (put the branch name in the cli job)
• Use buildkite integration instead of github action/secret