[CI] Re-enable PyTorch nightly build and test in nightly CI/CD

[atalman]

Summary

• Re-enable PyTorch nightly build and test pipeline in the CI/CD infrastructure
• Add torch_nightly flag to step definitions so test areas can opt into nightly testing
• Auto-run nightly builds on main branch; require manual trigger on feature branches
• Companion PR: [CI] Add PyTorch nightly build and test pipeline with CUDA 13.0 support  vllm#37226 (Dockerfile + build script + test area config)

Changes

buildkite_step.py

• Import get_torch_nightly_image from docker utilities
• Inject $IMAGE_TAG_TORCH_NIGHTLY variable into pipeline steps
• Collect steps marked with torch_nightly: true into a dedicated group
• New _create_torch_nightly_group(): creates a "vLLM Against PyTorch Nightly" group with an optional manual block step, a
  Docker image build step (using .buildkite/image_build/image_build_torch_nightly.sh), and individual test steps using the
  nightly image
• New _get_nightly_step_plugin(): determines the correct plugin configuration based on device type

step.py

• Add torch_nightly: Optional[bool] = False field to the Step model, allowing test area YAML files to opt steps into the
  nightly pipeline

docker_utils.py

• New get_torch_nightly_image(): returns the appropriate nightly Docker image tag based on branch (main vs. premerge)

test-template-ci.j2

• Auto-run nightly builds on main branch (block step only on non-main, non-nightly branches)
• Update CUDA architecture lists to include 12.0 and 12.0a

Test plan

• Successful BuildKite run: https://buildkite.com/vllm/ci/builds/56482#019cf88b-09f9-4385-b7e5-15bf3ff2b276

[khluu]

If it's essentially mirroring steps and run the same commands just on a different image, can we leverage the mirror field, same as AMD?

[atalman]

Hi @khluu

Summary of changes:

buildkite/pipeline_generator/step.py

• Kept torch_nightly: Optional[bool] = False for backward compatibility
• Added a normalize_torch_nightly_to_mirror pydantic validator that promotes torch_nightly:
  true into mirror: {torch_nightly: {}}, so downstream code only checks one place

buildkite/pipeline_generator/buildkite_step.py

• Changed nightly collection check from step.torch_nightly to step.mirror and
  step.mirror.get("torch_nightly") is not None (works because the validator normalizes the
  legacy field)

buildkite/test-template-ci.j2

• Changed Jinja2 check to step.torch_nightly or (step.mirror and step.mirror.torch_nightly
  is defined) — needs both since Jinja2 doesn't go through the pydantic validator

Result

• Existing vllm YAML files with torch_nightly: true keep working
• New steps can use the mirror pattern per Kevin's suggestion: mirror: { torch_nightly: {} }
• Consistent with how AMD mirroring works via the mirror field