Add Cosmos3-Nano GPU smoke tests + GPU CI; self-prep regression inputs #1
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |
| # SPDX-License-Identifier: OpenMDW-1.1 | |
| # Cosmos3-Nano 8-GPU SFT pipeline smoke test (convert -> train 5 -> export -> | |
| # t2i infer) on a self-hosted 8×H200 runner. | |
| # | |
| # Requires: | |
| # * a self-hosted runner labelled [self-hosted, gpu, h200] with 8 GPUs, | |
| # NVIDIA drivers, and `uv` on PATH; | |
| # * an `HF_TOKEN` repository secret (gated dataset/model downloads). | |
| # | |
| # Inputs (Cosmos3-Nano -> DCP, bridge dataset, Wan VAE) are downloaded / | |
| # converted in-test and cached under examples/ + the HF cache; the first run is | |
| # slow (~30 GB Nano + DCP convert + 5-step train + export + a t2i generation), | |
| # later runs reuse the cache. | |
| name: GPU Smoke (Training) | |
| on: | |
| push: | |
| branches: [main] | |
| pull_request: | |
| branches: [main] | |
| concurrency: | |
| group: gpu-smoke-training-${{ github.ref }} | |
| cancel-in-progress: true | |
| jobs: | |
| training-smoke: | |
| runs-on: [self-hosted, gpu, h200] | |
| timeout-minutes: 90 | |
| env: | |
| HF_TOKEN: ${{ secrets.HF_TOKEN }} | |
| HF_HUB_DISABLE_XET: "1" | |
| steps: | |
| - uses: actions/checkout@v6 | |
| - uses: astral-sh/setup-uv@v7 | |
| - name: Sync environment (cu128-train) | |
| run: uv sync --all-extras --group=cu128-train | |
| # Full SFT pipeline: download + convert Nano->DCP, train 5 steps (loss | |
| # trend), export to HF safetensors, then a t2i generation from the export. | |
| # MAX_GPUS defaults to 8. -s streams the live process log. | |
| - name: Nano SFT pipeline smoke (convert -> train 5 -> export -> t2i, 8 GPU) | |
| run: | | |
| export LD_LIBRARY_PATH= | |
| uv run --all-extras --group=cu128-train python -m pytest -v -s \ | |
| tests/nano_training_smoke_test.py --num-gpus=8 --levels=2 -o addopts= | |
| # Clear the heavy artifacts (even on failure): examples/checkpoints (the | |
| # Cosmos3-Nano DCP + Wan VAE, ~30 GB) and the pytest tmp dirs (the SFT | |
| # checkpoint + logs). The small examples/data dataset and the HF cache are | |
| # intentionally kept so subsequent runs reuse them. | |
| - name: Clean up run outputs | |
| if: always() | |
| run: | | |
| rm -rf examples/checkpoints || true | |
| rm -rf "${TMPDIR:-/tmp}"/pytest-of-* /tmp/pytest-of-* || true |