Skip to content

test(inference): smoke-test FP8 mixed-precision diffusion steps - #226

Merged
lfengad merged 2 commits into
mainfrom
pzeren/mixed-precision-inference-smoke
Aug 29, 2026
Merged

test(inference): smoke-test FP8 mixed-precision diffusion steps#226
lfengad merged 2 commits into
mainfrom
pzeren/mixed-precision-inference-smoke

Conversation

@pengcuo

@pengcuo pengcuo commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

What

Adds test_nano_fp8_mixed_precision_inference to tests/nano_inference_smoke_test.py: the same sharded-layout text2video run as test_nano_fp8_inference, plus the FP8 mixed-precision diffusion-step flags from #217 (--mixed-precision-first-steps=2 --mixed-precision-last-steps=2, default w8a16-cache=none — the only mode valid under FSDP sharding, which is exactly the layout used).

Pass criterion

The schedule itself, not just a valid video: the test parses the MIXED_PRECISION_TRACE log line and compares it exactly against the expected 2x W8A16 / 6x W8A8 / 2x W8A16 sequence for the 10-step run (num_steps is read back from the shared _FP8_GENERATION_ARGS so the expectation cannot drift). A run where the flags silently never engaged (all-W8A8 trace or no trace) fails even though its video would look fine. _assert_video_has_content still catches the numerically-broken-but-running case, and the existing FP8 swap-count guard confirms the FP8 path engaged at all.

CI wiring

No new job or trigger: the existing generator-inference-smoke job runs the whole file, so the case is picked up automatically. Like the other two FP8 cases it SKIPs (not fails) when HF_TOKEN cannot read nvidia/Cosmos3-Experimental. Job timeout bumped 90 → 105 min for the extra full-width run.

Verification

pytest --collect-only inside the inference container collects 5 tests including the new case; the GPU run itself is exercised by this PR's own generator-inference-smoke job.

🤖 Generated with Claude Code

Add test_nano_fp8_mixed_precision_inference to the Nano inference smoke
suite: the sharded FP8 run from test_nano_fp8_inference plus
--mixed-precision-first-steps=2 / --mixed-precision-last-steps=2 (the
feature from #217). Asserts the exact per-step precision schedule from
the MIXED_PRECISION_TRACE log line (2x W8A16 / 6x W8A8 / 2x W8A16 on the
10-step run) so a run where the flags silently never engaged fails, plus
the usual non-degenerate-video check.

Runs automatically in the existing generator-inference-smoke CI job
(which invokes the whole file); timeout bumped 90 -> 105 minutes for the
extra full-width FP8 run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@lfengad

lfengad commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Not introduced by this PR — flagging it here because it bounds how much the FP8 coverage, including the new mixed-precision case, can be trusted over time.

All three FP8 cases pytest.skip rather than fail when nvidia/Cosmos3-Experimental is unreachable. That is the right default for forks and local runs, and _download_fp8_checkpoint is careful to skip only on the credentials-shaped errors (GatedRepoError, RepositoryNotFoundError, HTTP 401/403) and still fail loudly on anything else. The gap is what happens on the one job whose stated purpose includes FP8 coverage: generator-inference-smoke can silently go from three full-width FP8 runs to zero FP8 coverage without ever turning red. An HF_TOKEN rotation, a permissions change on the gated repo, or the pinned revision being cleaned up would each do that, and the job would still report green in ~2 minutes less wall time than before — a difference nobody watches.

The workflow comment already says to grep the log for no access to nvidia/Cosmos3-Experimental before trusting a green run to have covered FP8, but that is a convention asking a human to remember, not an assertion.

Cheap fix if it seems worth it: have the generator-inference-smoke job export something like REQUIRE_FP8=1, and let _download_fp8_checkpoint promote its pytest.skip to a pytest.fail when that is set. Forks, local runs, and every other job keep the current skip behaviour; only the job that is supposed to cover FP8 goes red when it cannot get the checkpoint.

Worth noting this resolves itself once the checkpoint is published under its own name and registered in _CHECKPOINTS_download_fp8_checkpoint disappears entirely at that point, per the TODO already in the comment block. So "wait for the release" is a legitimate answer; raising it in case that is further out than expected.

… without the checkpoint

Per review: the three FP8 cases skip when nvidia/Cosmos3-Experimental is
unreachable for a credentials reason, which is right for forks and local
runs but let the generator-inference-smoke job silently drop from three
full-width FP8 runs to zero while staying green (token rotation, repo
permissions change). That job now sets REQUIRE_FP8=1, which
_download_fp8_checkpoint uses to promote the skip to a pytest.fail.
Everything else keeps the skip behaviour.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pengcuo

pengcuo commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator Author

Good call — implemented in 04d77a2: generator-inference-smoke now sets REQUIRE_FP8=1, and _download_fp8_checkpoint promotes its credentials-shaped pytest.skip to a pytest.fail when that is set. Forks, local runs, and every other job keep the skip behaviour; only the job whose stated purpose includes FP8 coverage goes red when it cannot reach the checkpoint. Also replaced the grep-the-log workflow comment, since a green run of that job now guarantees FP8 coverage by construction. The whole thing still disappears with _download_fp8_checkpoint once the checkpoint is released and registered in _CHECKPOINTS.

@lfengad lfengad left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Verified on the latest commit (04d77a28): all 9 checks green, and generator-inference-smoke genuinely covered FP8 rather than skipping — 5 passed, three Swapped 504 linears, and MIXED_PRECISION_TRACE steps=W8A16,W8A16,W8A8,W8A8,W8A8,W8A8,W8A8,W8A8,W8A16,W8A16 on the new case, matching the asserted 2/6/2 schedule exactly.

Thanks for picking up the REQUIRE_FP8 point — with it set on that job, a green run now proves FP8 coverage instead of relying on someone remembering to grep the log. That run also happened to hit a cold cache (41m56s vs 12m03s), which is a useful confirmation that the 90 → 105 timeout bump was needed.

@lfengad
lfengad merged commit 0e034bc into main Aug 29, 2026
9 checks passed
@lfengad
lfengad deleted the pzeren/mixed-precision-inference-smoke branch August 29, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants