Hopper: build joyomni_ops for sm_90a, not sm_90 (FP8 GEMM aborts at launch on H100) - #27
Open
stayinalive181 wants to merge 1 commit into
Open
stayinalive181 wants to merge 1 commit into
stayinalive181 wants to merge 1 commit into
Conversation
The cutlass FP8 GEMM this library provides selects KernelTmaWarpSpecializedPingpongFP8FastAccum, whose WGMMA path is arch-conditional. Compiled as plain sm_90 it builds cleanly, imports cleanly, and reports has_fp8() == True, then aborts on every kernel launch with "Arch conditional MMA instruction used without targeting appropriate compute capability". On the streaming server that surfaces as a device-side assert during CUDA graph capture and the process exits during FastAPI startup. Both the default arch list and the auto-detect snippet in DEPLOYMENT.md emit `90` for Hopper, so following the published instructions on an H100 cannot produce a working build. Blackwell is already handled correctly (sm_100a / sm_120a); Hopper needs the same suffix for the same reason. Verified on an H100 80GB SXM (CUDA 12.8, torch 2.9.1+cu128): with JOYOMNI_OPS_CUDA_ARCHS=90a the graph captures and the server serves; without it every run dies at startup. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
fp8_scaled_mmindeploy/joyomni_opsselects cutlass'KernelTmaWarpSpecializedPingpongFP8FastAccum, whose WGMMA path is arch-conditional. Compiled as plainsm_90it builds cleanly, imports cleanly and reportshas_fp8() == True, then aborts on every kernel launch:On the streaming server that shows up as a device-side assert during CUDA graph capture, after which
mem_get_infofails and the process exits during FastAPI startup:Both the default arch list in
deploy/joyomni_ops/setup.pyand the auto-detect one-liner in DEPLOYMENT.md §2 emit90for Hopper, so following the published instructions on an H100 cannot produce a working build. Blackwell is already handled correctly (sm_100a/sm_120a); Hopper needs the same suffix for the same reason.Change
deploy/joyomni_ops/setup.py:-gencode=arch=compute_90a,code=sm_90ainstead ofcompute_90/sm_90, with a comment saying why; the docstring's arch list follows.DEPLOYMENT.md: the auto-detect snippet appendsafor compute capability 9.x as well as 10.x and later.The
compute_90PTX fallback for toolchains older than 12.8 is untouched.Verification
H100 80GB SXM, CUDA 12.8, torch 2.9.1+cu128, cutlass at the pinned commit. With
JOYOMNI_OPS_CUDA_ARCHS=90athe CUDA graph captures, the server starts, and the 480p/24 and 720p/16 profiles run with FP8 on. Without it every run dies at startup with the output above. Nothing changes for Ampere, Ada or Blackwell builds.