Repository navigation
Conversation
The FP8 end-to-end requirements list NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1 unconditionally. On Blackwell (SM100+) Transformer Engine emulates the block-wise recipe with MXFP8 GEMMs, whose E8M0 scales carry only an exponent, and both of its block-quantize kernels assert power-of-two scales there, so a run with the variable set stops at the first FP8 quantization (the error reported in verl-project#6172). Say that the variable is for Hopper and that on Blackwell it must stay unset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wuxibin89
approved these changes
Sep 29, 2026
6 of 7 tasks
thecourageyu
pushed a commit
to thecourageyu/verl
that referenced
this pull request
Nov 11, 2026
…erl-project#8062) ### What does this PR do? The FP8 end-to-end section of `docs/low_precision/fp8.md` lists `NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1` as a general requirement. It only applies on Hopper. On Blackwell (SM100+), Transformer Engine has no native block-scaled GEMM. It runs the block-wise recipe through MXFP8 GEMMs, whose E8M0 scales hold only an exponent. So both of its block-quantize kernels assert power-of-two scales on SM100+, and a run with the variable set stops at the first FP8 quantization: ``` transformer_engine/common/transpose/quantize_transpose_vector_blockwise.cu:534 in function quantize_transpose_vector_blockwise: Assertion failed: pow2_scale. On Blackwell and newer, the FP8 block scaling recipe is emulated with MXFP8, which requires using power of two scaling factors. ``` This is the failure reported in verl-project#6172 (8×B200) when following this doc and the verl-recipe FP8 E2E script. This PR marks the variable as Hopper-only and says that on Blackwell it must stay unset; power-of-two scales are the Transformer Engine default. Docs only. Refs verl-project#6172. This does not fix verl-project#6172: that issue also reports garbled FP8 rollouts on Blackwell once the variable is removed, and this PR does not address that. ### Checklist Before Starting - [x] Search for similar PRs: - [NVTE_FP8_BLOCK_SCALING_FP32_SCALES](https://github.com/verl-project/verl/pulls?q=NVTE_FP8_BLOCK_SCALING_FP32_SCALES) - [blockwise Blackwell](https://github.com/verl-project/verl/pulls?q=blockwise+Blackwell) - No PR changes this requirement. Related: verl-project#6172 (the issue above); verl-project#7895 (sets the variable in its DeepSeek-V4 example scripts, does not touch this doc); verl-project#7519 (MXFP8 training, does not change this line). - [x] Format the PR title as `[{modules}] {type}: {description}` ### Test Documentation-only change. - `pre-commit run --files docs/low_precision/fp8.md`: all hooks pass, including "Check docs Last updated info". - Reproduced on 1×B200 (SM 10.0, torch 2.11.0+cu130, CUDA 13.0, Transformer Engine 2.16.1). `te.Linear` (2048→2048) and `te.GroupedLinear` (4 × 2048→768) ran forward and backward under `Float8BlockScaling(fp8_format=Format.E4M3)` and were compared with a bf16 run of the same module: | `NVTE_FP8_BLOCK_SCALING_FP32_SCALES` | `te.Linear` out / dgrad / wgrad rel. err | `te.GroupedLinear` out / dgrad / wgrad rel. err | |---|---|---| | unset (power-of-two scales) | 0.0377 / 0.0423 / 0.0462 | 0.0376 / 0.0475 / 0.0510 | | `1` | `RuntimeError`: the `pow2_scale` assertion above | same | The same modules under `MXFP8BlockScaling` give the same errors as the unset row, with or without the variable. Source references (Transformer Engine 2.16.1, commit c9877beb): - `transformer_engine/common/transpose/quantize_transpose_vector_blockwise.cu:533-536` and `transformer_engine/common/transpose/quantize_transpose_square_blockwise.cu:490-493`: `if (sm_arch() >= 100) NVTE_CHECK(pow2_scale, ...)`. - `transformer_engine/pytorch/csrc/extensions/gemm.cpp:289-300`: "Emulate the FP8 block scaling recipe with MXFP8 on Blackwell and newer as it is not natively supported by cublasLt". - `transformer_engine/common/swizzle/swizzle_block_scaling.cu:116-118, 205-206`: each FP32 block scale becomes an E8M0 scale by keeping only its exponent byte, `(sf >> 23) & 0xFF`. This is why the scales must be powers of two. ### API and Usage Example None. ### Design & Code Changes `docs/low_precision/fp8.md`: - The requirement line now says that the variable is Hopper-only, why it fails on Blackwell, and what to do there. - "Last updated" is bumped. The verl-recipe script `low_precision/run_dapo_qwen3_moe_30b_megatron_fp8e2e.sh` also exports the variable unconditionally. That lives in the recipe repository and can get a matching note separately. ### Checklist Before Submitting - [ ] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting). - [x] Add / Update [the documentation](https://github.com/verl-project/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/verl-project/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: documentation-only change; the behaviour it documents was checked on B200 as described under Test. - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1). - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit: not applicable. AI assistance: this PR was prepared with AI assistance (Claude Code). I reviewed the change and the evidence above, and the checks listed under Test were run for this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The FP8 end-to-end section of
docs/low_precision/fp8.mdlistsNVTE_FP8_BLOCK_SCALING_FP32_SCALES=1as a general requirement. It only applies on Hopper.On Blackwell (SM100+), Transformer Engine has no native block-scaled GEMM. It runs the block-wise recipe through MXFP8 GEMMs, whose E8M0 scales hold only an exponent. So both of its block-quantize kernels assert power-of-two scales on SM100+, and a run with the variable set stops at the first FP8 quantization:
This is the failure reported in #6172 (8×B200) when following this doc and the verl-recipe FP8 E2E script. This PR marks the variable as Hopper-only and says that on Blackwell it must stay unset; power-of-two scales are the Transformer Engine default. Docs only.
Refs #6172. This does not fix #6172: that issue also reports garbled FP8 rollouts on Blackwell once the variable is removed, and this PR does not address that.
Checklist Before Starting
[{modules}] {type}: {description}Test
Documentation-only change.
pre-commit run --files docs/low_precision/fp8.md: all hooks pass, including "Check docs Last updated info".te.Linear(2048→2048) andte.GroupedLinear(4 × 2048→768) ran forward and backward underFloat8BlockScaling(fp8_format=Format.E4M3)and were compared with a bf16 run of the same module:NVTE_FP8_BLOCK_SCALING_FP32_SCALESte.Linearout / dgrad / wgrad rel. errte.GroupedLinearout / dgrad / wgrad rel. err1RuntimeError: thepow2_scaleassertion aboveThe same modules under
MXFP8BlockScalinggive the same errors as the unset row, with or without the variable.Source references (Transformer Engine 2.16.1, commit c9877beb):
transformer_engine/common/transpose/quantize_transpose_vector_blockwise.cu:533-536andtransformer_engine/common/transpose/quantize_transpose_square_blockwise.cu:490-493:if (sm_arch() >= 100) NVTE_CHECK(pow2_scale, ...).transformer_engine/pytorch/csrc/extensions/gemm.cpp:289-300: "Emulate the FP8 block scaling recipe with MXFP8 on Blackwell and newer as it is not natively supported by cublasLt".transformer_engine/common/swizzle/swizzle_block_scaling.cu:116-118, 205-206: each FP32 block scale becomes an E8M0 scale by keeping only its exponent byte,(sf >> 23) & 0xFF. This is why the scales must be powers of two.API and Usage Example
None.
Design & Code Changes
docs/low_precision/fp8.md:The verl-recipe script
low_precision/run_dapo_qwen3_moe_30b_megatron_fp8e2e.shalso exports the variable unconditionally. That lives in the recipe repository and can get a matching note separately.Checklist Before Submitting
ci-requestchannel.recipesubmodule, please also update the reference to the submodule commit: not applicable.AI assistance: this PR was prepared with AI assistance (Claude Code). I reviewed the change and the evidence above, and the checks listed under Test were run for this PR.
🤖 Generated with Claude Code