Skip to content

[fsdp, rollout, vllm, model, hardware] feat: integrate MXFP QAT with vLLM-Ascend - #2

Merged
Sunwish merged 5 commits into
Ascend-TechDev:lab/A2_MXFP_RL_Labfrom
Sunwish:pr/02-rollout-mxfp-fixes
Oct 29, 2026
Merged

Sunwish merged 5 commits into
Ascend-TechDev:lab/A2_MXFP_RL_Labfrom
Sunwish:pr/02-rollout-mxfp-fixes

Conversation

@Sunwish

@Sunwish Sunwish commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Integrates MXFP QAT between FSDP training and vLLM-Ascend rollout.

This PR adds a rollout configuration bridge that exports verl MXFP QAT settings to the vLLM-Ascend environment contract. When enabled, vLLM keeps model weights and kernels on the high-precision path while applying MXFP fake quantization according to the actor QAT configuration.

The PR also fixes several FSDP MXFP training issues:

  • Keep MXFP weights in high precision during FSDP parameter synchronization.
  • Reduce packed MoE activation memory by recomputing backward operands when needed.
  • Preserve the expected transposed layout for MoE weight gradients.
  • Correct gradient coordinate handling when block rotation is enabled.
  • Pad MoE expert token segments independently so block transforms do not mix tokens from different experts.

Related work:

  • #7895 adds Megatron/ModelOpt MXFP4 expert QAT.
  • #8353 adds MiniMax M3-specific activation QAT.
  • #6992 adds experimental W4A8 QAT simulation.
  • #8327 focuses on MXFP8 rollout refit and runtime checks.

The current PR focuses on FSDP MXFP QAT correctness and its vLLM-Ascend rollout integration.

Checklist Before Starting

Test

Added or updated the following tests:

  • tests/utils/test_mxfp_qat_vllm_ascend.py
    • MXFP mode mapping to vLLM-Ascend modes.
    • QAT environment variable generation.
    • Hydra/OmegaConf input handling.
    • Group-size and rounding-mode validation.
    • Expert enable/disable behavior.
  • tests/utils/test_mxfp_quant_cpu.py
    • Dense linear rotation gradient equivalence.
  • tests/utils/test_mxfp_moe_cpu.py
    • MoE backward operand recomputation.
    • Saved-tensor memory behavior.
    • Expert-row padding.
    • Gradient layout and rotation correctness.
  • tests/workers/test_fsdp_mxfp_qat_weight_sync.py
    • High-precision MXFP weight synchronization.

Validation performed:

  • git diff --check 027ccfd9^..9636ae06 passed.
  • Attempted:
    python -m pytest tests/utils/test_mxfp_quant_cpu.py tests/utils/test_mxfp_moe_cpu.py tests/utils/test_mxfp_qat_vllm_ascend.py tests/workers/test_fsdp_mxfp_qat_weight_sync.py -q
  • Test collection could not run in the current environment because PyTorch is not installed.
  • No Ascend/vLLM-Ascend end-to-end smoke test, training curve, or rollout benchmark is included in this commit range. An Ascend validation result should be added before requesting review.

API and Usage Example

Example FSDP2 plus vLLM-Ascend configuration:

actor_rollout_ref:
  actor:
    fsdp_config:
      qat:
        enable: true
        mode: mxfp_w8a8
        group_size: 32
        quant_backend: npu
        fake_quant_targets: [Fprop, Dgrad, Wgrad]
        backward_reduction_axis_fake_quant: true
        rotation_enable: false
        experts:
          enable: true
          npu_backend: grouped_matmul

  rollout:
    name: vllm
    quantization: null
    qat_fake_quant: true
    qat: ${actor_rollout_ref.actor.fsdp_config.qat}

Supported rollout mappings include:

  • mxfp_w8a8 → w8a8_mxfp8
  • mxfp_w4a8 → w4a8_mxfp4
  • mxfp_w4a4 → w4a4_mxfp4
    group_size must be 32. Stochastic rounding modes such as random and hash require quant_backend: torch. Enabling rollout.qat_fake_quant requires an Ascend NPU and vLLM-Ascend.
    When rollout fake quantization is enabled, the server keeps the high-precision path, ignores rollout.quantization, and passes the MXFP settings through environment variables consumed by vLLM-Ascend.

Design & Code Changes

  1. Add verl.utils.qat.vllm_ascend to validate and translate verl MXFP QAT configuration into the vLLM-Ascend environment contract.
  2. Add rollout.qat_fake_quant and wire the inherited actor QAT configuration into the vLLM rollout server.
  3. Disable legacy NVFP4 packing during FSDP MXFP weight synchronization so rollout receives high-precision actor weights.
  4. Update the Ascend MoE patch to use shared fake-quantized activation inputs for gate/up projections and custom grouped-matmul backward handling.
  5. Recompute large MoE backward operands instead of retaining all transformed activations and weights.
  6. Pad expert token segments independently before block transforms and preserve the required transposed weight-gradient layout.
  7. Fix dense and MoE gradient coordinate handling for optional block rotation.
  8. Add CPU-level unit coverage and generated configuration updates.

Compatibility Note
The generated rollout configurations now inherit qat from actor_rollout_ref.actor.fsdp_config.qat only. The previous fallback to actor_rollout_ref.actor.megatron.qat was removed.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

@Sunwish
Sunwish merged commit 6f0d4fd into Ascend-TechDev:lab/A2_MXFP_RL_Lab Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant