Skip to content

Add NVFP4 quantization-aware distillation (QAD)#3889

Draft
andrewor14 wants to merge 1 commit into
pytorch:mainfrom
andrewor14:qad
Draft

Add NVFP4 quantization-aware distillation (QAD)#3889
andrewor14 wants to merge 1 commit into
pytorch:mainfrom
andrewor14:qad

Conversation

@andrewor14

@andrewor14 andrewor14 commented Jul 8, 2026

Copy link
Copy Markdown

Summary: QAD recovers the accuracy lost when a model is quantized to NVFP4 by distilling from the original bf16 model into an NVFP4 fake-quantized copy. The student's MoE experts and dense linear layers are fake-quantized with a straight-through estimator during training; a frozen bf16 teacher supervises via KL divergence (plus an optional hard-label cross-entropy term).

Adds:

  • NVFP4 fake-quant primitives: (can be moved to torchao if needed) torchtitan/components/quantization/ nvfp4_fake_quant.py -- self-contained two-level block-scale E2M1 fake-quant (STE), plus a single entry point apply_nvfp4_fake_quant(model) that quantizes both the MoE experts and the dense linears (excluding lm_head/router/gate).
  • KD/QAD trainer + configs: torchtitan/experiments/kd/ -- KDTrainer (frozen bf16 teacher + fake-quant student), kd_loss (KL + CE), and qwen3 QAD configs (debug + 30B-A3B).
  • MoE fake-quant hook: GroupedExperts.fake_quant_fn in models/common/moe.py, applied to expert weights and activations inside _experts_forward.
  • Post-init hook: post_model_init_fn in trainer.py, used to apply the fake-quant to the student after model init/parallelization.
  • Single-file HF checkpoint support: protocols/state_dict_adapter.py.

Test Plan:

MODULE=kd.qwen3 CONFIG=qwen3_30b_a3b_qad NGPU=8 ./run_train.sh \
    --training.steps 200 \
    --checkpoint.interval 50 \
    --checkpoint.initial_load_path /path/to/qwen3-30b-a3b-hf \
    --hf_assets_path /path/to/qwen3-30b-a3b-hf

Quick self-contained smoke test (tiny debug model, no external checkpoint):

MODULE=kd.qwen3 CONFIG=qwen3_moe_debug_qad NGPU=2 ./run_train.sh

Results (MATH accuracy, bf16 -> nvfp4):

  qwen3-30B-A3B
    baseline (200-step RL ckpt):  88.6% -> 86.6%   (-2.0)
    + QAD (800 steps):            nvfp4 88.6%       (+2.0)
    -> 100% of the gap recovered

  gpt-oss-20B
    baseline (100-step RL ckpt):  62.6% -> 50.8%   (-11.8)
    + QAD (1400 steps):           nvfp4 55.2%       (+4.4)
    -> 37% of the gap recovered

QAD closes the full NVFP4 gap on qwen3-30B-A3B and recovers a third of the (much larger) gap on gpt-oss-20B.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jul 8, 2026
@pytorch-bot

pytorch-bot Bot commented Jul 8, 2026

Copy link
Copy Markdown

Workflows were awaiting approval. CI has now been triggered for the ciflow labels on this PR.

@andrewor14 andrewor14 marked this pull request as draft July 8, 2026 21:14
@andrewor14 andrewor14 force-pushed the qad branch 2 times, most recently from 8f9a701 to 80639d4 Compare July 8, 2026 21:22
Summary:
QAD recovers the accuracy lost when a model is quantized to NVFP4 by
distilling from the original bf16 model into an NVFP4 fake-quantized copy. The
student's MoE experts and dense linear layers are fake-quantized with a
straight-through estimator during training; a frozen bf16 teacher supervises
via KL divergence (plus an optional hard-label cross-entropy term).

Adds:
- NVFP4 fake-quant primitives: torchtitan/components/quantization/
  nvfp4_fake_quant.py -- self-contained two-level block-scale E2M1 fake-quant
  (STE), plus a single entry point apply_nvfp4_fake_quant(model) that quantizes
  both the MoE experts and the dense linears (excluding lm_head/router/gate).
- KD/QAD trainer + configs: torchtitan/experiments/kd/ -- KDTrainer (frozen
  bf16 teacher + fake-quant student), kd_loss (KL + CE), and qwen3 QAD configs
  (debug + 30B-A3B).
- MoE fake-quant hook: GroupedExperts.fake_quant_fn in models/common/moe.py,
  applied to expert weights and activations inside _experts_forward.
- Post-init hook: post_model_init_fn in trainer.py, used to apply the
  fake-quant to the student after model init/parallelization.
- Single-file HF checkpoint support: protocols/state_dict_adapter.py.

The NVFP4 fake-quant primitives in nvfp4_fake_quant.py are intentionally
self-contained (no torchao dependency), but could be upstreamed to torchao if a
shared home is preferred.

Test Plan:
Distill a qwen3-30B-A3B checkpoint (student + bf16 teacher both init from the
same HF checkpoint):

  MODULE=kd.qwen3 CONFIG=qwen3_30b_a3b_qad NGPU=8 ./run_train.sh \
      --training.steps 200 \
      --checkpoint.interval 50 \
      --checkpoint.initial_load_path /path/to/qwen3-30b-a3b-hf \
      --hf_assets_path /path/to/qwen3-30b-a3b-hf

Quick self-contained smoke test (tiny debug model, no external checkpoint):

  MODULE=kd.qwen3 CONFIG=qwen3_moe_debug_qad NGPU=2 ./run_train.sh

Results (MATH accuracy, bf16 -> nvfp4):

  qwen3-30B-A3B
    baseline (200-step RL ckpt):  88.6% -> 86.6%   (-2.0)
    + QAD (800 steps):            nvfp4 88.6%       (+2.0)
    -> 100% of the gap recovered

  gpt-oss-20B
    baseline (100-step RL ckpt):  62.6% -> 50.8%   (-11.8)
    + QAD (1400 steps):           nvfp4 55.2%       (+4.4)
    -> 37% of the gap recovered

QAD closes the full NVFP4 gap on qwen3-30B-A3B and recovers a third of the
(much larger) gap on gpt-oss-20B.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/8gpu CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant