Add NVFP4 quantization-aware distillation (QAD)#3889
Draft
andrewor14 wants to merge 1 commit into
Draft
Conversation
|
|
8f9a701 to
80639d4
Compare
Summary:
QAD recovers the accuracy lost when a model is quantized to NVFP4 by
distilling from the original bf16 model into an NVFP4 fake-quantized copy. The
student's MoE experts and dense linear layers are fake-quantized with a
straight-through estimator during training; a frozen bf16 teacher supervises
via KL divergence (plus an optional hard-label cross-entropy term).
Adds:
- NVFP4 fake-quant primitives: torchtitan/components/quantization/
nvfp4_fake_quant.py -- self-contained two-level block-scale E2M1 fake-quant
(STE), plus a single entry point apply_nvfp4_fake_quant(model) that quantizes
both the MoE experts and the dense linears (excluding lm_head/router/gate).
- KD/QAD trainer + configs: torchtitan/experiments/kd/ -- KDTrainer (frozen
bf16 teacher + fake-quant student), kd_loss (KL + CE), and qwen3 QAD configs
(debug + 30B-A3B).
- MoE fake-quant hook: GroupedExperts.fake_quant_fn in models/common/moe.py,
applied to expert weights and activations inside _experts_forward.
- Post-init hook: post_model_init_fn in trainer.py, used to apply the
fake-quant to the student after model init/parallelization.
- Single-file HF checkpoint support: protocols/state_dict_adapter.py.
The NVFP4 fake-quant primitives in nvfp4_fake_quant.py are intentionally
self-contained (no torchao dependency), but could be upstreamed to torchao if a
shared home is preferred.
Test Plan:
Distill a qwen3-30B-A3B checkpoint (student + bf16 teacher both init from the
same HF checkpoint):
MODULE=kd.qwen3 CONFIG=qwen3_30b_a3b_qad NGPU=8 ./run_train.sh \
--training.steps 200 \
--checkpoint.interval 50 \
--checkpoint.initial_load_path /path/to/qwen3-30b-a3b-hf \
--hf_assets_path /path/to/qwen3-30b-a3b-hf
Quick self-contained smoke test (tiny debug model, no external checkpoint):
MODULE=kd.qwen3 CONFIG=qwen3_moe_debug_qad NGPU=2 ./run_train.sh
Results (MATH accuracy, bf16 -> nvfp4):
qwen3-30B-A3B
baseline (200-step RL ckpt): 88.6% -> 86.6% (-2.0)
+ QAD (800 steps): nvfp4 88.6% (+2.0)
-> 100% of the gap recovered
gpt-oss-20B
baseline (100-step RL ckpt): 62.6% -> 50.8% (-11.8)
+ QAD (1400 steps): nvfp4 55.2% (+4.4)
-> 37% of the gap recovered
QAD closes the full NVFP4 gap on qwen3-30B-A3B and recovers a third of the
(much larger) gap on gpt-oss-20B.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary: QAD recovers the accuracy lost when a model is quantized to NVFP4 by distilling from the original bf16 model into an NVFP4 fake-quantized copy. The student's MoE experts and dense linear layers are fake-quantized with a straight-through estimator during training; a frozen bf16 teacher supervises via KL divergence (plus an optional hard-label cross-entropy term).
Adds:
Test Plan:
Quick self-contained smoke test (tiny debug model, no external checkpoint):
Results (MATH accuracy, bf16 -> nvfp4):
QAD closes the full NVFP4 gap on qwen3-30B-A3B and recovers a third of the (much larger) gap on gpt-oss-20B.