Repository navigation
Conversation
Train the Megatron actor on the weights a ModelOpt MIXED_PRECISION (W4A16 NVFP4 + FP8) rollout checkpoint deploys: - qat.mode=modelopt_mixed replaces the weight quantizers of the quantized deployment layers with a requantize + STE quantizer, including TE grouped linears (weight-only ModelOpt quant modules, registered for one mtq.quantize call). - mixed_precision_export requantizes every full-tensor weight sync into the checkpoint format with the same rule and forwards the checkpoint's static activation scales. - hf_weights_path builds and initializes the Megatron model from a checkpoint other than model.path (BF16 masters for a quantized rollout). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
The exporter's receiver and the rules stay in Megatron, so both arms of the comparison share one requantize. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
The rollout inherits the actor's qat config; modelopt_mixed serves the checkpoint's own ModelOpt config and has no quantization_config_path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
The QAT forward quantizes each rank's shard and the exporter the full tensor, so their scales differ once a weight is split across ranks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Replace the hand-written requantize rule with ModelOpt 0.47: mtq.quantize with the model's recipe (static MSE weight calibration on the BF16 masters, FP8 input / KV quantizers pinned to the deployment checkpoint's static scales through constant_amax), forward on the dequantized exporter output with a straight-through gradient, and weight sync through the same exporter functions. No dependency on megatron.lite any more. Deployment scales are kept out of the quantizer attributes (ModelOpt pickles those into every module's _extra_state), and a checkpoint load takes _extra_state from the live model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Trainer side of weight QAT for a ModelOpt
MIXED_PRECISIONrollout checkpoint (W4A16 NVFP4 + FP8, here Nemotron-H Lightning NVFP4) on the Megatron engine, starting from BF16 masters. The quantization rule comes from ModelOpt 0.47, not from code in this repository.qat.mode=modelopt_mixed:load_recipeof ModelOpt's recipe for this model (models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6) and onemtq.quantizecall on the MCore model. Weight quantizers are calibrated statically (MSE) on the BF16 masters at start-up and stay frozen. The recipe's FP8 input quantizers (Mambain_proj/out_proj) and FP8 KV quantizers are pinned to the deployment checkpoint's static scales through the recipe fieldconstant_amax, so no calibration data is needed.dequantize(exporter(w))with a straight-through gradient, where the exporter is ModelOpt'sto_quantized_weighton the frozen scales and the dequantize isNVFP4QTensor.dequantize. The trained weight is therefore the served weight.mixed_precision_export=True: every full-tensor weight sync quantizes the current masters with the same function and scales and sends the staticinput_scale/k_scale/v_scaleof the quantizers.hf_weights_path: build and initialize the Megatron model from a checkpoint other thanmodel.path(BF16 masters whilemodel.pathis the quantized rollout checkpoint)._extra_statefrom the live model instead of the checkpoint.nvidia-modeloptpinned to 0.47.0 (first release with the recipe).ModelOpt ships this recipe as a PTQ recipe; there is no QAT recipe for the model. It is used here as the definition of the deployment format that the QAT forward has to reproduce.
What is still hand-written, and why
E4M3(BF16(w / scale))written outto_quantized_weightapplies this rule, but on the GPU PyTorch rounds its 0-dim scale to BF16 before dividing. On the released checkpoint's 46 FP8 tensors (8.9e8 codes): the written-out rule reproduces all codes;to_quantized_weighton GPU differs in 1.72% of codes (35 of 46 tensors); on CPU it reproduces all codes.GPTModelExporteris a one-shot whole-model export on CPU; Megatron-Bridge's ModelOpt export rejects static NVFP4 and has no FP8 path. The glue calls the same ModelOpt functions per tensor on Bridge's weight stream and gathers expert scales over the EP group once.hf_quant_config.json; start-up fails if the two sets differ.Dependencies
nvidia-modelopt==0.47.0.verl_mlite.rollout.vllm_worker.MLiteVLLMColocateWorkerExtensionfrom [mLite] Nemotron-H: vLLM refit receiver for ModelOpt MIXED_PRECISION checkpoints (baseline arm only) Megatron-LM#9. verl's own rollout reload only recognizesModelOptNvFp4Config. Nothing else frommegatron.liteis imported any more.Known limitations
quantizer.forward = ...); a ModelOpt hook for a custom forward would be cleaner.mixed_precision_exportsupports only the full-tensor sync (checkpoint_engine.backend=naive). The ref worker does not inherithf_weights_path.Test
No test files in this PR.
hf_quant_config.json: 2944 + 2944 routed expert projections, 46 shared-expert projections, 1 output layer, 46 FP8 Mamba weights with 46 input quantizers, 6 + 6 KV quantizers; exportedinput_scale/k_scale/v_scaleequal the checkpoint's bit for bit.NVFP4QTensor.dequantizecompared with the previous decode on 8.9e8 elements: identical.Not a duplicate of verl-project#6992 (FSDP W4A8 simulation) or verl-project#7895 (MXFP4 experts): this targets ModelOpt MIXED_PRECISION checkpoints on the Megatron engine.
AI assistance was used for this change.
🤖 Generated with Claude Code