I loaded an AWQ model with SGLang, but the total VRAM used per node didn't go down

[aw632]

I loaded the AWQ Qwen model from https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct-AWQ, and started as follows:

python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct-AWQ --served-model-name Qwen/Qwen2.5-VL-7B-Instruct --disaggregation-mode prefill --disaggregation-transfer-backend nixl --enable-multimodal --enable-torch-compile --chunked-prefill-size 1024 --tp 4

But it appears that the model's memory usage in nvidia-smi is still 136GB on B200, the same as the unquantized model, even though the model weights are definitely much smaller. Performance with bench_one_batch doesn't seem to improve as well, suggesting somehow the quantized model was not loaded. Why is this the case? Is there another flag I need to set?