Loading model takes more GPU memory than that as expected

[harryzwh]

Trying to move the latest version of vLLM(0.26.0 from docker) + vllm-gguf-plugin(0.4.0 from pipy) and the GPU memory usage increase 50% more than the old version with integrated support of gguf.

With vLLM=0.26.0 and vllm-gguf-plugin=0.4.0，run
vllm serve /models/Qwen3-8B/Qwen3-8B-Q4_K_M.gguf --tokenizer=/models/Qwen3-8B --served-model-name=Qwen/Qwen3-8B --max-model-len=16384 --max-num-seqs=16 --gpu-memory-utilization=0.79 --kv-cache-dtype=fp8
log shows a 7.5G of GPU memory usage, which prevent running a Q4-quantized 8B model on a graphics card with 8GB memory.

:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
INFO 08-06 08:37:39 [init.py:112] Registered model loader <class 'vllm_gguf_plugin.loader.GGUFModelLoader'> with load format gguf
INFO 08-06 08:37:39 [config.py:455] Registered config parser <class 'vllm_gguf_plugin.config_parser.GGUFConfigParser'> with config format gguf
(APIServer pid=1167) INFO 08-06 08:37:39 [api_utils.py:345]
(APIServer pid=1167) INFO 08-06 08:37:39 [api_utils.py:345] █ █ █▄ ▄█
(APIServer pid=1167) INFO 08-06 08:37:39 [api_utils.py:345] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.26.0
(APIServer pid=1167) INFO 08-06 08:37:39 [api_utils.py:345] █▄█▀ █ █ █ █ model /models/Qwen3-8B/Qwen3-8B-Q4_K_M.gguf
(APIServer pid=1167) INFO 08-06 08:37:39 [api_utils.py:345] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=1167) INFO 08-06 08:37:39 [api_utils.py:345]
(APIServer pid=1167) INFO 08-06 08:37:39 [api_utils.py:273] non-default args: {'model_tag': '/models/Qwen3-8B/Qwen3-8B-Q4_K_M.gguf', 'model': '/models/Qwen3-8B/Qwen3-8B-Q4_K_M.gguf', 'tokenizer': '/models/Qwen3-8B', 'max_model_len': 16384, 'served_model_name': ['Qwen/Qwen3-8B'], 'gpu_memory_utilization': 0.6, 'kv_cache_dtype': 'fp8', 'max_num_seqs': 16}
(APIServer pid=1167) INFO 08-06 08:37:39 [model.py:623] Resolved architecture: Qwen3ForCausalLM
(APIServer pid=1167) INFO 08-06 08:37:39 [model.py:1788] Using max model len 16384
(APIServer pid=1167) INFO 08-06 08:37:39 [cache.py:285] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=1167) INFO 08-06 08:37:39 [vllm.py:1109] Asynchronous scheduling is enabled.
(APIServer pid=1167) INFO 08-06 08:37:39 [kernel.py:295] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore pid=1254) INFO 08-06 08:37:47 [init.py:112] Registered model loader <class 'vllm_gguf_plugin.loader.GGUFModelLoader'> with load format gguf
(EngineCore pid=1254) INFO 08-06 08:37:47 [config.py:455] Registered config parser <class 'vllm_gguf_plugin.config_parser.GGUFConfigParser'> with config format gguf
(EngineCore pid=1254) INFO 08-06 08:37:47 [core.py:116] Initializing a V1 LLM engine (v0.26.0) with config: model='/models/Qwen3-8B', speculative_config=None, tokenizer='/models/Qwen3-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=gguf, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gguf, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=Qwen/Qwen3-8B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
(EngineCore pid=1254) INFO 08-06 08:37:48 [parallel_state.py:1615] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.0.3:57619 backend=nccl
(EngineCore pid=1254) INFO 08-06 08:37:48 [parallel_state.py:1946] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=1254) INFO 08-06 08:37:48 [gpu_worker.py:378] Using V2 Model Runner
(EngineCore pid=1254) INFO 08-06 08:37:48 [model_runner.py:284] Loading model from scratch...
(EngineCore pid=1254) INFO 08-06 08:38:07 [cuda.py:482] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
(EngineCore pid=1254) INFO 08-06 08:38:15 [model_runner.py:305] Model loading took 7.5 GiB and 27.175852 seconds

For comparison, with old version 0.18.0, it takes less than 5GB of GPU memory.

qwen3-8b-1 | (APIServer pid=1) INFO 08-06 10:08:29 [utils.py:297]
qwen3-8b-1 | (APIServer pid=1) INFO 08-06 10:08:29 [utils.py:297] █ █ █▄ ▄█
qwen3-8b-1 | (APIServer pid=1) INFO 08-06 10:08:29 [utils.py:297] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.18.0
qwen3-8b-1 | (APIServer pid=1) INFO 08-06 10:08:29 [utils.py:297] █▄█▀ █ █ █ █ model /models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf
qwen3-8b-1 | (APIServer pid=1) INFO 08-06 10:08:29 [utils.py:297] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
qwen3-8b-1 | (APIServer pid=1) INFO 08-06 10:08:29 [utils.py:297]
qwen3-8b-1 | (APIServer pid=1) INFO 08-06 10:08:29 [utils.py:233] non-default args: {'model_tag': '/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf', 'model': '/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf', 'tokenizer': '/models/Qwen/Qwen3-8B', 'hf_config_path': '/models/Qwen/Qwen3-8B', 'max_model_len': 16384, 'served_model_name': ['Qwen/Qwen3-8B'], 'gpu_memory_utilization': 0.79, 'kv_cache_dtype': 'fp8', 'kv_offloading_size': 16.0, 'max_num_seqs': 16, 'disable_hybrid_kv_cache_manager': True, 'disable_log_stats': True}
qwen3-8b-1 | (APIServer pid=1) INFO 08-06 10:08:41 [model.py:533] Resolved architecture: Qwen3ForCausalLM
qwen3-8b-1 | (APIServer pid=1) INFO 08-06 10:08:41 [model.py:1582] Using max model len 16384
qwen3-8b-1 | (APIServer pid=1) INFO 08-06 10:08:41 [cache.py:212] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor.
qwen3-8b-1 | (APIServer pid=1) INFO 08-06 10:08:41 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048.
qwen3-8b-1 | (APIServer pid=1) INFO 08-06 10:08:41 [vllm.py:754] Asynchronous scheduling is enabled.
qwen3-8b-1 | (APIServer pid=1) WARNING 08-06 10:08:41 [interface.py:525] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
qwen3-8b-1 | (EngineCore pid=110) INFO 08-06 10:08:46 [core.py:103] Initializing a V1 LLM engine (v0.18.0) with config: model='/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf', speculative_config=None, tokenizer='/models/Qwen/Qwen3-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=gguf, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gguf, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3-8B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
qwen3-8b-1 | (EngineCore pid=110) WARNING 08-06 10:08:46 [interface.py:525] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
qwen3-8b-1 | (EngineCore pid=110) INFO 08-06 10:08:46 [parallel_state.py:1395] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.18.0.2:54095 backend=nccl
qwen3-8b-1 | (EngineCore pid=110) INFO 08-06 10:08:46 [parallel_state.py:1717] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
qwen3-8b-1 | (EngineCore pid=110) INFO 08-06 10:08:47 [gpu_model_runner.py:4481] Starting to load model /models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf...
qwen3-8b-1 | (EngineCore pid=110) INFO 08-06 10:08:58 [cuda.py:317] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
qwen3-8b-1 | (EngineCore pid=110) :1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
qwen3-8b-1 | (EngineCore pid=110) :1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
qwen3-8b-1 | (EngineCore pid=110) INFO 08-06 10:09:05 [gpu_model_runner.py:4566] Model loading took 4.82 GiB memory and 17.757037 seconds

Just wondering if this is caused by the plugin-based GGUF support?