Misleading server logs during GGUF model loading

[Isotr0py]

• Currently, when serving a GGUF model, the model loading logs are quite misleading because it displays original models, but actually quantized models are loaded:

(EngineCore pid=18929) INFO 05-11 17:24:06 [core.py:109] Initializing a V1 LLM engine (v0.20.2rc1.dev229+g3f9c0c25b) with config: model='Qwen/Qwen3-0.6B', speculative_config=None, tokenizer='Qwen/Qwen3-0.6B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=40960, download_dir=None, load_format=gguf, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gguf, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3-0.6B-GGUF:Q8_0, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=False, moe_backend='auto')
(EngineCore pid=18929) INFO 05-11 17:24:06 [parallel_state.py:1410] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.19.2.2:34569 backend=nccl
[W511 17:24:06.077691980 socket.cpp:207] [c10d] The hostname of the client socket cannot be retrieved. err=-3
(EngineCore pid=18929) INFO 05-11 17:24:07 [parallel_state.py:1723] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=18929) INFO 05-11 17:24:08 [topk_topp_sampler.py:45] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=18929) INFO 05-11 17:24:08 [gpu_model_runner.py:4857] Starting to load model Qwen/Qwen3-0.6B...
Downloading (incomplete total...): 0.00B [00:00, ?B/s]                                                                              Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Fetching 1 files: 100%|████████████████████████████████████████████████████████████████████████████████| 1/1 [00:05<00:00,  5.47s/it]
Download complete: 100%|███████████████████████████████████████████████████████████████████████████| 639M/639M [00:05<00:00, 116MB/s]
(EngineCore pid=18929) ERROR 05-11 17:24:51 [fa_utils.py:171] Cannot use FA version 2 is not supported due to FA2 is only supported on devices with compute capability >= 8
(EngineCore pid=18929) INFO 05-11 17:24:51 [cuda.py:368] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=18929) INFO 05-11 17:25:02 [gpu_model_runner.py:4959] Model loading took 0.91 GiB memory and 53.869902 seconds

• It's because we're patching the engine args in a quite hacky way to bypass model config post-init validation:
  

  vllm-gguf-plugin/vllm_gguf_plugin/plugin.py

  Lines 37 to 82 in d6be5c1

  	def _get_gguf_config_source(
  	model: str,
  	tokenizer: str | None,
  	hf_config_path: str | None,
  	) -> str:
  	if hf_config_path is not None:
  	return hf_config_path
  	if tokenizer is not None and not _is_gguf_reference(tokenizer):
  	return tokenizer
  	if is_remote_gguf(model):
  	repo_id, _ = split_remote_gguf(model)
  	return repo_id
  	if check_gguf_file(model):
  	return str(Path(model).parent)
  	return model
  	def _patch_engine_args() -> None:
  	if getattr(EngineArgs, "_gguf_create_model_config_patched", False):
  	return
  	original_create_model_config = EngineArgs.create_model_config
  	@wraps(original_create_model_config)
  	def create_model_config(self, *args, **kwargs):
  	if _is_gguf_reference(self.model):
  	gguf_model = self.model
  	if self.quantization is None:
  	self.quantization = "gguf"
  	if self.load_format == "auto":
  	self.load_format = "gguf"
  	if self.config_format == "auto":
  	self.config_format = "gguf"
  	if not self.model_weights:
  	self.model_weights = gguf_model
  	if self.served_model_name is None:
  	self.served_model_name = [gguf_model]
  	self.model = _get_gguf_config_source(
  	gguf_model,
  	self.tokenizer if isinstance(self.tokenizer, str) else None,
  	self.hf_config_path,
  	)
  	return original_create_model_config(self, *args, **kwargs)
  	EngineArgs.create_model_config = create_model_config
  	EngineArgs._gguf_create_model_config_patched = True

• We need to find out a better method to handle this.