Unable to run inference with VibeVoice-Large Q4

[Puncia]

I am unable to load the quantized VibeVoice TTS model. But what's curious is that I am able to run the full precision model just fine, albeit very slowly because it offloads part of it to system RAM.

Important note: I am using a Pascal GPU (sm_61), therefore some software versions may be different from the standard ones (see below).

Full error + traceback

[ComfyUI-VibeVoice] Loading model 'VibeVoice-Large' with dtype: torch.bfloat16 and attention: 'sdpa'
We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
[ComfyUI-VibeVoice] Failed to load model 'VibeVoice-Large' with sdpa attention: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set `llm_int8_enable_fp32_cpu_offload=True` and pass a custom `device_map` to `from_pretrained`. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details. 
!!! Exception during processing !!! Failed to load model even with eager attention: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set `llm_int8_enable_fp32_cpu_offload=True` and pass a custom `device_map` to `from_pretrained`. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details. 
Traceback (most recent call last):
  File "C:\Users\puncia\ComfyUI\custom_nodes\ComfyUI-VibeVoice\modules\loader.py", line 192, in load_model
    model = VibeVoiceForConditionalGenerationInference.from_pretrained(model_path_or_none, **from_pretrained_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\puncia\ComfyUI\.venv\Lib\site-packages\transformers\modeling_utils.py", line 279, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\puncia\ComfyUI\.venv\Lib\site-packages\transformers\modeling_utils.py", line 4380, in from_pretrained
    device_map = _get_device_map(model, device_map, max_memory, hf_quantizer, torch_dtype, keep_in_fp32_regex)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\puncia\ComfyUI\.venv\Lib\site-packages\transformers\modeling_utils.py", line 1304, in _get_device_map
    hf_quantizer.validate_environment(device_map=device_map)
  File "C:\Users\puncia\ComfyUI\.venv\Lib\site-packages\transformers\quantizers\quantizer_bnb_4bit.py", line 104, in validate_environment
    raise ValueError(
ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set `llm_int8_enable_fp32_cpu_offload=True` and pass a custom `device_map` to `from_pretrained`. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details. 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\puncia\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\execution.py", line 496, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\puncia\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\execution.py", line 315, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\puncia\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\execution.py", line 289, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "C:\Users\puncia\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\execution.py", line 277, in process_inputs
    result = f(**inputs)
             ^^^^^^^^^^^
  File "C:\Users\puncia\ComfyUI\custom_nodes\ComfyUI-VibeVoice\vibevoice_nodes.py", line 107, in generate_audio
    model_management.load_model_gpu(patcher)
  File "C:\Users\puncia\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\comfy\model_management.py", line 688, in load_model_gpu
    return load_models_gpu([model])
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\puncia\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\comfy\model_management.py", line 683, in load_models_gpu
    loaded_model.model_load(lowvram_model_memory, force_patch_weights=force_patch_weights)
  File "C:\Users\puncia\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\comfy\model_management.py", line 494, in model_load
    self.model_use_more_vram(use_more_vram, force_patch_weights=force_patch_weights)
  File "C:\Users\puncia\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\comfy\model_management.py", line 523, in model_use_more_vram
    return self.model.partially_load(self.device, extra_memory, force_patch_weights=force_patch_weights)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\puncia\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\comfy\model_patcher.py", line 856, in partially_load
    self.patch_model(load_weights=False)
  File "C:\Users\puncia\ComfyUI\custom_nodes\ComfyUI-VibeVoice\modules\patcher.py", line 32, in patch_model
    self.model.load_model(target_device, self.attention_mode)
  File "C:\Users\puncia\ComfyUI\custom_nodes\ComfyUI-VibeVoice\modules\loader.py", line 75, in load_model
    self.model, self.processor = VibeVoiceLoader.load_model(self.model_pack_name, device, attention_mode, use_llm_4bit=self.use_llm_4bit)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\puncia\ComfyUI\custom_nodes\ComfyUI-VibeVoice\modules\loader.py", line 212, in load_model
    raise RuntimeError(f"Failed to load model even with eager attention: {e}")
RuntimeError: Failed to load model even with eager attention: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set `llm_int8_enable_fp32_cpu_offload=True` and pass a custom `device_map` to `from_pretrained`. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details.

With the help of AI I tried the following things:

1. adding bnb_4bit_quant_storage, device_map="auto" in loader.py
2. self.memory_required = lambda *args, **kwargs: 0 in patcher.py inside __init__.
3. use --force-fp32, --normalvram, --disable-smart-memory as ComfyUI flags

without meaningful results.

>> python --version
Python 3.12.11

>> python -c "import torch; print(f'Torch Version: {torch.__version__}'); print(f'CUDA Available: {torch.cuda.is_available()}'); print(f'CUDA Version: {torch.version.cuda}'); print(f'cuDNN Version: {torch.backends.cudnn.version()}')"
Torch Version: 2.4.0+cu118
CUDA Available: True
CUDA Version: 11.8
cuDNN Version: 90100

>> python -c "import torch; print(torch.cuda.get_device_name(0))"
NVIDIA GeForce GTX 1060 6GB

>> pip list | findstr "bitsandbytes transformers accelerate torch comfy"   
accelerate                 1.6.0
bitsandbytes               0.47.0
comfyui-embedded-docs      0.2.6
comfyui-workflow-templates 0.1.78
torch                      2.4.0+cu118
torchaudio                 2.4.0+cu118
torchsde                   0.2.6
torchvision                0.19.0+cu118
transformers               4.51.3