DeepSeek-V3 not loading (ggml_type 21 error)

[Ji9i]

I have ktransformers (25/02/2025 update) installed on a machine with Linux Mint. DeepSeek-R1 works without issues with all the quants I have tried. Yet, DeepSeek-V3 doesn't work. When I try loading it with

ktransformers \
--model_path DeepSeek/DeepSeek-V3/ \
--gguf_path ../modelli/DeepSeek-V3/ \
--max_new_tokens 8192 \
--cache_lens 32768 \
--total_context 32768 \

I get the following error:

Injecting model.layers.60.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as ktransformers.operators.linear . KTransformersLinear
loading token_embd.weight to cpu
loading token_embd.weight with CPU
Traceback (most recent call last):
  File "/home/us/miniconda3/envs/ktransformers/bin/ktransformers", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/us/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/main.py", line 115, in main
    create_interface(config=cfg, default_args=cfg)
  File "/home/us/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/us/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 61, in __init__
    optimize_and_load_gguf(self.model, optimize_config_path, gguf_path, config)
  File "/home/us/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf
    load_weights(module, gguf_loader)
  File "/home/us/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 108, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/home/us/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 110, in load_weights
    module.load()
  File "/home/us/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load
    utils.load_weights(child, self.gguf_loader, self.key+".")
  File "/home/us/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 108, in load_weights
    load_weights(child, gguf_loader, prefix+name+".")
  File "/home/us/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 106, in load_weights
    load_cur_state_dict(module, gguf_loader, prefix)
  File "/home/us/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 96, in load_cur_state_dict
    weights = load_dequantized_tensor(translated_key, device=device).to(dtype=target_dtype)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/us/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/custom_gguf.py", line 351, in load_gguf_tensor
    raise NotImplementedError(f"ggml_type {ggml_type} not implemented")
NotImplementedError: ggml_type 21 not implemented

The file used, DeepSeek-V3.i1-IQ3_XL.gguf, works without issues on llama.cpp.

According to https://docs.rs/llama_cpp_sys/latest/llama_cpp_sys/enum.ggml_type.html, ggml_type 21 is for IQ3_S, not for IQ3_XS.

[Ji9i]

I have tried a different quant (Q4_K_S) and it works flawlessly. I guess the issue is the mismatch between ggml_type 21 and IQ3_XS.