Falled to load Qwen3 model locally

[Linorman]

This the function that gives error: MLXLMCommon.ModelFactoryError error 0.
I call this function with type: 'qwen3'

func loadModelContext(modelConfiguration: ModelConfiguration, type: String) async throws -> ModelContext {
        let configurationURL = modelConfiguration.modelDirectory().appendingPathComponent("config.json")
        let baseConfig = try JSONDecoder().decode(BaseConfiguration.self, from: Data(contentsOf: configurationURL))
        let model = try LLMModelFactory.shared.typeRegistry.createModel(configuration: configurationURL, modelType: type)
        try loadWeights(modelDirectory: modelConfiguration.modelDirectory(), model: model, quantization: baseConfig.quantization)
        let tokenizer = try await loadTokenizer(configuration: modelConfiguration, hub: HubApi(downloadBase: modelConfiguration.modelDirectory()))
        let processor = LLMUserInputProcessor(tokenizer: tokenizer, configuration: modelConfiguration)
        return ModelContext(configuration: modelConfiguration, model: model, processor: processor, tokenizer: tokenizer)
    }

When I use Qwen2 model and change type to "qwen2", everything works well, so I can't figure out what's going on, and I am sure I use the latest version of mlx-examples (main branch).

[davidkoski]

That looks ok to me. I think you are seeing this error:

public enum ModelFactoryError: LocalizedError {
    case unsupportedModelType(String)

I presume from this line:

        let model = try LLMModelFactory.shared.typeRegistry.createModel(configuration: configurationURL, modelType: type)

and in current main there are qwen3 types registered:

            "qwen3": create(Qwen3Configuration.self, Qwen3Model.init),
            "qwen3_moe": create(Qwen3MoEConfiguration.self, Qwen3MoEModel.init),

Two suggestions:

• can you use llm-tool or one of the example apps to load the model?
• what model (hugging face id) are you using?

[davidkoski]

Possibly #291 (see #293 )

[Linorman]

That looks ok to me. I think you are seeing this error:

public enum ModelFactoryError: LocalizedError {
case unsupportedModelType(String)
I presume from this line:

    let model = try LLMModelFactory.shared.typeRegistry.createModel(configuration: configurationURL, modelType: type)

and in current main there are qwen3 types registered:

        "qwen3": create(Qwen3Configuration.self, Qwen3Model.init),
        "qwen3_moe": create(Qwen3MoEConfiguration.self, Qwen3MoEModel.init),

Two suggestions:

• can you use llm-tool or one of the example apps to load the model?
• what model (hugging face id) are you using?

Thanks. I am using a finetuned model based on Qwen3-1.7B and I haven't uploaded it to huggingface by now. And I do use a quant model which is quanted with mlx framework using python. I load this model successfully using python and mlx.

[Linorman]

There is no huggingface id so I can't test using llm-tool

[davidkoski]

It should work if you do --model /path/to/model/directory

[Linorman]

It should work if you do --model /path/to/model/directory

Thanks, I tried and it loaded successfully. So there is something wrong with other codes?

[davidkoski]

Nothing obvious, but I would look at this part:

        let configurationURL = modelConfiguration.modelDirectory().appendingPathComponent("config.json")
        let baseConfig = try JSONDecoder().decode(BaseConfiguration.self, from: Data(contentsOf: configurationURL))
        let model = try LLMModelFactory.shared.typeRegistry.createModel(configuration: configurationURL, modelType: type)

Typically you get the type from baseConfig.modelType. You might try printing the value being passed in vs what is in baseConfig. I suspect that type is simply wrong.

[Linorman]

Nothing obvious, but I would look at this part:

    let configurationURL = modelConfiguration.modelDirectory().appendingPathComponent("config.json")
    let baseConfig = try JSONDecoder().decode(BaseConfiguration.self, from: Data(contentsOf: configurationURL))
    let model = try LLMModelFactory.shared.typeRegistry.createModel(configuration: configurationURL, modelType: type)

Typically you get the type from baseConfig.modelType. You might try printing the value being passed in vs what is in baseConfig. I suspect that type is simply wrong.

It's a little bit tricky. I changed type to baseConfig.modelType instead but it still gives the same error. I checked the dependencies again and I am sure I use the correct version.

Do I have other ways to load model locally?

[davidkoski]

OK, what is the value of the type/modelType? That needs to match this list:

• https://github.com/ml-explore/mlx-swift-examples/blob/main/Libraries/MLXLLM/LLMModelFactory.swift#L31

[Linorman]

OK, what is the value of the type/modelType? That needs to match this list:

• https://github.com/ml-explore/mlx-swift-examples/blob/main/Libraries/MLXLLM/LLMModelFactory.swift#L31

qwen3, it's a finetuned model based on Qwen3-1.7B, this is the config file:

{
  "architectures": [
    "Qwen3ForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 2048,
  "initializer_range": 0.02,
  "intermediate_size": 6144,
  "max_position_embeddings": 40960,
  "max_window_layers": 28,
  "model_type": "qwen3",
  "num_attention_heads": 16,
  "num_hidden_layers": 28,
  "num_key_value_heads": 8,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000,
  "sliding_window": null,
  "tie_word_embeddings": true,
  "torch_dtype": "float16",
  "transformers_version": "4.51.3",
  "use_cache": true,
  "use_sliding_window": false,
  "vocab_size": 151936
}

[Linorman]

Ok, I find something strange, in the latest version of mlx-swift-examples, there is no asMessage() function in UserInput, and I still call this function in my codes but build without errors, so maybe here is the problem.