Text-only architectures implemented only in mlx_vlm (e.g. deepseek_v4) fail to load because dispatch checks only for vision_config

[True2456]

Summary

For a text-only model, mlx_engine/generate.py's model-kit selection dispatches purely on "vision_config" in config_json (around load_model, the is_batchable() branch). If that key is absent, it unconditionally calls mlx_lm_load(model_path, lazy=True) — even when the architecture in question has no mlx_lm implementation at all, but does have one in mlx_vlm, bundled in the exact same environment.

Concretely, for deepseek_v4 (a genuinely text-only architecture — no vision component): the bundled environment's mlx_lm/models/ stops at deepseek_v32, so loading fails outright. The same bundled environment's mlx_vlm/models/deepseek_v4/ is fully implemented (including an MTP speculative-decoding drafter), but it's never reached, because the dispatch never considers mlx_vlm for a model that lacks vision_config.

Repro

Load any local MLX checkpoint with model_type: deepseek_v4 (text-only, no vision_config key) via lms load or the GUI.

Result:

ValueError: Model type deepseek_v4 not supported.

At:
  .../mlx_lm/utils.py(192): _get_classes
  .../mlx_lm/utils.py(348): load_model
  .../mlx_lm/utils.py(509): load
  .../mlx_engine/generate.py(289): is_batchable

Workaround found

Adding an empty "vision_config": {} to the checkpoint's config.json routes the same load through the BatchedVisionModelKit → mlx_vlm path instead, which has the architecture and loads/generates correctly (confirmed end-to-end, including at extended context lengths). mlx_vlm.utils.load_image_processor degrades gracefully for a model class with no ImageProcessor attribute, so this doesn't require faking any real vision capability — it's purely a dispatch trick, and a fragile one for users who don't know to do it.

Suggested fix

Before deciding "no vision_config → use mlx_lm", check whether mlx_lm actually has an implementation for the model's model_type (a cheap _get_classes-style lookup) and fall back to mlx_vlm if not, rather than dispatching solely on the presence of a config key that's orthogonal to which package actually implements the architecture. This matters more as new architectures increasingly land in mlx_vlm first even when text-only.

Environment

• LM Studio 0.4.20 (build 1), macOS
• mlx-llm backend (nax-advsimd), engine build app-mlx-generate-mac26-arm64@33
• Model: local MLX conversion of deepseek-ai/DeepSeek-V4-Flash (deepseek_v4 architecture, text-only)
• Related: Manual context-length input capped at 2048 for architectures outside the auto-fit validated family list (e.g. deepseek_v4) #356 (the same underlying gap also causes the manual context-length control to be capped, since the auto-fit path that would otherwise report a real value is never reached for this architecture either)