Does LORA works with Qwen3.5, when using the Megatron backend?

[shamanez]

I do get initially errors.

• Hybrid architecture compatibility with mcore_adapter: Qwen3.5's hybrid architecture (full attention every 4 layers + GDN linear attention + Mamba SSM) is non-standard. The apply_megatron_lora() function in mcore_adapter was designed for standard transformer models (Qwen2.5). The GDN and Mamba layers may not be properly recognized or adapted.

• all-linear expansion: When lora_target: all-linear is used, find_all_linear_modules() auto-discovers linear layers. For Qwen3.5's hybrid layers (GDN projections like in_proj_qkv, in_proj_z, in_proj_b, in_proj_a), it's unclear if these get correctly identified and wrapped with LoRA adapters by the Megatron backend.

• VLM wrapper: Qwen3.5 loads as Qwen3_5ForConditionalGeneration (VLM) — LoRA needs to target only the text model, not the vision encoder. We use freeze_module_prefix: vision_model but need to verify this interacts correctly with the LoRA setup path

[chocoded]

Thanks for the detailed report.

1. Megatron backend support for Qwen3.5 + LoRA
   This is not fully supported yet and is still being adapted. The main items under verification are whether LoRA can be correctly applied to the GDN and stacked MoE parts of Qwen3.5, and whether the related format conversion works correctly.

2. all-linear expansion of Qwen3.5 GDN projection layers
   Yes. In the Megatron implementation, these layers are implemented as TELayerNormColumnParallelLinear and TERowParallelLinear, so they can be detected by all-linear.

3. Whether LoRA is injected only into the text model
   Yes. The ViT part does not use Megatron modules, so all-linear does not include the vision encoder, regardless of whether freeze_module_prefix is set.

In short: all-linear should cover the GDN linear layers, and the vision encoder is not part of the Megatron LoRA path. The remaining gap is full Qwen3.5 support in the Megatron backend, which is still in progress.

[shamanez]

Thanks , do you think Qwen3.5 lora would work with FSDP ?