Gemma4-12B Training Failed

[AaronCaoZJ]

Reminder

• I have read the above rules and searched the existing issues.

System Info

• llamafactory version: 0.9.6.dev0
• Platform: Linux-5.15.0-181-generic-x86_64-with-glibc2.35
• Python version: 3.12.12
• PyTorch version: 2.8.0+cu129 (GPU)
• Transformers version: 5.12.1 (For Gemma4)
• Datasets version: 4.0.0
• Accelerate version: 1.11.0
• PEFT version: 0.18.1
• GPU type: NVIDIA H200 NVL
• GPU number: 8
• GPU memory: 139.80GB
• TRL version: 0.24.0
• DeepSpeed version: 0.18.4
• vLLM version: 0.11.0
• Git commit: af667a8
• Default data directory: detected

Reproduction

I adopted the gemma4n template and registered gemma4_unified for Gemma4-12B-it in ./src/llamafactory/model/model_utils/visual.py with the following code:

_register_composite_model(
    model_type="gemma4_unified",
    projector_keys=["model.embed_vision", "model.embed_audio"],
    vision_model_keys=["vision_embedder"],
    lora_conflict_keys=["per_layer_projection_norm"],
)

The parameters were set as enable_thinking: false and freeze_vision_tower: true. I conducted LoRA instruction tuning on a dataset consisting of 2,000 samples, where each sample input contained 2 images plus a prompt, corresponding to 9 specific word outputs.

I found that the loss failed to decrease. Evaluation show that the model keeps outputting repeated strings of "MV_MV_MV_ ...", while MV_FWD / MV_LEFT / ... are my expected outputs.

By contrast, Qwen3.5-9B could converge properly on datasets of the same size when trained via LlamaFactory.

Could you help me check if there is an issue with model registration, whether special configurations are required for training Gemma4, or if there is a bug in the template within the repo?

Thanks!

Others

No response