[Feature-request] Add Megatron Core support Gemma 4 dense model (e.g. E4B)

[DOGEUNNKIM]

Is your feature request related to a problem? Please describe.

Megatron-LM does not currently support Gemma 4 dense models such as Gemma 4 E4B. This makes it difficult to load, validate, and train Gemma 4 dense checkpoints in Megatron Core while preserving Hugging Face parity.

Describe the solution you'd like

Based on the scope discussion in this issue, the implementation is split across two layers:

• Megatron Core (generic): GEGLU support and any generic hooks needed for Gemma 4 architectures, kept default-off so existing GPT models are unaffected. - tracked in [Feature-request] Add Megatron Core support geglu for Gemma 4 dense model (e.g. E4B) #5090
• Megatron-Bridge (Gemma-specific): Dual RoPE, per-layer embeddings, shared KV wiring, HF checkpoint conversion, parity checks, and example scripts — tracked in feat(model): add Gemma-4 E4B support (layer spec, checkpoint loader, parity check) NVIDIA-NeMo/Megatron-Bridge#4148.

Describe alternatives you've considered

One alternative is to keep Gemma 4 support outside Megatron-LM as a downstream fork. However, that makes checkpoint conversion, validation, and maintenance harder, especially as Megatron Core APIs evolve.

Additional context

A working implementation covering all Gemma-specific components is available in NVIDIA-NeMo/Megatron-Bridge#4148, including:

• Gemma 4 unit tests and HF layer/block parity checks
• Converted Megatron checkpoint parity against the original HF Gemma 4 E4B model
• TP1, TP2 parity checks for converted checkpoints

The concrete implementation review is tracked in #5090.

This issue serves as the feature tracking item.

[DOGEUNNKIM]

I opened a draft PR with an initial implementation: #5090

The PR focuses on Gemma 4 dense model support, with Gemma 4 E4B as the initial validated target.

[Connor-XY]

Hi @DOGEUNNKIM, thanks for filing this and for already opening #5090 with validation.

I looked at the linked PR. The main MCore issue is not whether Gemma 4 dense support is useful, but where the boundary should sit: MCore should keep generic hooks / GEGLU support, while Gemma-specific layer specs, dual RoPE, PLE/shared-KV wiring, conversion schema, and examples likely belong in Megatron-Bridge. The PR discussion already moved in that direction with NVIDIA-NeMo/Megatron-Bridge#4148, leaving the generic MCore pieces under review.

I assigned this issue to @yaoyu-33, who is already driving the MCore-vs-Bridge scope decision on #5090. We should use #5090 for the concrete implementation review and keep this issue as the feature tracking item.

[DOGEUNNKIM]

Hi @DOGEUNNKIM, thanks for filing this and for already opening #5090 with validation.

I looked at the linked PR. The main MCore issue is not whether Gemma 4 dense support is useful, but where the boundary should sit: MCore should keep generic hooks / GEGLU support, while Gemma-specific layer specs, dual RoPE, PLE/shared-KV wiring, conversion schema, and examples likely belong in Megatron-Bridge. The PR discussion already moved in that direction with NVIDIA-NeMo/Megatron-Bridge#4148, leaving the generic MCore pieces under review.

I assigned this issue to @yaoyu-33, who is already driving the MCore-vs-Bridge scope decision on #5090. We should use #5090 for the concrete implementation review and keep this issue as the feature tracking item.

Thanks @Connor-XY, understood.

I'll keep this issue as the feature tracking item and continue the concrete implementation review in #5090.

I've moved the Gemma4-specific pieces into NVIDIA-NeMo/Megatron-Bridge#4148, and will keep #5090 focused on the generic MCore pieces needed by Bridge. I'll continue working with @yaoyu-33 on the MCore-vs-Bridge boundary and update #5090 accordingly.

I've also updated the issue description to reflect the scope split (MCore generic hooks vs. Bridge Gemma-specific implementation).

Thanks for clarifying the expected scope.