Question regarding model config

[LeoXinhaoLee]

Hi, thank you so much for releasing code for this inspiring work. I noticed in lit_gt/config.py, for model "GatedDeltaNet_1.3B", n_head is set to 16. However, it seems like this value is never passed when constructing the key module: GatedDeltaNet(hidden_size=config.n_embd), so n_head is effectively 9.

I'm wondering would you mind sharingthe exact config of this model? Thank you very much for your time and help!

[sustcsonglin]

Sorry for the confusion. I used 9 heads in this experiment, and the head dimension should be 192.

[LeoXinhaoLee]

Thank you for your kind reply. Would you minding sharing the complete correct config for these two models so we could have a complete and exact reference? Thank you very much for your help.

dict(
        org="NVIDIA",
        name="GatedDeltaNet_1.3B",
        block_size=4096,
        vocab_size=32000,
        padding_multiple=64,
        gated_delta_per_layer=1,
        n_layer=16,
        n_head=16,
        n_embd=2400,
        rotary_percentage=1.0,
        parallel_residual=False,
        bias=False,
        _norm_class="FusedRMSNorm",
        norm_eps=1e-5,
        _mlp_class="LLaMAMLP",
        intermediate_size=5888,
        local_window = 2048,
        mamba_init = True,
    ),
    dict(
        org="NVIDIA",
        name="GatedDeltaNet_H1_1.3B",
        block_size=4096, 
        vocab_size=32000,
        padding_multiple=64,
        gated_delta_per_layer=2,
        n_layer=18,
        n_head=18,
        n_embd=2304,
        rotary_percentage=1.0,
        parallel_residual=False,
        bias=False,
        _norm_class="FusedRMSNorm",
        norm_eps=1e-5,
        _mlp_class="LLaMAMLP",
        intermediate_size=6144,
        local_window = 2048,
        mamba_init = True,
    ),