[QST] Clarification on `mup_width_multiplier`, `base width`, and `init_std` Values

[Yanksi]

Hi, I have a couple of questions regarding the variable mup_width_multiplier and related initialization choices in the code:

Regarding mup_width_multiplier:
Based on the paper and code comments, it seems mup_width_multiplier should always be the width divided by the base width, where the base width is set to 256. However, I noticed that this multiplier is only set within the .sh training scripts, instead of being computed directly as n_embd / 256 in the code. Is there a specific reason for this approach? Are there any corner cases or considerations that require handling this value outside the main code?

On the selection of base width and init_std:
The values for base width and the base standard deviation init_std (which is always set to 0.02) seem somewhat arbitrary. From my understanding, to keep the output of the layer normalized, init_std should be 1/sqrt(base_width), which would be 0.0625 in this case. Could you clarify the reasoning behind choosing 0.02 for init_std and the specific value for base width?

Any clarification on these points would be greatly appreciated!

[Quentin-Anthony]

Regarding mup_width_multiplier:

This is just personal choice. There's nothing special about the base width of 256 (chosen because it's large enough to show signal, but small enough to be low-overhead on our hardware). Passing it as an arg in the run script seems the most flexible to me.

On the selection of base width and init_std:

For base_width it's the same answer as above. It's empirical for this model size + hardware combo, but if I had more hparam sweeping budget I'd use a base_width as large as possible and for as many tokens as possible.

For init_std, we chose 0.02 to match the original GPT-2 init std. muP is more about how init_std scales with width rather than its pre-scaled value, and we scale the init_std by 1/sqrt(mup_width_multiplier) at

nanoGPT-mup/model.py

Line 166 in b2a5e60

torch.nn.init.normal_(p, mean=0.0, std=config.init_std / math.sqrt(config.mup_width_multiplier))

Nevertheless, I've heard of the 1/sqrt(base_width) suggestion. Lots of people claim very different behavior here (I think because this is a large-scale phenomenon and everyone is using wildly different combos of model arch, data, etc), so I suggest you read what everyone is doing and decide for yourself what's most sensible in your usecase:

• https://cloneofsimo.notion.site/What-to-do-to-scale-up-09e469d7c3444d6a90305397c38a46f5
• https://howtoscalenn.github.io/