[QUESTION] Has standalone_embedding_stage been supported yet in core?

[JiwenJ]

I met an issue and want to split the embedding layer out of transformer block to make it alone in single pp stage, but I found that it has not been supported in core. Am I right?

Megatron-LM/megatron/core/transformer/transformer_block.py

Line 164 in e33c8f7

# # TODO: add back standalone_embedding_stage

[zhaoyang-star]

Same question. Keep tuned.
I also have another question: As the language model head has the same shape with embedding layer, does Megatron-LM support standalone the language model head as a single pp stage? It will save memory and improve training efficiency.