dmoe from hf

[pretidav]

It is not clear if it is possible to convert a MOE model from hf, say Mixtral 8x7B into dmoe layers and exploit expert parallelism. Is it possible to start from such architecture, or the model must be defined from scratch (as in "testing_moe.yaml" ?
thank you.

[dakinggg]

It should be possible, although we don't have first class support for doing the conversion. You would need to (1) determine the MPT architecture options that match the mixtral 8x7B architecture, and (2) offline (or write code to do it online) convert the mixtral checkpoint into that architecture.