Example of MOE

[Juanhui28]

Hi,

Thanks for the amazing repo!

I want to use the MOE but cannot find an exmple. Is it possible to provide an tutorial/example to show how to use moe? For example, how to define main, training and test functions. DO we need to combine with multi-processing?

Thanks!

[Hannibal046]

Relevant issue: pytorch/pytorch#126616

I want to confirm if fairscale/FSDP support sparse training?

UPDATE: FSDP supports MoE training.