MoH: Multi-Head Attention as Mixture-of-Head Attention

[johndpope]

https://arxiv.org/html/2410.11842
https://wandb.ai/snoozie/IMF/runs/3f2e9892?nw=nwusersnoozie
https://github.com/johndpope/IMF/tree/feat/moh

trained to 3000 steps - ~ 10 mins

vanilla training now to compare
https://wandb.ai/snoozie/IMF/runs/vc26civb

[johndpope]

vanilla

VS MoH

pretty good - needs longer training - but this is an improvement.

[johndpope]

fyi - @hologerry