[Feautre] Sophia Optimizer

[namgyu-youn]

Hi there, i am Youn from South Korea. I have been used Shampoo, and thought the project is interesting.

So, i want to implement Sophia : A Scalable Stochastic Second-order Optimizer.

Before the start, i want to ask some opinions about this. Is there something should i care?

[tsunghsienlee]

Before the start, i want to ask some opinions about this. Is there something should i care?

Hi @namgyu-youn ,

Would you mind to elaborate more on the relation between implementing Sophia with this repo?

[namgyu-youn]

@tsunghsienlee Thanks for your comment.

At now, Shampoo is implemented with Adam. But in my experience, Sophia is better optimizer in LLM. It is initially applied into GPT-2.

Sophia explicitly uses Hessian, and process identical steps for optimization. (e.g. Clipping, Optimized parameters for LLM)

As a result, Sophia is more efficient optimizer in LLM domain (memory-efficient & better-performance). So i thought Sophia with Shampoo could be useful feature. (maybe it would be names as SophiaGraftingConfig)

[tsunghsienlee]

As a result, Sophia is more efficient optimizer in LLM domain (memory-efficient & better-performance). So i thought Sophia with Shampoo could be useful feature. (maybe it would be names as SophiaGraftingConfig)

I assume you meant using Sophia as the grafting optimizer rather than Adam based on the referring of SophiaGraftingConfig. Personally I thought that would be an interesting idea considering the importance of grafting. However, our efforts now is more prone to removing the need of grafting, like https://github.com/facebookresearch/optimizers/tree/main/distributed_shampoo#example-5-eigenvalue-corrected-shampoosoap because it is a more promising direction in the long run. However, if you are interested in implementing SophiaGraftingConfig, and would like people to review. I will be happy to review that.

[namgyu-youn]

I assume you meant using Sophia as the grafting optimizer rather than Adam based on the referring of SophiaGraftingConfig. Personally I thought that would be an interesting idea considering the importance of grafting. However, our efforts now is more prone to removing the need of grafting, like https://github.com/facebookresearch/optimizers/tree/main/distributed_shampoo#example-5-eigenvalue-corrected-shampoosoap because it is a more promising direction in the long run. However, if you are interested in implementing SophiaGraftingConfig, and would like people to review. I will be happy to review that.

Thanks for your feedback. I have been practicing implementation for Adam, Sophia.

Also, i love being reviewed. Thanks for your feedback again. I will try it and submit PR soon!