Skip to content

A bunch of new optimizers and schedules #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 70 commits into
base: main
Choose a base branch
from
Open

A bunch of new optimizers and schedules #21

wants to merge 70 commits into from

Conversation

Andron00e
Copy link
Collaborator

Here we are implementing a new methods for the llm-baselines. A complete list of the things is here and in the new README.md of the soap branch.

Andron00e and others added 30 commits October 15, 2024 13:50
@Andron00e
soap and muon has been added. also black + isort everything
ea9f4d0
@Andron00e
-fix annotations
d5153c4
@Andron00e
soap is ready, muon needs to be done
5dcc69a
@Andron00e
AdEMAMix and Lion is here, Muon still TODO
49b802d
@Andron00e
AdEMAMix and Lion is here, Muon still TODO
3d8b24a
@Andron00e
Schedule-Free AdamW is here, warmup_percent -> warmup_steps / iterations
a047786
@mpagli
eval on a fix subset + better lr decay
66b77f5
@Andron00e
Schedule-Free SGD + AdamW are here
7f158a3
@Andron00e
Schedule-Free SGD + AdamW are here
1f67f6d
@Andron00e
Merge branch 'soap' into small-upgrades
d3082e2
@Andron00e
Merge pull request #19 from epfml/small-upgrades
5fb1d57 
Eval on a fix subset + better lr decay
@mpagli
push to wandb team + display grad norm
7378449
@mpagli
Merge branch 'small-upgrades' of https://github.com/epfml/llm-baselines…
156fdab 
… into small-upgrades
@Andron00e
Merge pull request #20 from epfml/small-upgrades
6d8a47e 
Displaying grad-norm + support wandb with teams
@Andron00e
a code for schedules is here
2b0e71f
@Andron00e
new schedules
5d43a53
@Andron00e
cos_inf and wsd schedules are here
a908d96
@Andron00e
codestyle
2e4bc4e
@Andron00e
--fix in readme
c5345d6
@mpagli
fix grad norm display
9b2c2e6
@Andron00e
removed warmup_percent argument
e142dfc
@Andron00e
schedule-free fix, added scheduer check in optim/base.py
513b902
@Andron00e
--fix saving of a checkpoint if scheduler==none
8e2aa6d
@Andron00e
fix requirements
eda40bf
@Andron00e
Adam-mini is here
3649936
@mpagli
refactoring
e75b3be
@mpagli
Merge branch 'soap' into ademamix
a624c2a
@Andron00e
reviewed
5876db9
@Andron00e
Merge pull request #22 from epfml/ademamix
5bfbe3a 
Refactoring + reproducing AdEMAMix
@Andron00e
extra_args removed
94ca273
Andron00e and others added 28 commits November 5, 2024 15:10
@Andron00e
finewebedu
c8e6fd8
@Andron00e
sophia is in the process
99d7af6
@Andron00e
shampoo is here, needs major improvements (very memory- and time- con…
fd9aa7e 
…suming due to the torch.inverse)
@Andron00e
shampoo is here, problems with memory due to torch.inverse, needs imp…
479b56b 
…rovements
@Andron00e
muon is here, still fix shampoo
c811739
@Andron00e
--fix sophiag
361213d
@Andron00e
sophiag fixed, test two adamw runs using muon and soap branches, if t…
2b9d0f3 
…hey are the same, then merge into soap
@Andron00e
Merge pull request #24 from epfml/muon
e7780d6 
Muon
@Andron00e
--adopt todo, --fix sophia
19ff7b1
@Andron00e
clipped version are here, muon schedules todo
6709eb6
@Andron00e
updates in muon
0f05b19
@Andron00e
-- micro
729c820
@Andron00e
implemented a scheduler for muon
d81d0b2
@Andron00e
new scheduler, double decay, test it
6a8a159
@Andron00e
minor
c30eeca
@Andron00e
adopt is here
82dd1af
@Andron00e
adopt fix
755cf59
@Andron00e
adopt fix again
0c53df3
@Andron00e
adopt again
6cfd01b
@Andron00e
mars is here, ready to try
d4585ae
@Andron00e
--fix info
9700a50
@Andron00e
--small changes in mars train
29ce41b
@Andron00e
adafactor and lamb
aaff7fc
@Andron00e
--fix adopt
7ece053
@Andron00e
Merge pull request #25 from epfml/soap
664d2df 
soap --> muon
@Andron00e
muon-debug
10a2a3a
@Andron00e
--signum fix
4f5c061
@Andron00e
Merge pull request #26 from epfml/muon
5d63141 
Signum check
martinjaggi
martinjaggi approved these changes Nov 25, 2024
View reviewed changes
Copy link
Member

@martinjaggi martinjaggi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks a lot, this is looking quite impressive already

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@martinjaggi martinjaggi martinjaggi approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Andron00e @mpagli @martinjaggi