V0.1.1 Released Today!

[FrankLeeeee]

What's Changed

Features

• [MOE] changed parallelmode to dist process group by @1SAA in [MOE] changed parallelmode to dist process group #460
• [MOE] redirect moe_env from global_variables to core by @1SAA in [MOE] redirect moe_env from global_variables to core #467
• [zero] zero init ctx receives a dp process group by @ver217 in [zero] zero init ctx receives a dp process group #471
• [zero] ZeRO supports pipeline parallel by @ver217 in [zero] ZeRO supports pipeline parallel #477
• add LinearGate for MOE in NaiveAMP context by @1SAA in add LinearGate for MOE in NaiveAMP context #480
• [zero] polish sharded param name by @feifeibear in [zero] polish sharded param name #484
• [zero] sharded optim support hybrid cpu adam by @ver217 in [zero] sharded optim support hybrid cpu adam #486
• [zero] polish sharded optimizer v2 by @ver217 in [zero] polish sharded optimizer v2 #490
• [MOE] support PR-MOE by @1SAA in [MOE] support PR-MOE #488
• [zero] sharded model manages ophooks individually by @ver217 in [zero] sharded model manages ophooks individually #492
• [MOE] remove old MoE legacy by @1SAA in [MOE] remove old MoE legacy #493
• [zero] sharded model support the reuse of fp16 shard by @ver217 in [zero] sharded model support the reuse of fp16 shard #495
• [polish] polish singleton and global context by @feifeibear in [polish] polish singleton and global context #500
• [memory] add model data tensor moving api by @feifeibear in [memory] add model data tensor moving api #503
• [memory] set cuda mem frac by @feifeibear in [memory] set cuda mem frac #506
• [zero] use colo model data api in sharded optimv2 by @feifeibear in [zero] use colo model data api in sharded optimv2 #511
• [MOE] add MOEGPT model by @1SAA in [MOE] add MOEGPT model #510
• [zero] zero init ctx enable rm_torch_payload_on_the_fly by @ver217 in [zero] zero init ctx enable rm_torch_payload_on_the_fly #512
• [zero] show model data cuda memory usage after zero context init. by @feifeibear in [zero] show model data cuda memory usage after zero context init. #515
• [log] polish disable_existing_loggers by @ver217 in [log] polish disable_existing_loggers #519
• [zero] add model data tensor inline moving API by @feifeibear in [zero] add model data tensor inline moving API #521
• [cuda] modify the fused adam, support hybrid of fp16 and fp32 by @Gy-Lu in [cuda] modify the fused adam, support hybrid of fp16 and fp32 #497
• [zero] refactor model data tracing by @feifeibear in [zero] refactor model data tracing #522
• [zero] added hybrid adam, removed loss scale in adam by @Gy-Lu in [zero] added hybrid adam, removed loss scale in adam #527

Bug Fix

• fix discussion buttom in issue template by @binmakeswell in fix discussion buttom in issue template #504
• [zero] fix grad offload by @feifeibear in [zero] fix grad offload #528

Unit Testing

• [MOE] add unitest for MOE experts layout, gradient handler and kernel by @1SAA in [MOE] add unitest for MOE experts layout, gradient handler and kernel #469
• [test] added rerun on exception for testing by @FrankLeeeee in [test] added rerun on exception for testing #475
• [zero] fix init device bug in zero init context unittest by @feifeibear in [zero] fix init device bug in zero init context unittest #516
• [test] fixed rerun_on_exception and adapted test cases by @FrankLeeeee in [test] fixed rerun_on_exception and adapted test cases #487

CI/CD

• [devops] remove tsinghua source for pip by @FrankLeeeee in [devops] remove tsinghua source for pip #505
• [devops] remove tsinghua source for pip by @FrankLeeeee in [devops] remove tsinghua source for pip #507
• [devops] recover tsinghua pip source due to proxy issue by @FrankLeeeee in [devops] recover tsinghua pip source due to proxy issue #509

Documentation

• [doc] update rst by @ver217 in [doc] update rst #470
• Update Experiment result about Colossal-AI with ZeRO by @Sze-qq in Update Experiment result about Colossal-AI with ZeRO #479
• [doc] docs get correct release version by @ver217 in [doc] docs get correct release version #489
• Update README.md by @fastalgo in Update README.md #514
• [doc] update apidoc by @ver217 in [doc] update apidoc #530

Model Zoo

• [model zoo] fix attn mask shape of gpt by @ver217 in [model zoo] fix attn mask shape of gpt #472
• [model zoo] gpt embedding remove attn mask by @ver217 in [model zoo] gpt embedding remove attn mask #474

Miscellaneous

• [install] run with out rich by @feifeibear in [install] run with out rich #513
• [refactor] remove old zero code by @feifeibear in [refactor] remove old zero code #517
• [format] polish name format for MOE by @feifeibear in [format] polish name format for MOE #481

New Contributors

• @fastalgo made their first contribution in Update README.md #514

Full Changelog: v0.1.1...v0.1.0

---

This discussion was created from the release V0.1.1 Released Today!.