v0.1.6 Released!

[ver217]

Main features

1. ColoTensor supports hybrid parallel (tensor parallel and data parallel)
2. ColoTensor supports ZeRO (with chunk)
3. Config tensor parallel by module via ColoTensor
4. ZeroInitContext and ShardedModelV2 support loading checkpoint and hugging face from_pretrain()

What's Changed

ColoTensor

• [tensor] refactor colo-tensor by @ver217 in [tensor] refactor colo-tensor #992
• [tensor] refactor parallel action by @ver217 in [tensor] refactor parallel action #1007
• [tensor] impl ColoDDP for ColoTensor by @ver217 in [tensor] impl ColoDDP for ColoTensor #1009
• [Tensor] add module handler for linear by @Wesley-Jzy in [Tensor] add module handler for linear #1021
• [Tensor] add module check and bert test by @Wesley-Jzy in [Tensor] add module check and bert test #1031
• [Tensor] add Parameter inheritance for ColoParameter by @Wesley-Jzy in [Tensor] add Parameter inheritance for ColoParameter #1041
• [tensor] ColoTensor supports ZeRo by @ver217 in [tensor] ColoTensor supports ZeRo #1015
• [zero] add chunk size search for chunk manager by @ver217 in [zero] add chunk size search for chunk manager #1052

Zero

• [zero] add load_state_dict for sharded model by @ver217 in [zero] add load_state_dict for sharded model #894
• [zero] add zero optimizer for ColoTensor by @ver217 in [zero] add zero optimizer for ColoTensor #1046

Hotfix

• [hotfix] fix colo init context by @ver217 in [hotfix] fix colo init context #1026
• [hotfix] fix some bugs caused by size mismatch. by @YuliangLiu0306 in [hotfix] fix some bugs caused by size mismatch. #1011
• [kernel] fixed the include bug in dropout kernel by @FrankLeeeee in [kernel] fixed the include bug in dropout kernel #999
• fix typo in constants by @ryanrussell in fix typo in constants #1027
• [engine] fixed bug in gradient accumulation dataloader to keep the last step by @FrankLeeeee in [engine] fixed bug in gradient accumulation dataloader to keep the last step #1030
• [hotfix] fix dist spec mgr by @ver217 in [hotfix] fix dist spec mgr #1045
• [hotfix] fix import error in sharded model v2 by @ver217 in [hotfix] fix import error in sharded model v2 #1053

Unit test

• [unit test] refactor test tensor by @ver217 in [unit test] refactor test tensor #1005

CI

• [ci] update the docker image name by @FrankLeeeee in [ci] update the docker image name #1017
• [ci] added nightly build ([ci] added nightly build #1018) by @FrankLeeeee in [ci] added nightly build (#1018) #1019
• [ci] fixed nightly build workflow by @FrankLeeeee in [ci] fixed nightly build workflow #1022
• [ci] fixed nightly build workflow by @FrankLeeeee in [ci] fixed nightly build workflow #1029
• [ci] fixed nightly build workflow by @FrankLeeeee in [ci] fixed nightly build workflow #1040

CLI

• [cli] remove unused imports by @FrankLeeeee in [cli] remove unused imports #1001

Documentation

• Hotfix/format by @binmakeswell in Hotfix/format #987
• [doc] update docker instruction by @FrankLeeeee in [doc] update docker instruction #1020

Misc

• [NFC] Hotfix/format by @binmakeswell in [NFC] Hotfix/format #984
• Revert "[NFC] Hotfix/format" by @ver217 in Revert "[NFC] Hotfix/format" #986
• remove useless import in tensor dir by @feifeibear in remove useless import in tensor dir #997
• [NFC] fix download link by @binmakeswell in [NFC] fix download link #998
• [Bot] Synchronize Submodule References by @github-actions in [Bot] Synchronize Submodule References #1003
• [NFC] polish colossalai/kernel/cuda_native/csrc/colossal_C_frontend.c… by @zhengzangw in [NFC] polish colossalai/kernel/cuda_native/csrc/colossal_C_frontend.c… #1010
• [NFC] fix paper link by @binmakeswell in [NFC] fix paper link #1012
• [p2p]add object list send/recv by @YuliangLiu0306 in [p2p]add object list send/recv #1024
• [Bot] Synchronize Submodule References by @github-actions in [Bot] Synchronize Submodule References #1034
• [NFC] add inference by @binmakeswell in [NFC] add inference #1044
• [titans]remove model zoo by @YuliangLiu0306 in [titans]remove model zoo #1042
• [NFC] add inference submodule in path by @binmakeswell in [NFC] add inference submodule in path #1047
• [release] update version.txt by @FrankLeeeee in [release] update version.txt #1048
• [Bot] Synchronize Submodule References by @github-actions in [Bot] Synchronize Submodule References #1049
• updated collective ops api by @kurisusnowdeng in updated collective ops api #1054
• [pipeline]refactor ppschedule to support tensor list by @YuliangLiu0306 in [pipeline]refactor ppschedule to support tensor list #1050

New Contributors

• @ryanrussell made their first contribution in fix typo in constants #1027

Full Changelog: v0.1.5...v0.1.6

---

This discussion was created from the release v0.1.6 Released! .