V0.1.3 Released!

[ver217]

Overview

Here are the main improvements of this release:

1. Gemini: Heterogeneous memory space manager
2. Refactor the API of pipeline parallelism

What's Changed

Features

• [zero] initialize a stateful tensor manager by @feifeibear in [zero] initialize a stateful tensor manager #614
• [pipeline] refactor pipeline by @YuliangLiu0306 in [pipeline] refactor pipeline #679
• [zero] stateful tensor manager by @ver217 in [zero] stateful tensor manager #687
• [zero] adapt zero hooks for unsharded module by @1SAA in [zero] adapt zero hooks for unsharded module #699
• [zero] refactor memstats collector by @ver217 in [zero] refactor memstats collector #706
• [zero] improve adaptability for not-shard parameters by @1SAA in [zero] improve adaptability for not-shard parameters #708
• [zero] check whether gradients have inf and nan in gpu by @1SAA in [zero] check whether gradients have inf and nan in gpu #712
• [refactor] refactor the memory utils by @feifeibear in [refactor] refactor the memory utils #715
• [util] support detection of number of processes on current node by @FrankLeeeee in [util] support detection of number of processes on current node #723
• [utils] add synchronized cuda memory monitor by @1SAA in [utils] add synchronized cuda memory monitor #740
• [zero] refactor ShardedParamV2 by @1SAA in [zero] refactor ShardedParamV2 #742
• [zero] add tensor placement policies by @ver217 in [zero] add tensor placement policies #743
• [zero] use factory pattern for tensor_placement_policy by @feifeibear in [zero] use factory pattern for tensor_placement_policy #752
• [zero] refactor memstats_collector by @1SAA in [zero] refactor memstats_collector #746
• [gemini] init genimi individual directory by @feifeibear in [gemini] init genimi individual directory #754
• refactor shard and gather operation by @1SAA in refactor shard and gather operation #773

Bug Fix

• [zero] fix init bugs in zero context by @1SAA in [zero] fix init bugs in zero context #686
• [hotfix] update requirements-test by @ver217 in [hotfix] update requirements-test #701
• [hotfix] fix a bug in 3d vocab parallel embedding by @kurisusnowdeng in [hotfix] fix a bug in 3d vocab parallel embedding #707
• [compatibility] fixed tensor parallel compatibility with torch 1.9 by @FrankLeeeee in [compatibility] fixed tensor parallel compatibility with torch 1.9 #700
• [hotfix]fixed bugs of assigning grad states to non leaf nodes by @Gy-Lu in [hotfix]fixed bugs of assigning grad states to non leaf nodes #711
• [hotfix] fix stateful tensor manager's cuda model data size by @ver217 in [hotfix] fix stateful tensor manager's cuda model data size #710
• [bug] fixed broken test_found_inf by @FrankLeeeee in [bug] fixed broken test_found_inf #725
• [util] fixed activation checkpointing on torch 1.9 by @FrankLeeeee in [util] fixed activation checkpointing on torch 1.9 #719
• [util] fixed communication API with PyTorch 1.9 by @FrankLeeeee in [util] fixed communication API with PyTorch 1.9 #721
• [bug] removed zero installation requirements by @FrankLeeeee in [bug] removed zero installation requirements #731
• [hotfix] remove duplicated param register to stateful tensor manager by @feifeibear in [hotfix] remove duplicated param register to stateful tensor manager #728
• [utils] correct cpu memory used and capacity in the context of multi-process by @feifeibear in [utils] correct cpu memory used and capacity in the context of multi-process #726
• [bug] fixed grad scaler compatibility with torch 1.8 by @FrankLeeeee in [bug] fixed grad scaler compatibility with torch 1.8 #735
• [bug] fixed DDP compatibility with torch 1.8 by @FrankLeeeee in [bug] fixed DDP compatibility with torch 1.8 #739
• [hotfix] fix memory leak in backward of sharded model by @ver217 in [hotfix] fix memory leak in backward of sharded model #741
• [hotfix] fix initialize about zero by @ver217 in [hotfix] fix initialize about zero #748
• [hotfix] fix prepare grads in sharded optim by @ver217 in [hotfix] fix prepare grads in sharded optim #749
• [hotfix] layernorm by @kurisusnowdeng in [hotfix] layernorm #750
• [hotfix] fix auto tensor placement policy by @ver217 in [hotfix] fix auto tensor placement policy #753
• [hotfix] fix reuse_fp16_shard of sharded model by @ver217 in [hotfix] fix reuse_fp16_shard of sharded model #756
• [hotfix] fix test_stateful_tensor_mgr by @ver217 in [hotfix] fix test_stateful_tensor_mgr #762
• [compatibility] used backward-compatible API for global process group by @FrankLeeeee in [compatibility] used backward-compatible API for global process group #758
• [hotfix] fix the ckpt hook bugs when using DDP by @Gy-Lu in [hotfix] fix the ckpt hook bugs when using DDP #769
• [hotfix] polish sharded optim docstr and warning by @ver217 in [hotfix] polish sharded optim docstr and warning #770

Unit Testing

• [ci] replace the ngc docker image with self-built pytorch image by @FrankLeeeee in [ci] replace the ngc docker image with self-built pytorch image #672
• [ci] fixed compatibility workflow by @FrankLeeeee in [ci] fixed compatibility workflow #678
• [ci] update workflow trigger condition and support options by @FrankLeeeee in [ci] update workflow trigger condition and support options #691
• [ci] added missing field in workflow by @FrankLeeeee in [ci] added missing field in workflow #692
• [ci] remove ipc config for rootless docker by @FrankLeeeee in [ci] remove ipc config for rootless docker #694
• [test] added missing decorators to model checkpointing tests by @FrankLeeeee in [test] added missing decorators to model checkpointing tests #727
• [unitest] add checkpoint for moe zero test by @1SAA in [unitest] add checkpoint for moe zero test #729
• [test] added a decorator for address already in use error with backward compatibility by @FrankLeeeee in [test] added a decorator for address already in use error with backward compatibility #760
• [test] refactored with the new rerun decorator by @FrankLeeeee in [test] refactored with the new rerun decorator #763

Documentation

• add PaLM link by @binmakeswell in add PaLM link #704
• [doc] removed outdated installation command by @FrankLeeeee in [doc] removed outdated installation command #730
• add video by @binmakeswell in add video #732
• [readme] polish readme by @feifeibear in [readme] polish readme #764
• [readme] sync CN readme by @binmakeswell in [readme] sync CN readme #766

Miscellaneous

• [Bot] Synchronize Submodule References by @github-actions in [Bot] Synchronize Submodule References #556
• [Bot] Synchronize Submodule References by @github-actions in [Bot] Synchronize Submodule References #695
• [refactor] zero directory by @feifeibear in [refactor] zero directory  #724
• [Bot] Synchronize Submodule References by @github-actions in [Bot] Synchronize Submodule References #751

Full Changelog: v0.1.2...v0.1.3

---

This discussion was created from the release V0.1.3 Released!.

[DustinEwan]

[zero] check whether gradients have inf and nan in gpu by @1SAA in #712

I occasionally see a warning from this: WARNING colossalai - ShardedOptimizerV2 - WARNING: found inf during ShardedOptimV2 step

Is there something I should be doing about this, or is this just like a "hey it happens, but thought you should know" sort of warning?

[feifeibear]

The warning is also existing in the other frameworks using mixed precision training. If you see it occasionally that's fine. The system just skipped the grad overflowing iterations. If you see nan in every iteration, you should tune the hyperparameters, a smaller learning rate for example to avoid grad value exceeding the range of float fp16.