how to use 'load_in_8bit=True' when train

[Haoran1234567]

I want to train model llama-13b ，SFT stage,7b is ok when i use 8*24g(3090). but 13B is OOM. i have try all the ways in deepspeedchat to reduce memory,all OOM!
i want to try use 'load_in_8bit=True' when load model,but ERROR!
how to modify the code???

[laoda513]

try actor_gradient_checkpointing and lora?

[Haoran1234567]

yes,i have try

[samadejacobs]

@Haoran1234567, have you tried ZeRO3? Could you share memory usage from your log? DeepSpeed by default should let you know what memory usage is. FYI, we are also working on ZeRO offload support for DeepSpeed Chat models, this support will be made available soon.