feat(cpu_offload.py): support selective layers' activation cpu offload async#391

Open

huangting4201 wants to merge 7 commits intoInternLM:developfrom huangting4201:feat/activation-ckpt-combine-with-offload

+1,124-27

Commits on Dec 3, 2024

Commits on Dec 10, 2024

feat(attn_offload.py): update attn offload manager
huangting4201
committed

Commits on Dec 17, 2024

Merge branch 'develop' into feat/selective-ckpt-cpu-offload
huangting4201
committed
fix(conflicts): fix conflicts from merging develop
huangting4201
committed
feat(cpu_offload.py): support activation cpu offload async and can be used with ckpt/selective_ckpt
huangting4201
committed

Commits on Dec 18, 2024

fix(test_loss.py): fix isp test case since change 7B_isp_sft.py config MHA to GQA
huangting4201
committed