[FSDP] Support Context parallelism for FSDP using ring-flash-attn#467

Merged

zhuzilin merged 18 commits intoTHUDM:mainfrom

PopSoda2002:feat/support_normal_cp

Nov 16, 2025

Commits on Nov 15, 2025

Support CP
PopSoda2002
committed
Fix
PopSoda2002
committed
Ensure cp group has same data
PopSoda2002
committed
Use device mesh for fsdp for proper gradient update and improve logs

Hecate0821
authored and
PopSoda2002
committed
Clean code
PopSoda2002
committed
Change device mesh for dp
PopSoda2002
committed
Fix training/infer log probs mismatch
PopSoda2002
committed
Refactor device mesh setup

Hecate0821
authored and
PopSoda2002
committed
Update sript

Hecate0821
authored and
PopSoda2002
committed
All gather logp and entropy instead of logits
PopSoda2002
committed
Update script and eliminate debugging logs

Hecate0821
authored and
PopSoda2002
committed
clean code
PopSoda2002
committed
clean
PopSoda2002
committed
Clean
PopSoda2002
committed
Clean code
PopSoda2002
committed

Commits on Nov 16, 2025