[Bug] Update torch.optim.Optimizer parameter states after tensor parallelism#3835

SunMarc merged 5 commits intohuggingface:mainhuggingface/accelerate:mainfrom

naomili0924:fix_tp_and_fsdp_optimizernaomili0924/accelerate:fix_tp_and_fsdp_optimizerCopy head branch name to clipboard