多机分布式训练，加载模型，报a leaf Variable that requires grad is being used in an in-place operation错误

[sc-lj]

使用deepspeed 多机分布式训练，加载opt-1.3b 模型的时候，报a leaf Variable that requires grad is being used in an in-place operation错误

[sc-lj]

colorful: Traceback (most recent call last):
colorful: File "DeepSpeed-Chat/training/main_sup.py", line 339, in
colorful: main()
colorful: File "DeepSpeed-Chat/training/main_sup.py", line 286, in main
colorful: model, optimizer, _, lr_scheduler = deepspeed.initialize(
colorful: File "/home/vocust001/miniconda3/envs/ldm/lib/python3.9/site-packages/deepspeed/init.py", line 156, in initialize
colorful: engine = DeepSpeedEngine(args=args,
colorful: File "/home/vocust001/miniconda3/envs/ldm/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 286, in init
colorful: self._configure_distributed_model(model)
colorful: File "/home/vocust001/miniconda3/envs/ldm/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1087, in _configure_distributed_model
colorful: self._broadcast_model()
colorful: File "/home/vocust001/miniconda3/envs/ldm/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1017, in _broadcast_model
colorful: dist.broadcast(p, groups._get_broadcast_src_rank(), group=self.data_parallel_group)
colorful: File "/home/vocust001/miniconda3/envs/ldm/lib/python3.9/site-packages/deepspeed/comm/comm.py", line 120, in log_wrapper
colorful: return func(*args, **kwargs)
colorful: File "/home/vocust001/miniconda3/envs/ldm/lib/python3.9/site-packages/deepspeed/comm/comm.py", line 217, in broadcast
colorful: return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
colorful: File "/home/vocust001/miniconda3/envs/ldm/lib/python3.9/site-packages/deepspeed/comm/torch.py", line 81, in broadcast
colorful: return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
colorful: File "/home/vocust001/miniconda3/envs/ldm/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1201, in broadcast
colorful: work.wait()
colorful: RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

[AltenLi]

+1