enable_hybrid_engine issue

[llllooong]

Error Info:
File "/data/rooter_use/conda/envs/llama-env39/lib/python3.9/site-packages/deepspeed/runtime/hybrid_engine.py", line 398, in step

actor_loss, critic_loss = trainer.train_rlhf(exp_data)
File "/data/rooter_use/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 173, in train_rlhf
actor_loss, critic_loss = trainer.train_rlhf(exp_data)
if(self._inference_containers[0].module.attention.attn_qkvw is not None and \ File "/data/rooter_use/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 173, in train_rlhf

self.actor_model.step()
File "/data/rooter_use/conda/envs/llama-env39/lib/python3.9/site-packages/deepspeed/runtime/hybrid_engine.py", line 398, in step
IndexError: list index out of range
self.actor_model.step()
File "/data/rooter_use/conda/envs/llama-env39/lib/python3.9/site-packages/deepspeed/runtime/hybrid_engine.py", line 398, in step
if(self._inference_containers[0].module.attention.attn_qkvw is not None and
IndexErrorif(self._inference_containers[0].module.attention.attn_qkvw is not None and :
list index out of range
IndexError: list index out of range

[llllooong]

Hello, I'm runing step 3 with 2 llama models for both actor and rw part, and the two models are both small (2G for actor and 1G for rw). I got error when I set enable_hybrid_engine as True. Is it possible that enable_hybrid_engine does not support llama model, or enable_hybrid_engine is only available when the model is large?

[hijkzzz]

I also encountered the error with enable_hybrid_engine + bloomz-560m + TP=8

 self._inference_containers[layer_id].apply_tensor_parallelism(
  File "/home/jianh/.local/lib/python3.8/site-packages/deepspeed/module_inject/containers/features/meta_tensor.py", line 27, in apply_tensor_parallelism
        dst.data.copy_(src[:, self.gpu_index * dst_shape[self.in_dim]: (self.gpu_index + 1) * dst_shape[self.in_dim]] if inner_dim == 1 else \super().apply_tensor_parallelism(mp_replace, mp_group, tp_size)

  File "/home/jianh/.local/lib/python3.8/site-packages/deepspeed/module_inject/containers/base.py", line 206, in apply_tensor_parallelism
RuntimeError: unsupported operation: some elements of the input tensor and the written-to tensor refer to a single memory location. Please clone() the tensor before performing the operation.
    self.attention_qkv_mp(mp_replace, reversed_dim=reversed_dim)
  File "/home/jianh/.local/lib/python3.8/site-packages/deepspeed/module_inject/containers/bloom.py", line 32, in attention_qkv_mp
    self.module.attention.attn_qkvw = mp_replace.copy(self.module.attention.attn_qkvw, self.qkvw)
  File "/home/jianh/.local/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 107, in copy
    dst.data.copy_(src[:, self.gpu_index * dst_shape[self.in_dim]: (self.gpu_index + 1) * dst_shape[self.in_dim]] if inner_dim == 1 else \
RuntimeError: The size of tensor a (3072) must match the size of tensor b (0) at non-singleton dimension 0

[mcc311]

Same issue here 🙋‍♂️ 🙌

[devinzhang91]

I meet the same issue ! When "Actor" is llama2 and "Critic" is opt-350m.

Maybe the "Actor" and "Critic" is difference model, it look like caused by tokenizer encode, the two different tokenizer encode ids are quite different.

I change "Actor" and "Critic" so that the models come form the same.
When "Actor" and "Critic" are llama2 both, it work....

[GasolSun36]

I meet the same issue ! When "Actor" is llama2 and "Critic" is opt-350m.

Maybe the "Actor" and "Critic" is difference model, it look like caused by tokenizer encode, the two different tokenizer encode ids are quite different.

I change "Actor" and "Critic" so that the models come form the same. When "Actor" and "Critic" are llama2 both, it work....

Hi, I use two llama2-7b as the actor and critic, but it didn't work. Can you share your running sh? thanks a lot!

[devinzhang91]

Hi, I use two llama2-7b as the actor and critic, but it didn't work. Can you share your running sh? thanks a lot!

Just same as applications/DeepSpeed-Chat/training

[xhwang22]

I meet the same issue ! When "Actor" is llama2 and "Critic" is opt-350m.
Maybe the "Actor" and "Critic" is difference model, it look like caused by tokenizer encode, the two different tokenizer encode ids are quite different.
I change "Actor" and "Critic" so that the models come form the same. When "Actor" and "Critic" are llama2 both, it work....

Hi, I use two llama2-7b as the actor and critic, but it didn't work. Can you share your running sh? thanks a lot!

Hi, I meet the same issue. Have you resolved it?

[lucywang720]

I meet the same issue ! When "Actor" is llama2 and "Critic" is opt-350m.
Maybe the "Actor" and "Critic" is difference model, it look like caused by tokenizer encode, the two different tokenizer encode ids are quite different.
I change "Actor" and "Critic" so that the models come form the same. When "Actor" and "Critic" are llama2 both, it work....

Hi, I use two llama2-7b as the actor and critic, but it didn't work. Can you share your running sh? thanks a lot!

Hi, I meet the same issue. Have you resolved it?

I meet too...Have you resolved it?

[GasolSun36]

Latest update:
I haved solved the issue, see deepspeedai/DeepSpeed#4229 (comment)

[gongel]

Latest update: I haved solved the issue, see microsoft/DeepSpeed#4229 (comment)

It works. Thx