RLHF model return '{: {: {:' of every input

[kuangdao]

I used as readme of deepspeed chat.

training/step1_supervised_finetuning/training_scripts/single_node/run_1.3b.sh
training/step2_reward_model_finetuning/training_scripts/single_node/run_350m.sh
training/step3_rlhf_finetuning/training_scripts/single_node/run_1.3b.sh


python3 chat.py --path output/step3-models/1.3b/actor

bug when i want to try chat.py . the result is :

Enter input (type 'quit' to exit, 'clear' to clean memory): What is Edge?
------------------------------ Round 1 ------------------------------
 Human: What is Edge?
 Assistant:  {: {: {: {: {: {: {: {: {: {:
Enter input (type 'quit' to exit, 'clear' to clean memory): clear
Enter input (type 'quit' to exit, 'clear' to clean memory): hello
------------------------------ Round 1 ------------------------------
 Human: hello
 Assistant:  {: {: {: {: {: {: {: {: {: {: {: {: {:
Enter input (type 'quit' to exit, 'clear' to clean memory): clear
Enter input (type 'quit' to exit, 'clear' to clean memory): who are you?                    
------------------------------ Round 1 ------------------------------
 Human: who are you?
 Assistant:  {: {: {: {: {: {: {: {: {: {:
Enter input (type 'quit' to exit, 'clear' to clean memory):

every inputs it returns "{: {: {: {: {: {: {: {: {: {:" .

[laoda513]

see #503

[xiaoxiawu-microsoft]

@laoda513 and @kuangdao. I would like to take a look at the logs. Please send the training log to let me take a look.

[laoda513]

@laoda513 and @kuangdao. I would like to take a look at the logs. Please send the training log to let me take a look.

sry， i am on trip this week.
I provided some log in #503
and pervious logs looks quite normal（including step1）
SInce my gpu cards are 20 series, may this be the possible reason?

[xiaoxiawu-microsoft]

One of the problems I can think of is probably due to the RLHF making experience part https://github.com/microsoft/DeepSpeedExamples/blob/master/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py#L76 As you can see here, it forces the sequence to be exactly the sequence length.