Merge branch 'microsoft:master' into master

Author: shenzheyu

Diff for: applications/DeepSpeed-Chat/dschat/utils/data/data_utils.py

@@ -211,12 +211,7 @@ def create_dataset_split(current_dataset, raw_dataset, train_phase, tokenizer,
                                          padding="max_length",
                                          truncation=True,
                                          return_tensors="pt")
-                chosen_token["input_ids"] = chosen_token["input_ids"]
-                chosen_token["attention_mask"] = chosen_token["attention_mask"]
                 chosen_dataset.append(chosen_token)
-
-                reject_token["input_ids"] = reject_token["input_ids"]
-                reject_token["attention_mask"] = reject_token["attention_mask"]
                 reject_dataset.append(reject_token)
         print(
             f'Creating dataset {raw_dataset.dataset_name_clean} for {train_phase=} size={len(chosen_dataset)}'

Diff for: applications/DeepSpeed-Chat/training/step2_dpo_finetuning/README.md

@@ -0,0 +1,26 @@
+# 🐕 Direct Preference Optimization (DPO) finetuning
+[Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290) is a novel approach to preference learning, which directly optimizes the policy without explicit reward modeling or reinforcement learning. It leverages a specific parameterization of the reward model that enables the extraction of the corresponding optimal policy in closed form. By using a simple classification loss, DPO aligns language models with human preferences, avoiding the complexity and instability often associated with RLHF.
+
+As the paper says, "Your Language Model is Secretly a Reward Model." Therefore, the training arguments and the training process of DPO are mostly the same as the reward model, as shown in [step2 "Reward Model (RM) finetuning"](../step2_reward_model_finetuning/README.md). After the training of DPO, you will get a model that has been aligned with human preferences.
+
+## 🏃 How to train the model
+
+We provide the script for OPT-350m, which you can test by launching the command
+
+```bash
+training_scripts/opt/single_node/run_350m.sh
+```
+
+We also provide the script for llama2, which you can test by launching the command
+
+```bash
+training_scripts/llama2/run_llama2_7b.sh
+```
+
+## 🏃 How to evaluate the DPO checkpoint?
+
+The checkpoint of DPO is exactly the language model that can be evaluated as [step1 "Supervised Finetuning"](../step1_supervised_finetuning/README.md).
+
+## 💁 Datasets
+
+Because DPO treats the language model as a reward model, the dataset for DPO is in the same format as that used for reward model fine-tuning. Each item in the dataset includes one "chosen" and one "rejected" output for the same input.