pretrain bugfix + add demo data (#2824)

Author: Jonathans575

examples/README.md

@@ -45,6 +45,13 @@ save_to_hf: false
 ...
 ```
 
+为了方便测试，我们也提供了[demo 数据集](https://paddleformers.bj.bcebos.com/datasets/pt_data.tar.gz)可以直接使用：
+
+```shell
+wget https://paddleformers.bj.bcebos.com/datasets/pt_data.tar.gz
+mkdir -p data/pt && tar -xf pt_data.tar.gz -C data/sft/
+```
+
 #### 1.1.2. 离线数据流
 
 我们也可以选择使用离线的比特预训练数据流，更节省内存。离线数据流制作方法如下：

paddleformers/datasets/finetuning.py

@@ -485,8 +485,30 @@ def __iter_func(self):
                             self.estimate = False
                             yield []
 
-            if len(batch_sequence) > 0:
+            # If the entire dataset has been fully traversed, return the remaining data.
+            if len(all_tokenized_tokens) > 0:
+                cut_tokens = all_tokenized_tokens
+                cut_tokens = cut_tokens + [self.tokenizer.eos_token_id]
+                res_tokens = cut_tokens[:-1]
+                res_labels = cut_tokens[1:]
+                loss_mask = [1] * len(res_tokens)
+                pos_ids = list(range(len(res_tokens)))
+                sequence = Sequence(
+                    token_ids=res_tokens,
+                    position_ids=pos_ids,
+                    labels=res_labels,
+                    loss_mask=loss_mask,
+                    num_examples=actual_example_num,
+                )
+                batch_sequence = [sequence]
                 yield batch_sequence
+                if self.estimate:
+                    self.used_estimate_samples += actual_example_num
+                    if self.used_estimate_samples >= self.max_estimate_samples:
+                        self.used_estimate_samples = 0
+                        # Set flag to False and yield empty list to signal the end of estimation
+                        self.estimate = False
+                        yield []
         else:
             if not self.packing:
                 for _ in range(len(self.mix_datasets)):