Skip to content

Commit 4486025

Browse files
committed
docs(regen): dataset choices come from the shared registry
The CLI doc's hand-copied dataset list had already drifted (gsm8k was never added); point at the shared registry instead of maintaining a third copy, and document the prepare-data-parity ingestion. Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com>
1 parent 9db7928 commit 4486025

1 file changed

Lines changed: 5 additions & 6 deletions

File tree

docs/cli/response_regeneration.md

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -69,9 +69,9 @@ python scripts/response_regeneration/script.py --dataset magpie
6969

7070
#### Data Arguments
7171

72-
- **`--dataset`** (str, default: `ultrachat`, choices: `magpie`, `ultrachat`) Dataset to process.
72+
- **`--dataset`** (str, default: `ultrachat`) Dataset preset to process. Accepts any preset from the shared dataset registry (`DATASET_CONFIGS` in `speculators/data_generation/configs.py`); see `--help` for the current list.
7373

74-
- **`--split`** (str, default: dataset-specific) Dataset split. Defaults to `train` for magpie and `train_sft` for ultrachat.
74+
- **`--split`** (str, default: preset-specific) Dataset split. Defaults to the preset's split.
7575

7676
- **`--limit`** (int, default: `None`) Stop after N rows.
7777

@@ -110,10 +110,9 @@ python scripts/response_regeneration/script.py \
110110

111111
## Supported Datasets
112112

113-
| Dataset | HuggingFace ID | Prompt Field | Default Split |
114-
| --------- | ------------------------------------------------- | ------------- | ------------- |
115-
| Magpie | `Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered` | `instruction` | `train` |
116-
| UltraChat | `HuggingFaceH4/ultrachat_200k` | `prompt` | `train_sft` |
113+
Every preset in the shared dataset registry (`DATASET_CONFIGS` in `speculators/data_generation/configs.py`) is supported — the same registry the `prepare-data` pipeline uses, so a dataset defined once works in both. Rows are ingested exactly as in `prepare-data` (`filter_fn`, then `normalize_fn`): system and user turns are kept, and the assistant turns are regenerated. Rows without a usable conversation fall back to the preset's bare-prompt column (`prompt_field`).
114+
115+
Note: `sharegpt4v_coco` additionally requires an image-capable endpoint and local COCO files, matching its `prepare-data` requirements.
117116

118117
## Output Format
119118

0 commit comments

Comments
 (0)