feat: Add utility to create responses datasets based on HF datasets

[mayabar]

Create a new utility under the cmd folder which gets as an input parameter an Hugging Face dataset (we need to define which formats will be supported) and a model name to be used for the tokenization. The utility should generate a SQLite file mapping each prompt hash value to the appropriate response.

Datasets created by this utility will be placed in the llm-d organization in HF - https://huggingface.co/llm-d/datasets

[mayabar]

@pancak3 do you have a such utility that you can share?

[pancak3]

@pancak3 do you have a such utility that you can share?

The custom dataset feature I worked on was fully offline. I currently only have the conversion on local datasets, and the code is in Python, but I am happy to take on this feat if it is not urgent.

Correct me if wrong, the feature here requires

• an utility command, sth like --convert_dataset /path/to/local or --convert_dataset hf://org/dataset
  • to convert HF datasets to llm-d datasets.
    • Use the converted at runtime
    • option to push the converted to https://huggingface.co/llm-d/datasets
      • In this case, we'd probably be better off hosting this conversion code on the repo https://huggingface.co/llm-d/datasets instead. The benefit is that datasets can be managed independently on the HF repo, accepting dataset PRs from contributors. As long as the hash algorithm is unified, our project don't need to accept dataset prs. This should keep inference-sim repo clean.
      • Actually, I have created a very similar repo during the last pr: https://huggingface.co/datasets/hf07397/inference-sim-datasets. I'm Happy to standardize it and migrate it to https://huggingface.co/llm-d/datasets
  • The datasets on the repo can be reused via arg with dataset identifiers, sth like --remote_dataset hf://llm-d/datasets/<DATASET_IDENTIFIER>
    • Reuse local dataset cache

There is related code in other projects I am using, which loads remote hf datasets to benchmark llm infer services:

• inference-perf, in python, https://github.com/kubernetes-sigs/inference-perf/blob/e8e0aa99c57f2ffa0912df7ba1fbd2a8a596a041/inference_perf/datagen/hf_sharegpt_datagen.py#L54

[mayabar]

Thanks a lot for your explanation.
I would keep the utility output local
It can get 3 parameters:

• input (HF) dataset, which is defined by local path or HF url
• output (LLM-D) dataset, which is defined by local path where the llm-d dataset will be stored
• model, will discuss this later
  After local dataset creation, the user will be able to use this dataset as a local SQLite or upload to llm-din HF.

Regarding the model parameter: each model can tokenize the same input in different ways. We need to make clear which model was used for tokenization for each dataset. Option: maybe output will be a folder with 2 files, the sqlite3 file and a "ds card file" which contains link and name of the original dataset and name of the model used for tokenization.

What do you think about this? Do you think it's possible/easy enough to create this utility in go?

[pancak3]

Yes, I think it is feasible to implement in Go. I remember Ira implemented something about a tokenizer, and some of the code may be reused.

To ensure I understand correctly, let me confirm the following,

Regarding the model parameter: each model can tokenize the same input in different ways. We need to make clear which model was used for tokenization for each dataset.

There will be two parameters, --input (an example name) and --tokenizer. Each of their combinations can generate a unique output. The output can be reused with the same argument, i.e., --input and --tokenizer.

Option: maybe output will be a folder with 2 files, the sqlite3 file and a "ds card file" which contains link and name of the original dataset and name of the model used for tokenization.

Standardizing the "ds card file" looks like a great idea. The format should be further discussed in the draft of a related PR for this issue.

[mayabar]

@pancak3 I want to start porting the dataset generation tool from python to go, is it ok with you?

[pancak3]

Sry for the late reply. Let me /assign myself

[pancak3]

/assign

[mayabar]

@pancak3 please note #257
I hope it will be merged soon.
It defined a template for wrapping messages in request as a constant, please use the same constants/functions in the dataset creation utility to be sure that texts are in-sync.

[pancak3]

Hi Maya, sorry for this long delay.

Regarding

We need to make clear which model was used for tokenization for each dataset.

What granularity of different tokenizers do we want to apply? Some datasets do not have a model column (e.g., ShareGPT) while some do (e.g., WildChat and LMSYS).
I assume applying a single tokenizer to a whole dataset is easier at this stage. However, many factors may need to be considered in the future for the project's advanced use.

If we accurately manipulate the input/output lengths, a related tokenizer for the model and chat template will need to be integrated into this project. However, there seems to be no such tokenizer or chat template project in Go that is widely agreed upon.

If we don't accurately manipulate the lengths and we only receive chat messages and respond with texts, we can simply use the dataset name + model name as a unique identifier for this feature.

What are your thoughts on this? Let me know if I’m overthinking the implementation or if I've missed a simpler path. Thanks!

[pancak3]

/unassign