Fix 'standardize_data_formats' when using iterable datasets #126
Conversation
|
Much apologies on the delay @marcandrelarochelle ! Thanks for the PR! |
There was a problem hiding this comment.
@Erland366 Could you take a final review and confirm if iterable datasets work fine? Appreciate it
| @@ -405,10 +405,10 @@ def standardize_data_formats( | |||
| if "conversations" not in column_names: | |||
| return dataset | |||
|
|
|||
| convos = dataset[:10]["conversations"] | |||
| examples = itertools.islice(dataset, 10) | |||
There was a problem hiding this comment.
Is this specifically for iterable datasets? Ie itertools.islice works for iterable datasets whilst dataset[:10] does not?
There was a problem hiding this comment.
Exactly, you can't use an index or in this case [:10] on iterators
| for convo in convos: | ||
| for message in convo: | ||
| for example in examples: | ||
| for message in example["conversations"]: |
There was a problem hiding this comment.
The main reason why I did "conversations" outside is to make it somewhat faster, but I guess since its 10 examples, no big deal
| return dataset.map( | ||
| _standardize_dataset, | ||
| batched = True, | ||
| batch_size = dataset._ex_iterable.batch_size, |
There was a problem hiding this comment.
Interesting on ._ex_iterable - I actually am not super interested with this
There was a problem hiding this comment.
The reason I did this is to pass the batch_size, if you are using a iterable dataset, you might have a resource restrained system and by default it would use a batch size of 1000, but this way it will keep the same batch size it had before the _standardize_datasetformatting.
Adds support to Iterable datasets for the 'standardize_data_formats'
num_procis not supported byIterableDatasetand cannot be accessed via indexPS: Sorry had to recreated my fork, did a mistake on my side