Hi, I want to use disk dataset, however, it gives:
stats = {}
for dataset_idx, dataset in enumerate(self.datasets):
assert (
"s3" in dataset.folder_path
), "Only S3 paths are supported for consumption stats" # TODO: remove this
stats[dataset.folder_path] = {"tokens": self.consumed_tokens[dataset_idx]}
return stats
I wonder why only S3 paths are supported for consumption stats