Feature request
For flat permission file systems, os.chmod simply fails when trying to cache datasets - in some places such as: https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L4043-L4048 for arrow datasets.
This seems to be optional to obtained desired behavior as simply commenting it out makes our workload work as expected. Hence the feature request would be to see if it would be possible to expose an API to disable os.chmod operations when users asks to do so (e.g. through an env variable?)
Motivation
Being able to save properly datasets on flat permission filesystems
Your contribution
Can test the feature if implemented
Feature request
For flat permission file systems,
os.chmodsimply fails when trying to cache datasets - in some places such as: https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L4043-L4048 for arrow datasets.This seems to be optional to obtained desired behavior as simply commenting it out makes our workload work as expected. Hence the feature request would be to see if it would be possible to expose an API to disable
os.chmodoperations when users asks to do so (e.g. through an env variable?)Motivation
Being able to save properly datasets on flat permission filesystems
Your contribution
Can test the feature if implemented