Downloading datasets for offline use.

[eaubin]

I'd like to download datasets from HF and then run completely disconnected. I can do this by using file paths, but it's not ideal. I'd really like to use the repo_id and have it resolved locally when offline. Is there any way to do this? My failing workflow looks like:

export HF_HOME=/shared/huggingface
export HF_HUB_CACHE=/shared/huggingface/hub
export HF_DATASETS_CACHE=/shared/huggingface/datasets
export HF_TOKEN=...

uvx hf download TIGER-Lab/MMLU-Pro --repo-type dataset

# Remove network access.
export HF_HUB_OFFLINE=1
export HF_DATASETS_OFFLINE=1
python -c 'import datasets; datasets.load_dataset("TIGER-Lab/MMLU-Pro")'

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import datasets; datasets.load_dataset("TIGER-Lab/MMLU-Pro")
                     ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/llm-benchmark-validation/.venv/lib/python3.13/site-packages/datasets/load.py", line 1488, in load_dataset
    builder_instance = load_dataset_builder(
        path=path,
    ...<10 lines>...
        **config_kwargs,
    )
  File "/mnt/llm-benchmark-validation/.venv/lib/python3.13/site-packages/datasets/load.py", line 1133, in load_dataset_builder
    dataset_module = dataset_module_factory(
        path,
    ...<5 lines>...
        cache_dir=cache_dir,
    )
  File "/mnt/llm-benchmark-validation/.venv/lib/python3.13/site-packages/datasets/load.py", line 1032, in dataset_module_factory
    raise e1 from None
  File "/mnt/llm-benchmark-validation/.venv/lib/python3.13/site-packages/datasets/load.py", line 974, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach '{path}' on the Hub ({e.__class__.__name__})") from e
ConnectionError: Couldn't reach 'TIGER-Lab/MMLU-Pro' on the Hub (OfflineModeIsEnabled)

[eaubin]

It seems that problem is that $HF_DATASETS_CACHE cannot be populated offline. If I load the dataset prior to removing network access it does not fail.

[KOKOSde]

hf download is only populating the Hub cache here. You need to call load_dataset(...) once while still online so the datasets cache gets built, then the same repo id should work offline.