[download] Replace HfFileSystem.find() with HfApi.list_repo_files() to fix silent truncation

[claude]

HfFileSystem.find() silently truncates file lists on large HuggingFace repos due to a pagination bug, causing incomplete downloads (e.g. nvidia/Nemotron-CC-v2 returned 8330/8971 files). This replaces find() with HfApi.list_repo_files() which handles pagination correctly. Bucket paths still use find() since they are not standard HF repos. Adds tests verifying the new code path is used and produces correct HfFileSystem-format paths.

Fixes #4170

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e25d56f8e5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Normalize repo-type prefix before HfApi lookup

_list_repo_files maps cfg.hf_repo_type_prefix by exact key, so values like "datasets/" or "spaces/" fall through to the default "model". This is a regression from the previous hf_fs.find path-based behavior: configs that include the trailing slash (which is still documented in DownloadConfig) will now call list_repo_files(..., repo_type="model") for dataset/space repos and fail to list files (typically 404/not found). Strip trailing / (or otherwise normalize) before the lookup to preserve existing inputs.

Useful? React with 👍 / 👎.

[Helw150]

@claude I'm not sure this is a good root cause fix. We don't want to use list_repo_files as the way to access these things, but we want to cross-reference the file count from hf_fs.find with list_repo_files.

[claude]

Claude finished @Helw150's task in 3m 9s —— View job

---

Refactored to cross-reference approach per feedback

• Gather context — read current PR changes and understand the code
• Implement new approach: keep hf_fs.find() as primary, cross-reference count with list_repo_files()
• Update tests to match new behavior
• Run pre-commit and tests
• Push changes

What changed:

Per @Helw150's feedback, I refactored the approach so that hf_fs.find() remains the primary file listing method. Instead of replacing it with list_repo_files(), we now cross-reference the file count:

1. hf_fs.find() is called as before to get the file list
2. HfApi.list_repo_files() is called to get the expected count
3. If the counts don't match, a RuntimeError is raised with a clear message about the pagination bug

Key changes:

• _list_repo_files() → _get_expected_file_count() — now returns just the count instead of a full file list
• download_hf() calls hf_fs.find() first, then validates the count against _get_expected_file_count() for non-bucket paths
• Added test_download_hf_raises_on_truncated_find — verifies the error is raised when counts mismatch
• Updated existing tests to match the new flow

Testing: All 10 tests in tests/download/test_huggingface.py pass. Pre-commit checks pass.
·

[Helw150]

@claude these mocked tests aren't super useful, so you can remove them

[claude]

Claude finished @Helw150's task in 2m 56s —— View job

---

Removing mocked tests

• Read current test file to identify mocked tests
• Remove mocked tests (test_get_expected_file_count_returns_count, test_download_hf_cross_references_find_with_list_repo_files, test_download_hf_raises_on_truncated_find)
• Clean up existing tests to patch _get_expected_file_count instead of mocking HfApi
• Run pre-commit and tests
• Commit and push

Removed the 3 mocked tests and cleaned up the existing download_hf tests to patch _get_expected_file_count → None (skipping the cross-reference check) instead of mocking HfApi. All 7 remaining tests pass. Pre-commit checks pass.

Testing: uv run pytest tests/download/test_huggingface.py -v — 7 passed.

View job

[github-actions]

This pull request has been inactive for 23 days and is marked as stale.
If there is no further activity within 7 days, it will be automatically closed.
If you believe this PR should remain open, please add a comment or update the PR.

[github-actions]

This pull request has been automatically closed due to inactivity.
If you would like to continue working on this, please reopen it or create a new PR.