xet too broken on windows: downloading a huggingface asset with xet hangs while others in the same method succeed

[doctorpangloss]

Describe the bug

sometimes hf_hub_download with xet deadlocks or errors with the logs below and makes no progress and blocks indefinitely

the file system is ReFS

Reproduction

!pip install huggingface_hub hf_xet
import os
os.environ["HF_XET_HIGH_PERFORMANCE"] = "True"
from huggingface_hub import hf_hub_download
hf_hub_download(repo_id="Comfy-Org/Qwen-Image_ComfyUI", filename="split_files/text_encoders/qwen_2.5_vl_7b.safetensors",
                                               repo_type="model",
                                               )

many times, eventually this error occurs

Logs

2025-08-07 13:47:48 [DEBUG] [comfy.model_downloader] [model_downloader.py:141] text_encoders/qwen_2.5_vl_7b_fp8_scaled.safetensors is being downloaded from Comfy-Org/Qwen-Image_ComfyUI/split_files/text_encoders/qwen_2.5_vl_7b_fp8_scaled.safetensors candidate_str_match=True candidate_filename_match=False candidate_alternate_filenames_match=False candidate_save_filename_match=False
2025-08-07 13:47:48 [DEBUG] [filelock] [_api.py:331] Attempting to acquire lock 1834501623440 on D:/.huggingface\hub\.locks\models--Comfy-Org--Qwen-Image_ComfyUI\cb5636d852a0ea6a9075ab1bef496c0db7aef13c02350571e388aea959c5c0b4.lock
2025-08-07 13:47:48 [DEBUG] [filelock] [_api.py:334] Lock 1834501623440 acquired on D:/.huggingface\hub\.locks\models--Comfy-Org--Qwen-Image_ComfyUI\cb5636d852a0ea6a9075ab1bef496c0db7aef13c02350571e388aea959c5c0b4.lock
{"timestamp":"2025-08-07T20:47:49.627761Z","level":"WARN","fields":{"message":"Status Code: 500. Retrying...","request_id":"01K234MFBJH2VJE24C1T5VNZFS"},"filename":"D:\\a\\xet-core\\xet-core\\cas_client\\src\\http_client.rs","line_number":236}
{"timestamp":"2025-08-07T20:47:49.627817Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 2.6959748s before the next attempt"},"filename":"C:\\Users\\runneradmin\\.cargo\\registry\\src\\index.crates.io-1949cf8c6b5b557f\\reqwest-retry-0.7.0\\src\\middleware.rs","line_number":171}

System info

- huggingface_hub version: 0.33.0
- Platform: Windows-2022Server-10.0.20348-SP0
- Python version: 3.12.10
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: C:\Users\bberman\.cache\huggingface\token
- Has saved token ?: True
- Who am I ?: doctorpangloss
- Configured git credential helpers: manager
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.7.1+cu126
- Jinja2: 3.1.6
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: 11.2.1
- hf_transfer: 0.1.9
- gradio: N/A
- tensorboard: N/A
- numpy: 2.2.6
- pydantic: 2.11.7
- aiohttp: 3.12.13
- hf_xet: 1.1.4
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: C:\Users\bberman\.cache\huggingface\hub
- HF_ASSETS_CACHE: C:\Users\bberman\.cache\huggingface\assets
- HF_TOKEN_PATH: C:\Users\bberman\.cache\huggingface\token
- HF_STORED_TOKENS_PATH: C:\Users\bberman\.cache\huggingface\stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

[doctorpangloss]

lock = {WindowsFileLock} <filelock._windows.WindowsFileLock object at 0x00000261ED0C70E0>
 blocking = {bool} True
 is_locked = {bool} False
 is_singleton = {bool} False
 lock_counter = {int} 0
 lock_file = {str} 'D:/.huggingface\\hub\\.locks\\models--Comfy-Org--Qwen-Image_ComfyUI\\98763a127701eb6fb59096f7742cb3aa7d64ed510b9f4e882d8351f8176e3ce3.lock'
 mode = {int} 420
 timeout = {int} -1

Curiously, in this download only, on Windows, the lock says it is not locked and released, despite:

2025-08-07 16:25:15 [DEBUG] [filelock] [_api.py:331] Attempting to acquire lock 2619612098784 on D:/.huggingface\hub\.locks\models--Comfy-Org--Qwen-Image_ComfyUI\98763a127701eb6fb59096f7742cb3aa7d64ed510b9f4e882d8351f8176e3ce3.lock
2025-08-07 16:25:15 [DEBUG] [filelock] [_api.py:334] Lock 2619612098784 acquired on D:/.huggingface\hub\.locks\models--Comfy-Org--Qwen-Image_ComfyUI\98763a127701eb6fb59096f7742cb3aa7d64ed510b9f4e882d8351f8176e3ce3.lock

[doctorpangloss]

2025-08-07 16:38:42 [DEBUG] [comfy.model_downloader] [model_downloader.py:141] diffusion_models/qwen_image_fp8_e4m3fn.safetensors is being downloaded from Comfy-Org/Qwen-Image_ComfyUI/split_files/diffusion_models/qwen_image_fp8_e4m3fn.safetensors candidate_str_match=True candidate_filename_match=False candidate_alternate_filenames_match=False candidate_save_filename_match=False
Request 7b158ab7-207f-4256-b074-1867a283f16d: HEAD https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/diffusion_models/qwen_image_fp8_e4m3fn.safetensors (authenticated: True)
2025-08-07 16:38:42 [DEBUG] [huggingface_hub.utils._http] [_http.py:87] Request 7b158ab7-207f-4256-b074-1867a283f16d: HEAD https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/diffusion_models/qwen_image_fp8_e4m3fn.safetensors (authenticated: True)
2025-08-07 16:38:42 [DEBUG] [urllib3.connectionpool] [connectionpool.py:1022] Starting new HTTPS connection (1): huggingface.co:443
2025-08-07 16:38:42 [DEBUG] [urllib3.connectionpool] [connectionpool.py:475] https://huggingface.co:443 "HEAD /Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/diffusion_models/qwen_image_fp8_e4m3fn.safetensors HTTP/1.1" 302 0
2025-08-07 16:38:42 [DEBUG] [filelock] [_api.py:331] Attempting to acquire lock 2867315184016 on D:/.huggingface\hub\.locks\models--Comfy-Org--Qwen-Image_ComfyUI\98763a127701eb6fb59096f7742cb3aa7d64ed510b9f4e882d8351f8176e3ce3.lock
2025-08-07 16:38:42 [DEBUG] [filelock] [_api.py:334] Lock 2867315184016 acquired on D:/.huggingface\hub\.locks\models--Comfy-Org--Qwen-Image_ComfyUI\98763a127701eb6fb59096f7742cb3aa7d64ed510b9f4e882d8351f8176e3ce3.lock
Downloading 'split_files/diffusion_models/qwen_image_fp8_e4m3fn.safetensors' to 'D:\.huggingface\hub\models--Comfy-Org--Qwen-Image_ComfyUI\blobs\98763a127701eb6fb59096f7742cb3aa7d64ed510b9f4e882d8351f8176e3ce3.incomplete' (resume from 20430635136/20430635136)

with debug verbosity shows nothing of interest

[doctorpangloss]

i think xet is just broken on windows and should be disabled

[jsulz]

Hey @doctorpangloss thanks for the report. I'll see if I'm able to reproduce on a similar setup.

Does this only happen for you on Comfy-Org/Qwen-Image_ComfyUI?

Also, a few quick troubleshooting steps while I look into this, do any of the following make any difference on your setup:

• Upgrading to the most current version of hf-xet and huggingface-hub (1.1.7 and 0.34.3 respectively)
• Turning off HF_XET_HIGH_PERFORMANCE to reduce concurrency

[jsulz]

After some testing, I've been unable to replicate this on a similar setup with hf-xet and huggingface-hub set to their most current versions.

I think my comment above as a first pass is still valid to troubleshoot on your end. If 1.1.7 doesn't work, it might also be worthwhile to test 1.1.5 (version 1.1.7 introduces a lot of process/thread safety mechanisms, but it's possible these won't play well in your setup).

Does this happen do you only on this repository/with the reproduction steps provided above?

[rajatarya]

@doctorpangloss Can you retry the repro steps using the latest hf-xet and huggingface-hub?

Also, if you disable HF_XET (set HF_HUB_DISABLE_XET=1 environment variable) does this still reproduce?

Do you know what is using the [filelock] python module? I don't see that in the huggingface-hub code and it isn't used in the hf-xet code either.

I also see in the logs that the D:\ is being used for the download - are you setting the HF_HUB env variables to D:\ somewhere?

[rajatarya]

@doctorpangloss Can you try to reproduce the error you are seeing with the diagnostic script we've published for Windows? Hopefully it can shed light on the issue you are encountering.

[tals]

I was just bitten by this, on a fresh install on Windows 11.

Removed hf_xet to work around the problem.
This was with google/siglip2-large-patch16-256

[jsulz]

@tals could you provide an output of hf env so we could see what is installed in your Python environment where this occurred along with any error statements that were printed out?

If possible, could you try to follow the steps in the diagnostic script that has been published for Windows? Due to the complex nature of where this could be happening, this might help us see where the problem resides.

[doctorpangloss]

i'll follow up with the diagnostic later this week, thanks for your patience, I already stopped dealing with Windows

By the way, this is going to be a stupid thing, like the file locks using a path that looks like D:/.huggingface\\hub\\.locks\\models--Comfy-Org--Qwen-Image_ComfyUI\\98763a127701eb6fb59096f7742cb3aa7d64ed510b9f4e882d8351f8176e3ce3.lock, which isn't going to be valid for the way that Python package calls the Windows API's file locking mechanisms that aren't on C:, i.e., with incorrect affordances for the fact that forward slashes are perfectly valid in most contexts.

[rajatarya]

Quick follow up @doctorpangloss @tals : we released hf-xet 1.2.0 on Friday and it includes disabling the xet disk cache (chunk cache) by default. From our testing this cache was causing negative download performance without much benefit (its main benefit is when editing files in a repo). By disabling this cache hf-xet no longer writes to the local disk like it used to - so I am hopeful that this change will resolve the download issues you both are experiencing on Windows. Can you try using the latest hf-xet, pip install -U huggingface_hub hf_xet and report if you continue to encounter issues?

btw, huggingface-hub 1.0 now depends on hf-xet 1.2.0 so by updating huggingface-hub you should pick up the latest hf-xet.