Skip to content

[Xet] Document adaptive concurrency for xet transfers#2260

Merged
rajatarya merged 4 commits intomainfrom
rajatarya/xet-adaptive-concurrency
Mar 11, 2026
Merged

[Xet] Document adaptive concurrency for xet transfers#2260
rajatarya merged 4 commits intomainfrom
rajatarya/xet-adaptive-concurrency

Conversation

@rajatarya
Copy link
Contributor

@rajatarya rajatarya commented Mar 2, 2026

Summary

  • Document that hf_xet uses adaptive concurrency by default, dynamically adjusting parallelism based on real-time network conditions (default: 1→64 streams)
  • Update HF_XET_HIGH_PERFORMANCE=1 docs to reflect it now raises adaptive concurrency bounds (initial=16, max=124) and increases buffer sizes, rather than setting fixed values
  • Add HF_XET_FIXED_DOWNLOAD_CONCURRENCY / HF_XET_FIXED_UPLOAD_CONCURRENCY as documented escape hatches for advanced users
  • Add adaptive concurrency notes to the xet upload and download protocol specs

Files changed

File Change
docs/hub/models-downloading.md Rewrite "Faster downloads" section
docs/hub/mlx.md Add brief explanation of HP mode
docs/hub/xet/using-xet-storage.md Update recommendations with adaptive concurrency + fixed concurrency env vars
docs/xet/upload-protocol.md Add paragraph on adaptive concurrency in "Ordering and Concurrency"
docs/xet/download-protocol.md Add bullet on adaptive concurrency under "Parallel downloads"

Test plan

  • Verify doc build passes in CI
  • Confirm env var names match xet-core source (HF_XET_HIGH_PERFORMANCE, HF_XET_FIXED_DOWNLOAD_CONCURRENCY, HF_XET_FIXED_UPLOAD_CONCURRENCY)
  • Confirm default values (initial=1, max=64) and HP values (initial=16, min=4, max=124) match xet_config/src/groups/client.rs and xet_config/src/xet_config.rs

🤖 Generated with Claude Code


Note

Low Risk
Documentation-only changes describing hf_xet/xet-core transfer behavior and new/updated environment-variable guidance; no code or runtime behavior is modified.

Overview
Documents that hf_xet uses adaptive concurrency by default for Xet transfers, and updates download guidance to clarify that HF_XET_HIGH_PERFORMANCE=1 raises concurrency bounds and buffer sizes (intended for high-bandwidth machines with �64GB RAM).

Expands using-xet-storage.md with a dedicated Environment Variables section (including fixed-concurrency escape hatches via HF_XET_FIXED_{DOWNLOAD,UPLOAD}_CONCURRENCY) and adds adaptive-concurrency notes to the Xet upload/download protocol specs; also adds a brief HP-mode note to the MLX Hub guide.

Written by Cursor Bugbot for commit 4ba1a67. This will update automatically on new commits. Configure here.

Describe the adaptive concurrency feature that xet-core uses by default
to dynamically adjust parallelism based on network conditions. Update
HF_XET_HIGH_PERFORMANCE documentation to reflect that it now raises
adaptive concurrency bounds (initial=16, max=124) rather than setting
fixed values. Add HF_XET_FIXED_DOWNLOAD/UPLOAD_CONCURRENCY as advanced
escape hatches.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@rajatarya rajatarya requested a review from assafvayner March 2, 2026 23:31
Add a comprehensive "Environment Variables" section to the Xet storage
docs covering adaptive concurrency, network/retry, data transfer,
download buffers, and logging. All variable names and defaults verified
against xet-core source.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@rajatarya rajatarya requested a review from julien-c March 3, 2026 03:06
@rajatarya
Copy link
Contributor Author

@julien-c : (not urgent, can wait until you are back.)

- Clarify that default adaptive concurrency already saturates most
  network paths — no tuning needed for most users
- Document HF_XET_HIGH_PERFORMANCE as requiring >=64 GB RAM, warn it
  may degrade performance on smaller machines
- Mark adaptive concurrency env vars as advanced settings
- Fix AC_TARGET_RTT description per reviewer suggestion

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Collaborator

@hoytak hoytak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Member

@julien-c julien-c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

let's communicate widely when this is shipped, as it's very cool feature for power users and casual users alike

Remove redundant pip install line and fix code fence language tag.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@rajatarya rajatarya merged commit e6f63f7 into main Mar 11, 2026
4 checks passed
@rajatarya rajatarya deleted the rajatarya/xet-adaptive-concurrency branch March 11, 2026 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants