[Rust Frontend] Migrate to hf-hub 1.0

[BugenZhao]

Purpose

This draft explores migrating the Rust frontend from hf-hub 0.5 to 1.0.

The implementation follows the redesigned 1.0 API:

• reuse one HFClient and a typed HFRepository<RepoTypeModel>
• resolve cached files through download_file().local_files_only(true)
• classify structured HFError variants and preserve their source chains
• use the builder-based info() and download_file() APIs
• migrate the tokenizer benchmarks
• remove the unused llm-multimodal hf-hub feature so the dependency graph contains one hf-hub version

Blockers

1. Rust TLS and crypto violate the workspace dependency policy

hf-hub 1.0.0 has two paths that activate Rust TLS:

1. Its direct reqwest 0.13 dependency keeps reqwest's default features, which select default-tls -> rustls.
2. Its unconditional hf-xet dependency reaches xet-client, whose default feature is rustls-tls.

reqwest 0.13's rustls feature selects the AWS-LC provider. As a result, cargo deny check bans fails for:

• rustls
• aws-lc-rs
• aws-lc-sys

The Rust workspace deliberately uses native-tls/OpenSSL and bans these crates in rust/deny.toml. hf-hub 1.0.0 exposes no downstream feature combination that selects native-tls for both paths.

2. Xet-backed cold-cache downloads stall on GCP

On a GCP GB200 host, a cold-cache download for moonshotai/Kimi-K2.5 consistently behaves as follows:

• tokenizer_config.json downloads successfully.
• The 2,795,286-byte Xet-backed tiktoken.model remains at zero downloaded bytes until the test timeout.
• Reducing Xet concurrency and setting explicit connect/read timeouts produces the same result.
• A direct HTTP range request to the same resolve URL returns 206 immediately.

Related upstream reports include xet-core#800 and xet-core#869. The proposal to make hf-xet optional was closed in huggingface/hf-hub#181.

Dependency impact

The lockfile grows from 575 to 668 packages. The 93-package net increase comes from the unconditional Xet stack, reqwest 0.13, rustls/AWS-LC, and their platform-specific dependencies.

This PR will remain a draft while the two blockers above are active.

Test Plan

cd rust
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo nextest run --workspace
cargo deny check bans

Cold-cache reproduction:

cd rust
HF_HUB_CACHE="$(mktemp -d)" \
  cargo nextest run -p vllm-text \
  tiktoken_real_kimi_k25_tokenizer_files_load_and_handle_special_tokens \
  --run-ignored ignored-only

Test Result

• cargo fmt --all -- --check: passed
• cargo clippy --workspace --all-targets -- -D warnings: passed
• cargo nextest run --workspace: 1,205 passed, 1 skipped
• cache-only resolution test with an unreachable endpoint: passed
• cargo deny check bans: failed on rustls, aws-lc-rs, and aws-lc-sys as documented above
• cold-cache Xet reproduction: stalled at zero bytes for tiktoken.model

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR is documented.
• The test plan is documented.
• Test results and current blockers are documented.
• Documentation impact was reviewed; this dependency migration requires no user-facing documentation update.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @BugenZhao.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork