feat(python): hf:// URL scheme for reading Vortex files from the Hugging Face Hub

[joseph-isaacs]

Summary

Adds support for reading Vortex files directly from the Hugging Face Hub via hf:// URLs, following the HfFileSystem URL convention:

import vortex as vx

vxf = vx.open("hf://datasets/my-org/my-dataset/data/train.vortex")
vxf.scan(["score"], expr=...)  # lazy: only downloads the bytes it needs

• vortex/hf/_resolve.py — parses hf://[datasets/|spaces/]namespace/name[@revision]/path URLs and translates them to the Hub's resolve endpoint, served as ranged HTTP reads through HTTPStore. Honors HF_ENDPOINT overrides (local mirrors get allow_http automatically). Tokens for gated/private repos resolve from HF_TOKEN/HUGGING_FACE_HUB_TOKEN or the huggingface_hub token cache file, and attach as an authorization header.
• vortex/hf/__init__.py — public API: vortex.hf.open() (accepts an explicit token and HTTP client config), plus HFLocation, resolve_url, http_store, store_and_path, token, endpoint.
• vortex/file.py — vx.open("hf://...") routes through the resolver.
• vortex/store/__init__.py — vortex.store.from_url("hf://...") returns an HTTPStore rooted at the resolve URL.

Because the Hub's resolve endpoint supports HTTP range requests, Vortex's lazy scans (projection, predicate pushdown, row indices) work against Hub-hosted files without downloading them: in local measurements a projected single-column scan of a 200k-row shard downloads ~15% of the file.

Unnamespaced repo ids (e.g. hf://datasets/squad/...) are rejected: they are ambiguous without a Hub API call, so repo ids must be fully qualified namespace/name.

A follow-up PR stacked on this one adds a Hugging Face datasets builder and a torch-compatible map-style dataset.

Test plan

• New test/test_hf.py (23 tests) runs against a local Range-supporting stand-in for the Hub resolve endpoint (test/hub_server.py, exposed as the shared local_hub fixture) — no network or HF account needed. Covers URL parsing, token resolution and 401→token→success auth, revision pinning, and laziness (a projected scan must download strictly less than the file, using only ranged reads).
• Full vortex-python suite: 152 passed, 2 skipped, 1 xfailed.
• basedpyright vortex-python: 0 errors. ruff check + ruff format --check: clean.

https://claude.ai/code/session_01Q7vfwXk1FwrcgaDS8sHYth

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 1529 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	bitwise_not_vortex_buffer_mut[128]	215.3 ns	244.4 ns	-11.93%
⚡	Simulation	encode_varbin[(1000, 4)]	159.4 µs	142.4 µs	+11.95%
⚡	Simulation	encode_varbin[(1000, 32)]	164.6 µs	148.1 µs	+11.2%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/vortex-hf-url-scheme-kib4lk (b649eea) with develop (3d7bbfb)}