Core data structures for the Hugging Face Xet storage system, including Merkle hashes, metadata shards, and Xorb objects.
- MerkleHash — 256-bit content-addressed hash used throughout the system
- Metadata shards — Compact shard format mapping file ranges to Xorb chunks
- Xorb objects — Content-addressed storage objects with byte-grouping compression
- Data structures — Specialized hash maps and utilities for deduplication
This crate is part of xet-core, the Rust backend for huggingface_hub.
Apache-2.0