hfgit

Download Hugging Face repositories with plain git + git-lfs and install them into the Hugging Face hub cache, so that transformers, vllm, diffusers, etc. find them exactly as if hf download had fetched them.

Use it when the official hf / huggingface-cli tool is misbehaving, or when you simply prefer git's well-understood transport.

hfgit Qwen/Qwen3-0.6B

hfimport does the same for a model directory that already exists on the local machine (e.g. one you created or converted), with no git involved:

hfimport ./my-model MyOrg/my-model

Requirements

• git
• git-lfs (apt install git-lfs / brew install git-lfs) — a one-time git lfs install is not required; the script passes the filters it needs.
• bash 4+

hfimport needs only bash and shasum.

hfgit

hfgit <org>/<name> [FILE ...] [options]

Option	Description
--repo-type TYPE	model (default), dataset, or space
--revision REV	Branch, tag, or full 40-char commit SHA. Default main
--include PATTERN	Only fetch LFS files matching the glob. Repeatable, or comma-separated
--exclude PATTERN	Skip LFS files matching the glob. Repeatable, or comma-separated
--local-dir DIR	Clone into DIR as a normal git checkout instead of the cache
--cache-dir DIR	Cache root. Default $HF_HUB_CACHE, then $HF_HOME/hub, then ~/.cache/huggingface/hub
--token TOKEN	Auth token. Default $HF_TOKEN, then ~/.cache/huggingface/token
--jobs N	Concurrent LFS transfers (default 8)
--endpoint URL	Hub endpoint. Default $HF_ENDPOINT or https://huggingface.co
--force	Re-download even if the snapshot already exists
-q, --quiet	Print only the final path
-h, --help	Show help

Positional FILE arguments are shorthand for --include FILE.

On success the path of the installed snapshot is printed to stdout (like hf download), and all progress goes to stderr, so it's safe to use in $(...).

Examples

# Whole model into the cache
hfgit Qwen/Qwen3-0.6B

# Only specific files
hfgit Qwen/Qwen3-0.6B config.json tokenizer.json

# Only safetensors, skipping any consolidated/duplicate weights
hfgit mistralai/Mistral-7B-Instruct-v0.3 \
    --include '*.safetensors' --exclude 'consolidated*'

# Gated model (accept the license on the model page first)
hfgit meta-llama/Llama-3.1-8B-Instruct --token hf_xxxxxxxx

# A dataset subset
hfgit HuggingFaceFW/fineweb --repo-type dataset --include 'sample/10BT/*'

# Pin an exact commit
hfgit Qwen/Qwen3-0.6B --revision 83b669136f67d235730c7705afe0e633ec25eabb

# Plain checkout outside the cache (keeps .git, so `git pull` works later)
hfgit Qwen/Qwen3-0.6B --local-dir ./qwen3-0.6b

# Feed the path straight to something
python -m vllm.entrypoints.openai.api_server --model "$(hfgit -q Qwen/Qwen3-0.6B)"

Authentication

Gated and private repos need a token with at least read scope (https://huggingface.co/settings/tokens). The script looks, in order, at --token, $HF_TOKEN, and the file ~/.cache/huggingface/token that hf auth login writes.

The token is handed to git through an ephemeral credential helper for the duration of the command. It is never written to .git/config, never placed in the clone URL, and does not appear in ps output as part of a URL.

How it works

Every Hugging Face repo is a git repository with the large files in Git LFS, and the "revision" the hub client keys on is simply the git commit SHA. So:

1. Shallow-clone the repo with GIT_LFS_SKIP_SMUDGE=1 — this fetches only the small files and ~130-byte LFS pointers, and is fast.
2. git lfs pull with the include/exclude filters to fetch the weights you actually asked for. Any LFS file that was filtered out is deleted so no application ever opens a pointer file thinking it's a model.
3. Remove .git and move the tree into <cache>/models--<org>--<name>/snapshots/<commit>/.
4. Write the commit SHA to <cache>/models--<org>--<name>/refs/<revision>.

huggingface_hub resolves a repo id by reading refs/<revision> to get the commit, then opening snapshots/<commit>/<file>. That is the entire read path, so those two directories are all that is created.

The official downloader also writes blobs/ (content-addressed storage that snapshots/ symlinks into, for dedup across revisions), trees/ (a cached file listing per commit, new in huggingface_hub 1.28, which lets it skip a HEAD request per file when online), and .no_exist/ (a negative cache of 404s). None of these are consulted when loading a model, and their absence does not affect anything except one extra HTTP round-trip per file the first time an online lookup happens.

Disk usage

The clone happens in a temp directory inside the cache (<cache>/models--…/.hfgit-tmp.XXXXXX/), never in /tmp, so the final move is an atomic rename on the same filesystem and no weights are ever copied.

During git lfs pull, git-lfs keeps one copy of each object in .git/lfs/objects/ and one in the working tree, so you transiently need about 2× the model size free. The .git directory is deleted immediately after the pull, bringing it back to 1×. In --local-dir mode, .git/lfs/objects is removed but the rest of .git is kept so the checkout stays updatable.

Re-running

• Same repo + revision already installed, no filters: exits immediately and prints the existing snapshot path. Use --force to re-fetch.
• With --include/--exclude: always clones and merges new files into the existing snapshot for that commit, so you can fetch a repo incrementally (e.g. config.json first, weights later).
• Note that refs/main is only updated by running hfgit again; upstream changes are not detected automatically.

Caveats

• Online lookups can bypass the cache. When not in offline mode, huggingface_hub first asks the Hub for the current commit of main. If the upstream repo has been updated since your download, the commit will not match your snapshot and the library will try to download the new revision. Either set HF_HUB_OFFLINE=1, pass revision=<sha> in from_pretrained, or re-run hfgit to fetch the update.
• Clones are always --depth 1, so repos with a long history of huge weight commits cost nothing extra.
• For cache inspection use hf cache scan / huggingface-cli scan-cache, which read the same directories and work fine with snapshots produced by these tools (they will just report "no blobs").

hfimport

hfimport <src-dir> <org>/<name> [options]

Option	Description
--repo-type TYPE	model (default), dataset, or space
--revision NAME	Ref name to write under refs/. Default main
--cache-dir DIR	Cache root. Same resolution order as hfgit
--force	Replace an existing snapshot with the same id
-q, --quiet	Print only the final path
-h, --help	Show help

<org>/<name> is the id applications will load the model by; it does not need to exist on the Hub. On success the snapshot path is printed to stdout.

How it works

There is no git commit to key the snapshot on, so the snapshot id is a sha1 over the sorted relative paths and contents of every regular file in the source directory. It is 40 hex characters like a real commit and is deterministic: the same content always yields the same id.

1. Hash the source directory to get the id.
2. cp -a the directory into a temp dir inside the cache, then rename it to <cache>/models--<org>--<name>/snapshots/<id>/.
3. Write the id to <cache>/models--<org>--<name>/refs/<revision>.

Hashing reads every file once, so importing a large model costs roughly one read of the weights on top of the copy.

Re-running

• Same content already installed: skips the copy, repoints refs/<revision> at the existing snapshot, and prints its path.
• Changed content: a new snapshot id, a new copy, and refs/<revision> moves to it. Old snapshots are left in place.
• --force deletes and recreates the snapshot for the current id.

Verifying an install

HF_HUB_OFFLINE=1 python -c "
from transformers import AutoConfig
print(AutoConfig.from_pretrained('Qwen/Qwen3-0.6B'))
"

HF_HUB_OFFLINE=1 guarantees the result came from the local cache and not from a fresh download.