feat: add HuggingFace as a local-first model provider (GGUF/llama-cpp)

[Empreiteiro]

Draft for discussion. Opens up a parallel local-inference path that doesn't require Ollama or an external API key. Happy to split into smaller PRs (catalog/registry + adapter + frontend icon) if that's preferable for review.

Summary

• Registers HuggingFace alongside the other configurable providers, running models locally via llama-cpp-python + GGUF (no torch, no transformers, fork-safe on macOS arm64).
• Onboarding ships a single bundled model — bartowski/SmolLM2-360M-Instruct-GGUF (Q4_K_M, ~270MB). Settings → Model Providers shows one toggle for it, on by default.
• Toggle-on triggers background download — flipping a HuggingFace model on in POST /enabled_models schedules a single-file download so the cache is warm by the next invocation.
• Add-more-models via API — POST /api/v1/models/huggingface/download {"model_id": "<gguf-repo-id>"} accepts any HF repo id that publishes GGUF weights.
• Optional startup prefetch — LANGFLOW_PREFETCH_HF_DEFAULT=true warms the cache on lifespan start. Off by default.
• Subprocess-isolated downloads — if hf_hub_download ever crashes the worker, a subprocess retry kicks in so the parent uvicorn worker survives.

Why GGUF / llama-cpp instead of transformers

The first iteration used langchain_huggingface.HuggingFacePipeline (transformers + torch). On macOS arm64 + Python 3.12, two SIGSEGV vectors broke this end-to-end:

1. Importing torch inside a forked uvicorn worker SIGSEGV'd at first device init.
2. huggingface_hub.snapshot_download's parallel fetcher SIGSEGV'd at 0% download progress.

Mitigations (device=-1, low_cpu_mem_usage=True, max_workers=1, OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES) only narrowed the failure window. Switching to llama-cpp-python removes the torch dependency entirely — it's a C/C++ backend with a thin Python binding, fork-safe, and quantized models are smaller and faster on CPU.

What changed

Provider registry & catalog

• src/lfx/src/lfx/base/models/huggingface_constants.py (new) — single bundled GGUF entry (bartowski/SmolLM2-360M-Instruct-GGUF, default=True).
• src/lfx/src/lfx/base/models/model_metadata.py — registers HuggingFace in MODEL_PROVIDER_METADATA. HUGGINGFACEHUB_API_TOKEN is optional (only needed for gated repos).
• src/lfx/src/lfx/base/models/unified_models/provider_queries.py — wires HUGGINGFACE_MODELS_DETAILED into get_models_detailed().
• src/lfx/src/lfx/base/models/unified_models/class_registry.py — adds ChatHuggingFace to the lazy import map.

Local chat adapter (llama-cpp backend)

• src/lfx/src/lfx/base/models/huggingface_chat_model.py (new) — factory ChatHuggingFace translates the unified kwargs into a langchain_community.chat_models.ChatLlamaCpp instance. Picks the right .gguf filename via a per-repo override (GGUF_FILENAME_BY_REPO) with a <name>-Q4_K_M.gguf fallback for bartowski-style repos.
  • In-process cache (_LLAMA_CACHE) keys instances by (model_path, temperature, max_tokens) so repeat calls don't re-mmap the model.
  • download_model() uses hf_hub_download (single-file). Disables xet, hf_transfer, telemetry, and progress bars at module import to avoid worker-fork SIGSEGV. If the in-process call still raises, retries the same call inside an isolated subprocess so a hard crash can't take down the parent worker.
  • list_installed_models() filters cache entries to repos that actually contain a .gguf file.
  • n_threads = cpu_count() - 1, n_gpu_layers = 0 by default (CPU-only).

Local-provider plumbing

• src/lfx/src/lfx/base/models/unified_models/instantiation.py — HuggingFace joins Ollama in the "no API key required" exception.
• src/lfx/src/lfx/base/models/unified_models/credentials.py — providers whose variables are all optional now stay enabled by default; also adds an HF token validator that pings huggingface.co/api/whoami-v2.
• src/lfx/src/lfx/base/models/unified_models/build_config.py — for optional provider variables, only auto-install load_from_db=True when the variable is actually configured. Fixes "<VAR> variable not found." errors at flow build time.

API

• src/backend/base/langflow/api/v1/models.py
  • GET /models/huggingface/installed — list GGUF repo ids in the local Hub cache.
  • POST /models/huggingface/download — download a .gguf file for the given repo id (validates length, reuses saved token for gated pulls).
  • POST /enabled_models schedules a background hf_hub_download whenever a HuggingFace model is toggled on. Failures are logged but never block the toggle save.

Lifespan startup

• src/backend/base/langflow/main.py — _prefetch_default_huggingface_model() runs as a background task during lifespan startup only when LANGFLOW_PREFETCH_HF_DEFAULT=true. Tracked alongside sync_flows_from_fs_task and mcp_init_task; cancelled on lifespan shutdown.

Frontend

• src/frontend/src/controllers/API/queries/models/use-get-model-providers.ts — adds HuggingFace to the hardcoded provider→icon lookup. Without this, the Agent component (which filters tool_calling=True and so doesn't see the HF model in its options list) was falling through to the default Bot lucide icon for HF entries while the Language Model component rendered the correct HF logo.

Packaging

• src/backend/base/pyproject.toml — adds langflow-base[local] to the [complete] extra so fresh pip install langflow installs pull llama-cpp-python automatically.
• uv.lock — regenerated. The Makefile uses uv sync --frozen for make backend/make run_cli, which only installs what's pinned in the lockfile; without this regeneration contributors wouldn't get the new dep on a fresh checkout.

How to use

1. Start Langflow — make backend. Server boots clean, no prefetch by default.
2. Settings → Model Providers → HuggingFace — toggle is on. Toggling fires a background hf_hub_download of the GGUF file. First invocation from a flow loads the GGUF and answers.
3. Install another model — point at any GGUF repo + filename pair:

   curl -X POST http://localhost:7860/api/v1/models/huggingface/download \
        -H 'Content-Type: application/json' \
        -d '{"model_id": "bartowski/Qwen2.5-0.5B-Instruct-GGUF"}'

   Filename auto-resolves to Qwen2.5-0.5B-Instruct-Q4_K_M.gguf via the bartowski-style heuristic. For non-conforming repos, add an entry to GGUF_FILENAME_BY_REPO.
4. Gated models: paste HUGGINGFACEHUB_API_TOKEN in Settings → Global Variables. Required for the few GGUF repos that gate downloads.
5. (Optional) Warm the cache at startup: LANGFLOW_PREFETCH_HF_DEFAULT=true make backend.

Tool calling

The bundled SmolLM2-360M-Instruct does not have reliable tool calling — it's marked tool_calling=False in the catalog. To use the HuggingFace provider with the Agent component's tools, install a tool-calling-capable GGUF on demand, e.g.:

curl -X POST http://localhost:7860/api/v1/models/huggingface/download \
     -H 'Content-Type: application/json' \
     -d '{"model_id": "bartowski/Hermes-3-Llama-3.2-3B-GGUF"}'

Dependencies

• llama-cpp-python — already in langflow-base[local], now pulled by [complete] (which langflow consumes).
• langchain-community — already a core dep; provides ChatLlamaCpp.
• huggingface_hub — already a core dep; provides hf_hub_download.

No new top-level dependencies. The transformers / torch / langchain-huggingface stack is no longer touched by the HF chat path.

Env var summary

Variable	Default	Effect
LANGFLOW_PREFETCH_HF_DEFAULT	unset / false	When true/1/yes, prefetch the bundled GGUF during lifespan startup.
HUGGINGFACEHUB_API_TOKEN	unset	Optional. Required only to download gated repos.
HF_HUB_DISABLE_XET	1 (forced)	Disabled by the adapter to avoid the parallel xet backend.
HF_HUB_ENABLE_HF_TRANSFER	0 (forced)	Disabled by the adapter to keep downloads in the plain HTTP path.

Test plan

• ruff check + ruff format clean across all modified files.
• Existing model tests pass (test_max_tokens_propagation.py, test_model_input_fixes.py) — 37 passed, 1 skipped.
• Smoke: catalog has 1 entry (bartowski/SmolLM2-360M-Instruct-GGUF, default=True); _pick_gguf_filename resolves to SmolLM2-360M-Instruct-Q4_K_M.gguf; list_installed_models() reads the real cache.
• Manual: macOS arm64 — server boots cleanly, no SIGSEGV. Toggle the bundled model → background download completes → flow invocation returns a response from the local GGUF.
• Manual: Agent component on macOS arm64 — HF model dropdown trigger renders the HuggingFace logo (after frontend rebuild + hard reload).
• Manual: POST /models/huggingface/download with a different bartowski-style GGUF repo → confirm download → flow can switch to it.
• Manual: try a tool-calling-capable model (bartowski/Hermes-3-Llama-3.2-3B-GGUF) with the Agent component and confirm Calculator/URL tools fire.
• Manual: try a gated model (e.g. bartowski/Llama-3.2-3B-Instruct-GGUF) with a saved HF token.
• Manual: LANGFLOW_PREFETCH_HF_DEFAULT=true on a fresh cache actually warms it during lifespan startup without crashing.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 848239e8-e197-483e-9fd6-faa20599910a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}