fix(embeddings): self-heal from unresolved Git LFS pointer, don't fail forever

Root cause of the embeddings_status: "failed" incident: a checkout without
git-lfs installed leaves the vendored default embedding model
(crates/ci-core/assets/potion-code-16m/model.safetensors) as a ~130-byte LFS
pointer stub instead of real weights. `include_bytes!` bakes that stub into
the binary either way, so the build "succeeds" and the failure only
surfaces at runtime, permanently, with no clear diagnosis.

- Embedder::load now detects an unusable vendored asset and automatically
  falls back to a one-time HuggingFace Hub download of the same default
  model (cached locally afterward), instead of failing forever. This is a
  functionality/reliability concern, not a privacy one — no code or repo
  content is ever sent anywhere; only a public static model file is fetched.
- New semantic_search.allow_network_fallback config flag (default true) for
  anyone who wants semantic search to stay strictly zero-network: with it
  set to false, an unusable vendored asset reports the new
  embeddings_status: "offline_unavailable" (a known policy outcome) instead
  of silently attempting a network call or reporting the more generic
  "failed".
- Hardened the infra that produced the incident: session-start-build-ci.sh
  and the documented cloud Setup Script now attempt `git lfs pull` before
  building; scripts/mcp-launcher.sh's is_binary_fresh also treats vendored
  assets as freshness inputs, so a fixed LFS asset invalidates a
  previously-built stale binary instead of being silently ignored.
- Regression test (default_vendored_asset_is_not_an_lfs_pointer) catches
  this exact failure mode at test time instead of at first-run in
  production.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X83MzmbseyW5v3j9mka5Sc

Author: claude

.claude/hooks/session-start-build-ci.sh

@@ -31,6 +31,28 @@ if ! command -v cargo >/dev/null 2>&1; then
   exit 0
 fi
 
+# Resolve any unresolved Git LFS pointer stubs (assets/potion-code-16m/* —
+# the vendored embedding model — and .ci-bin/**/ci) BEFORE building. Real
+# incident this guards against: a checkout without git-lfs installed leaves
+# ~130-byte pointer text in place of real file content; `cargo build` still
+# succeeds (it just bakes that pointer text into the binary via
+# `include_bytes!`), and the failure only surfaces later, silently, as
+# `embeddings_status: "failed"` at runtime — not a build error, so nothing
+# here would have caught it otherwise. Every step is best-effort and
+# non-fatal (`|| true`) — this hook must degrade, not break the session, and
+# `Embedder::load`'s own runtime fallback (network download) plus
+# `embeddings_status: "offline_unavailable"` messaging are the safety net if
+# this doesn't fully resolve it (e.g. no apt, no network, git-lfs install
+# blocked). See docs/cloud-environment-setup.md for the full picture.
+if command -v git >/dev/null 2>&1 && grep -q 'filter=lfs' .gitattributes 2>/dev/null; then
+  if ! git lfs version >/dev/null 2>&1 && command -v apt-get >/dev/null 2>&1; then
+    apt-get install -y git-lfs >/dev/null 2>&1 || true
+  fi
+  if git lfs version >/dev/null 2>&1; then
+    git lfs pull >/dev/null 2>&1 || true
+  fi
+fi
+
 build_output=$(cargo build --quiet -p ci-cli 2>&1)
 build_status=$?
 

README.md

@@ -103,8 +103,15 @@ agent: "tôi cần sửa hàm getUserByEmail"
   còn phụ thuộc extension C nào, nên hoạt động giống hệt trên mọi platform release (trước đây
   `sqlite-vec` không compile được trên musl libc, khiến bản Linux/Docker bị tắt semantic). Model mặc
   định (`minishlab/potion-code-16M`, MIT license) được vendor sẵn vào binary lúc compile
-  (`crates/ci-core/assets/potion-code-16m/`, qua Git LFS) — load model mặc định không cần mạng, chỉ
-  model tuỳ biến qua `semantic_search.model` mới tải từ HuggingFace Hub.
+  (`crates/ci-core/assets/potion-code-16m/`, qua Git LFS) — load model mặc định thường không cần
+  mạng. Nếu asset vendor bị hỏng/thiếu (vd checkout thiếu `git-lfs` nên còn nguyên LFS pointer thay
+  vì nội dung thật — không giả định, đã xảy ra thật), `Embedder::load` tự fallback sang tải model
+  mặc định đó qua HuggingFace Hub 1 lần rồi cache local, thay vì `embeddings_status` treo ở
+  `"failed"` vĩnh viễn; set `semantic_search.allow_network_fallback: false` để tắt hẳn fallback này
+  và giữ đúng zero-network tuyệt đối (lúc đó status báo `"offline_unavailable"` thay vì mập mờ). Model
+  tuỳ biến qua `semantic_search.model` luôn tải từ HuggingFace Hub như trước, không đổi. Lưu ý: đây
+  chỉ là tải 1 file model tĩnh, công khai — không liên quan tới cam kết "không gọi ra ngoài" của `ci`
+  (cam kết đó là về code/dữ liệu repo, không phải về việc tải asset).
 - **Grep/glob thật, quét trực tiếp trên đĩa** — `search(kind="grep")` dùng regex thật (crate `regex`)
   + glob filter (`globset`) qua walker tôn trọng `.gitignore`/`.git/info/exclude` thật (crate
   `ignore`), không qua FTS/DB nên phủ được cả file indexer không parse (`Cargo.toml`, `docs/*.md`).

crates/ci-core/src/config.rs

@@ -102,6 +102,21 @@ pub struct SemanticSearchConfig {
     pub model: String,
     pub dimensions: usize,
     pub index_on_startup: bool,
+    /// When the vendored default-model asset is unusable (e.g. an unresolved
+    /// Git LFS pointer left by a checkout that never ran `git lfs pull`/had
+    /// git-lfs installed — not hypothetical, see the incident this field was
+    /// added for), `true` (default) lets `Embedder::load` fall back to a
+    /// one-time HuggingFace Hub download of the same default model, cached
+    /// locally afterward (`~/.cache/huggingface`) — degrade to *slower on
+    /// this run only*, not permanently `failed`. `false` keeps semantic
+    /// search strictly zero-network: embeddings report
+    /// `embeddings_status: "offline_unavailable"` instead of ever touching
+    /// the network, until the vendored asset is fixed locally. Either way,
+    /// this governs recovery from a *broken local asset* only — it does not
+    /// change plain `search`/`callers`/etc., which never touch the network
+    /// and never send code/repo content anywhere; that guarantee is
+    /// independent of this flag.
+    pub allow_network_fallback: bool,
 }
 
 impl Default for SemanticSearchConfig {
@@ -114,6 +129,7 @@ impl Default for SemanticSearchConfig {
             model: crate::embedding::DEFAULT_MODEL_ID.into(),
             dimensions: 256,
             index_on_startup: true,
+            allow_network_fallback: true,
         }
     }
 }

crates/ci-core/src/embedding.rs

@@ -88,18 +88,55 @@ mod imp {
     static DEFAULT_TOKENIZER: &[u8] = include_bytes!("../assets/potion-code-16m/tokenizer.json");
     static DEFAULT_WEIGHTS: &[u8] = include_bytes!("../assets/potion-code-16m/model.safetensors");
 
+    /// True if `bytes` is an unresolved Git LFS pointer stub rather than real
+    /// file content — happens when `git lfs pull`/the smudge filter never ran
+    /// during checkout (e.g. git-lfs not installed in the environment). Real
+    /// model weights are tens of MB; a pointer stub is ~130 bytes starting
+    /// with this exact line (mirrors `is_lfs_pointer` in
+    /// `scripts/mcp-launcher.sh`, which checks the same thing for the
+    /// prebuilt `ci` binary — kept as a separate copy here since that's a
+    /// shell script and this is compiled into the binary itself). Length-
+    /// capped so a real, coincidentally-short binary blob is never misread as
+    /// a pointer. `pub(crate)` (not exported at the module's public surface)
+    /// purely so the `tests` module below — a sibling of `imp`, not a
+    /// descendant — can unit-test the length-cap/content-match logic
+    /// directly instead of only indirectly through `default_vendored_asset_unusable`.
+    pub(crate) fn is_lfs_pointer_stub(bytes: &[u8]) -> bool {
+        bytes.len() < 512 && bytes.starts_with(b"version https://git-lfs")
+    }
+
+    /// True if the vendored default-model asset baked into this binary is an
+    /// unresolved Git LFS pointer stub, not real weights — checked before
+    /// `Embedder::load` decides whether a network fallback is even needed,
+    /// and exposed publicly so `ci-server`'s `bootstrap_embeddings` can
+    /// short-circuit to `EmbedStatus::OfflineUnavailable` without a network
+    /// attempt when `semantic_search.allow_network_fallback` is `false`.
+    pub fn default_vendored_asset_unusable() -> bool {
+        is_lfs_pointer_stub(DEFAULT_WEIGHTS)
+    }
+
     /// A loaded static embedding model.
     pub struct Embedder {
         model: StaticModel,
         dim: usize,
     }
 
     impl Embedder {
-        /// Load `model_id`. The default model id (`DEFAULT_MODEL_ID`) loads
-        /// from the bytes vendored into the binary; any other id (a custom
-        /// model configured via `semantic_search.model`) still resolves via
-        /// `from_pretrained` — a local path, or a HuggingFace Hub download.
-        /// Output is L2-normalised so cosine distance behaves well.
+        /// Load `model_id`. The default model id (`DEFAULT_MODEL_ID`)
+        /// normally loads from the bytes vendored into the binary — zero-I/O,
+        /// zero-network. If that fails (most commonly: the vendored asset is
+        /// an unresolved Git LFS pointer, see `is_lfs_pointer_stub`), this
+        /// automatically falls back to `from_pretrained`, which downloads the
+        /// same default model from the HuggingFace Hub once and caches it
+        /// locally (`~/.cache/huggingface`) — the caller (`bootstrap_embeddings`)
+        /// is expected to have already checked `semantic_search.allow_network_fallback`
+        /// via `default_vendored_asset_unusable` before ever calling this, so
+        /// reaching this fallback here always means the caller already
+        /// consented to a network attempt. A custom model id (configured via
+        /// `semantic_search.model`) always resolves via `from_pretrained` — a
+        /// local path, or a HuggingFace Hub download, same as before this
+        /// fallback existed. Output is L2-normalised so cosine distance
+        /// behaves well.
         ///
         /// `dim` is only a hint (from `semantic_search.dimensions` in
         /// config) — model2vec-rs exposes no API to query a loaded model's
@@ -109,13 +146,37 @@ mod imp {
         /// silently mislabeling every vector this `Embedder` ever produces.
         pub fn load(model_id: &str, dim: usize) -> anyhow::Result<Self> {
             let model = if model_id == DEFAULT_MODEL_ID {
-                StaticModel::from_bytes(
+                match StaticModel::from_bytes(
                     DEFAULT_TOKENIZER,
                     DEFAULT_WEIGHTS,
                     DEFAULT_CONFIG,
                     Some(true),
-                )
-                .map_err(|e| anyhow::anyhow!("load vendored embedding model: {e}"))?
+                ) {
+                    Ok(m) => m,
+                    Err(vendored_err) => {
+                        if is_lfs_pointer_stub(DEFAULT_WEIGHTS) {
+                            tracing::warn!(
+                                "vendored embedding model asset is an unresolved Git LFS \
+                                 pointer (git-lfs not installed, or the checkout skipped the \
+                                 smudge filter) — falling back to a one-time HuggingFace Hub \
+                                 download of '{model_id}', cached locally afterward"
+                            );
+                        } else {
+                            tracing::warn!(
+                                "vendored embedding model failed to load ({vendored_err}) — \
+                                 falling back to a one-time HuggingFace Hub download of \
+                                 '{model_id}', cached locally afterward"
+                            );
+                        }
+                        StaticModel::from_pretrained(DEFAULT_MODEL_ID, None, Some(true), None)
+                            .map_err(|e| {
+                                anyhow::anyhow!(
+                                    "vendored load failed ({vendored_err}); \
+                                     network fallback also failed: {e}"
+                                )
+                            })?
+                    }
+                }
             } else {
                 StaticModel::from_pretrained(model_id, None, Some(true), None)
                     .map_err(|e| anyhow::anyhow!("load embedding model '{model_id}': {e}"))?
@@ -418,6 +479,13 @@ mod imp {
         Ok(())
     }
 
+    /// Always `false` — there's no vendored asset to be unusable when the
+    /// `embeddings` feature itself is off; `Embedder::load`'s own stub
+    /// failure below is what surfaces this build's real limitation.
+    pub fn default_vendored_asset_unusable() -> bool {
+        false
+    }
+
     /// Stub embedder — `load` always fails, so callers keep `None` and degrade.
     pub struct Embedder;
 
@@ -460,9 +528,9 @@ mod imp {
 }
 
 pub use imp::{
-    Embedder, create_chunk_embedding_table, create_embedding_table, embed_pending,
-    embed_pending_chunks, knn, knn_chunks, prune_orphaned_chunk_vecs, store_chunk_embedding,
-    store_embedding,
+    Embedder, create_chunk_embedding_table, create_embedding_table, default_vendored_asset_unusable,
+    embed_pending, embed_pending_chunks, knn, knn_chunks, prune_orphaned_chunk_vecs,
+    store_chunk_embedding, store_embedding,
 };
 
 #[cfg(test)]
@@ -478,6 +546,49 @@ mod tests {
         assert_eq!(symbol_doc("run", "", ""), "run");
     }
 
+    /// Regression for the exact incident this function was added for: a
+    /// checkout without git-lfs installed (or one that skipped the smudge
+    /// filter) leaves `assets/potion-code-16m/model.safetensors` as a ~130-
+    /// byte Git LFS pointer stub instead of real weights — `include_bytes!`
+    /// happily bakes that stub into the binary, and `Embedder::load` used to
+    /// fail permanently (`embeddings_status: "failed"`) with no clear signal
+    /// why. If this test ever fails, `git lfs pull` didn't run before this
+    /// crate was built.
+    #[cfg(feature = "embeddings")]
+    #[test]
+    fn default_vendored_asset_is_not_an_lfs_pointer() {
+        assert!(
+            !imp::default_vendored_asset_unusable(),
+            "DEFAULT_WEIGHTS looks like an unresolved Git LFS pointer, not real model weights \
+             — run `git lfs pull`"
+        );
+    }
+
+    #[cfg(feature = "embeddings")]
+    #[test]
+    fn is_lfs_pointer_stub_detects_pointer_text_not_real_weights() {
+        let pointer =
+            b"version https://git-lfs.github.com/spec/v1\noid sha256:abc\nsize 123\n".as_slice();
+        assert!(imp::is_lfs_pointer_stub(pointer));
+        assert!(
+            !imp::is_lfs_pointer_stub(&[0u8; 1000]),
+            "1000 zero bytes is not a pointer stub by content, regardless of size"
+        );
+    }
+
+    #[cfg(feature = "embeddings")]
+    #[test]
+    fn is_lfs_pointer_stub_length_cap_avoids_false_positive_on_large_content() {
+        let mut huge = b"version https://git-lfs.github.com/spec/v1".to_vec();
+        huge.resize(10_000, 0);
+        assert!(
+            !imp::is_lfs_pointer_stub(&huge),
+            "10KB of content must not be misread as a pointer stub even if it starts \
+             with the marker text — real weights are never this small either way, but \
+             the cap is the actual safety net, not the content match alone"
+        );
+    }
+
     /// Regression for the vendored default model (`include_bytes!` in
     /// `Embedder::load`): loads with zero network, produces the right
     /// dimensionality, and is L2-normalised. Catches a bad asset path, a

crates/ci-core/src/types.rs

@@ -108,6 +108,14 @@ pub enum EmbedStatus {
     Embedding,
     Ready,
     Failed,
+    /// The vendored default-model asset is unusable (e.g. an unresolved Git
+    /// LFS pointer) and `semantic_search.allow_network_fallback` is `false`,
+    /// so semantic search stays off rather than reaching the network — a
+    /// deliberate policy outcome, distinct from `Failed` (an unexpected
+    /// error). `indexing_status(retry_embeddings: true)` re-checks this the
+    /// same way it reclaims `Failed`, so flipping the config and retrying
+    /// recovers without a restart.
+    OfflineUnavailable,
 }
 
 impl EmbedStatus {
@@ -118,6 +126,7 @@ impl EmbedStatus {
             Self::Embedding => "embedding",
             Self::Ready => "ready",
             Self::Failed => "failed",
+            Self::OfflineUnavailable => "offline_unavailable",
         }
     }
 }

crates/ci-server/src/lib.rs

@@ -138,6 +138,21 @@ pub async fn serve_stdio_with_preset(
     Ok(())
 }
 
+/// True when semantic search should stop before ever attempting a network
+/// call: the configured model is the vendored default, that vendored asset
+/// is unusable (see `ci_core::embedding::default_vendored_asset_unusable`),
+/// and the config has not opted into a network fallback. Pulled out as a
+/// pure function (taking the three already-evaluated booleans, not
+/// re-deriving them) so the policy logic is unit-testable without touching
+/// the real vendored asset or the network — see the `tests` module below.
+fn embeddings_blocked_by_offline_policy(
+    is_default_model: bool,
+    vendored_asset_unusable: bool,
+    allow_network_fallback: bool,
+) -> bool {
+    is_default_model && vendored_asset_unusable && !allow_network_fallback
+}
+
 /// Load the embedding model, create the vector table, embed all symbols, and
 /// publish the model + status. Runs on the indexer thread after the graph is
 /// built (and again from `indexing_status`'s `retry_embeddings` after a prior
@@ -149,6 +164,20 @@ pub fn bootstrap_embeddings(
     status: &Arc<RwLock<EmbedStatus>>,
 ) {
     *status.write().unwrap() = EmbedStatus::Downloading;
+    if embeddings_blocked_by_offline_policy(
+        semantic.model == ci_core::embedding::DEFAULT_MODEL_ID,
+        ci_core::embedding::default_vendored_asset_unusable(),
+        semantic.allow_network_fallback,
+    ) {
+        tracing::warn!(
+            "Vendored embedding model is an unresolved Git LFS pointer and \
+             semantic_search.allow_network_fallback is false — embeddings unavailable this \
+             run. Run `git lfs pull` to fix the vendored asset, or set \
+             allow_network_fallback=true to download it instead, then retry_embeddings."
+        );
+        *status.write().unwrap() = EmbedStatus::OfflineUnavailable;
+        return;
+    }
     if semantic.model == ci_core::embedding::DEFAULT_MODEL_ID {
         tracing::info!(
             "Loading embedding model `{}` (vendored in the binary, no network needed)...",
@@ -306,3 +335,35 @@ fn current_git_head_short(project_root: &std::path::Path) -> Option<String> {
     let trimmed = text.trim();
     (!trimmed.is_empty()).then(|| trimmed.to_string())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::embeddings_blocked_by_offline_policy as blocked;
+
+    /// All three conditions must hold — a custom (non-default) model, a fine
+    /// vendored asset, or an allowed network fallback each independently
+    /// mean "don't block", only their conjunction does.
+    #[test]
+    fn embeddings_blocked_by_offline_policy_only_when_all_three_conditions_hold() {
+        assert!(
+            blocked(true, true, false),
+            "default model + unusable vendored asset + fallback disabled -> blocked"
+        );
+        assert!(
+            !blocked(false, true, false),
+            "a custom model was never going to use the vendored asset — unaffected"
+        );
+        assert!(
+            !blocked(true, false, false),
+            "vendored asset is fine — no fallback ever needed"
+        );
+        assert!(
+            !blocked(true, true, true),
+            "fallback explicitly allowed — proceed to Embedder::load's own fallback"
+        );
+        assert!(!blocked(false, false, false));
+        assert!(!blocked(false, false, true));
+        assert!(!blocked(false, true, true));
+        assert!(!blocked(true, false, true));
+    }
+}

crates/ci-server/src/tools/common.rs

@@ -213,16 +213,20 @@ impl CodeIntelligenceServer {
 
     /// Re-runs the embedding bootstrap in the background when it previously
     /// failed (model load, vector-table creation, or embedding all set status
-    /// to `Failed`). No-op for any other status: `Ready`/`Embedding`/
-    /// `Downloading` are already done or in flight, and `Disabled` means
-    /// semantic search isn't turned on in config. Opens its own DB connection
-    /// so the retry doesn't hold the shared connection mutex for its duration.
+    /// to `Failed`) or was blocked by offline policy (`OfflineUnavailable` —
+    /// e.g. the caller since flipped `semantic_search.allow_network_fallback`
+    /// to `true` or ran `git lfs pull` and wants to try again). No-op for any
+    /// other status: `Ready`/`Embedding`/`Downloading` are already done or in
+    /// flight, and `Disabled` means semantic search isn't turned on in
+    /// config. Opens its own DB connection so the retry doesn't hold the
+    /// shared connection mutex for its duration.
     pub(crate) fn retry_embeddings_if_failed(&self) {
-        // Claim the retry synchronously (Failed -> Downloading) so two
-        // overlapping `retry_embeddings` requests can't both spawn a bootstrap.
+        // Claim the retry synchronously (Failed/OfflineUnavailable ->
+        // Downloading) so two overlapping `retry_embeddings` requests can't
+        // both spawn a bootstrap.
         {
             let mut status = self.embed_status.write().unwrap();
-            if *status != EmbedStatus::Failed {
+            if *status != EmbedStatus::Failed && *status != EmbedStatus::OfflineUnavailable {
                 return;
             }
             *status = EmbedStatus::Downloading;