fix(daemon): auto-respawn on config mismatch

Author: simonsysun

CHANGELOG.md

@@ -11,6 +11,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `seeklink get PATH:LINE -C N` prints a grep-style context window around a search hit, returning `N` lines before and after the requested line while preserving direct filesystem reads and path-escape protection.
 - `seeklink search --json` and `seeklink status --json` emit stable machine-readable stdout for agents that should not scrape the human text format.
 
+### Fixed
+- `seeklink search` and `seeklink index` now auto-restart a stale daemon when its vault, embedder, or reranker config no longer matches the caller, avoiding repeated cold-start fallbacks after switching vaults or model settings.
+
 ## [0.3.2] - 2026-04-23
 
 Repository cleanup pass. No code changes affecting runtime behavior

README.md

@@ -117,7 +117,7 @@ seeklink search "machine learning"
 # Warm reranker-disabled path: ~10ms per query.
 ```
 
-The daemon stays resident across terminal sessions until you `kill` it or restart. `seeklink search` and `seeklink index` auto-spawn it when missing; `seeklink status` is always cold-start (it only reads SQLite stats, no model load) and `seeklink get` is a direct filesystem read (no daemon involved either).
+The daemon stays resident across terminal sessions until you `kill` it, restart, or switch the default vault/model config. `seeklink search` and `seeklink index` auto-spawn it when missing and auto-restart it if a stale daemon is bound to the wrong vault or model settings; `seeklink status` is always cold-start (it only reads SQLite stats, no model load) and `seeklink get` is a direct filesystem read (no daemon involved either).
 
 ## For agents
 
@@ -158,7 +158,7 @@ Options:
 
 Starts a Unix-socket daemon that keeps the embedding model (and reranker, if enabled) resident in memory. First query after startup takes ~2s for model warmup; warm queries return in ~1-2s with the reranker on (default) or ~10ms with `SEEKLINK_RERANKER_MODEL=""`.
 
-**You almost never run this directly.** `seeklink search` and `seeklink index` auto-spawn a daemon on cold machines when `--vault` is not passed. `seeklink status` is always cold-start (no model load). `seeklink get` is a direct filesystem read (no daemon). The daemon uses `SEEKLINK_VAULT` (or cwd) as its vault and never auto-exits — kill it with `kill` or restart your machine.
+**You almost never run this directly.** `seeklink search` and `seeklink index` auto-spawn a daemon on cold machines when `--vault` is not passed. If an existing daemon was started for a different vault or model config, the CLI shuts it down and starts a matching one. `seeklink status` is always cold-start (no model load). `seeklink get` is a direct filesystem read (no daemon). The daemon uses `SEEKLINK_VAULT` (or cwd) as its vault.
 
 Passing `--vault` always uses cold-start instead of the daemon, because the daemon binds to a single vault at startup.
 

TODOS.md

@@ -48,14 +48,6 @@ indexed files have drifted on disk. The daemon path does not propagate
 those warnings back to clients. Adding a `warnings` field to the daemon
 JSON response would let `cli_client` surface them in the same shape.
 
-### Daemon auto-respawn on config mismatch
-`cli_client.call()` refuses to reuse a daemon bound to a different vault
-or started with a different embedder / reranker (correctness), but falls
-back to cold-start on every subsequent CLI call after a switch until the
-user manually kills the stale daemon. Add a `shutdown` command to the
-daemon protocol so the client can shut down and respawn on mismatch,
-keeping the auto-spawn workflow intact across vault / model switches.
-
 ### Multi-vault daemon support
 The daemon binds to a single socket (`~/.rhizome/seeklink.sock`) regardless
 of vault. For multiple vaults to run concurrent daemons, hash the vault

llms.txt

@@ -42,7 +42,7 @@ seeklink get    PATH:LINE -l N         # read window around a hit
 seeklink get    PATH:LINE -C N         # read N lines before/after a hit
 ```
 
-Set `SEEKLINK_VAULT=<path>` once to omit `--vault` on every call and route through the resident daemon (first call after boot: ~2s; warm: ~1-2s with reranker, ~10ms without).
+Set `SEEKLINK_VAULT=<path>` once to omit `--vault` on every call and route through the resident daemon (first call after boot: ~2s; warm: ~1-2s with reranker, ~10ms without). If that env/model config changes, `search` and `index` auto-restart a stale daemon instead of silently serving the old vault.
 
 ### Output contract
 
@@ -79,5 +79,5 @@ No other codes.
 ### Common failure modes
 
 - Empty results on a fresh vault → index not built yet. Run `seeklink index --vault PATH`.
-- Daemon won't auto-spawn → `--vault` was passed (cold-start mandatory in that case), or another daemon is bound to a different vault. Run `status` to see `vault` / `embedder` / `reranker` and respawn if needed.
+- Daemon won't auto-spawn → `--vault` was passed, which intentionally forces cold-start. Without `--vault`, `search` / `index` should auto-spawn and auto-restart stale daemons when `vault` / `embedder` / `reranker` no longer match.
 - Line numbers look wrong → file was edited after indexing. Re-index. `status` prints a freshness warning on cold-start.

seeklink/cli_client.py

@@ -7,11 +7,10 @@
 fall back to a cold-start in-process execution if they prefer.
 
 Vault-binding guard: the daemon is single-vault (bound at startup via
-SEEKLINK_VAULT or cwd). If a caller passes ``expected_vault`` to
-``call()``, the client first probes the daemon's reported vault via a
-``status`` request and refuses to reuse a daemon bound to a different
-vault — preventing a stale daemon from silently serving (or mutating)
-the wrong database when the user changes SEEKLINK_VAULT or cwd.
+SEEKLINK_VAULT or cwd). If a caller passes expected config to ``call()``,
+the client first probes the daemon's reported status. On mismatch it asks
+the stale daemon to shut down, waits for the socket to clear, and then
+auto-spawns a daemon under the caller's current env/cwd.
 """
 
 from __future__ import annotations
@@ -33,6 +32,7 @@
 # Deferred until multi-vault becomes a real use case.
 SOCKET_PATH = Path.home() / ".rhizome" / "seeklink.sock"
 SPAWN_WAIT_SECONDS = 60.0  # cold start includes model load, give it time
+SHUTDOWN_WAIT_SECONDS = 10.0
 
 
 def call(
@@ -46,21 +46,19 @@ def call(
     """Send a command to the daemon. Auto-spawns daemon if unreachable.
 
     If any ``expected_*`` is provided, probes the daemon's reported
-    status first and returns a failure response if the daemon's vault
-    or model config does not match what the caller expects. Callers
-    should pass these whenever the intended config is inferred from
-    env/cwd rather than an explicit flag — this prevents a stale
-    daemon from:
+    status first and restarts the daemon if its vault or model config
+    does not match what the caller expects. Callers should pass these
+    whenever the intended config is inferred from env/cwd rather than
+    an explicit flag — this prevents a stale daemon from:
 
     - serving or mutating the wrong vault's database (vault mismatch), or
     - returning results computed with the wrong embedder/reranker model
       (config mismatch), which can silently corrupt rankings and, if
       the embedder's vector width changes, make ``search_vec`` fail.
 
-    On mismatch, callers are expected to fall back to an in-process
-    cold-start (which will pick up the current env). A follow-up that
-    auto-respawns the daemon instead of falling back is tracked in
-    TODOS.md.
+    On mismatch, this function requests daemon shutdown and retries
+    through the normal spawn path. If restart fails, callers can still
+    fall back to an in-process cold-start.
 
     Returns the daemon's JSON response as a dict. Never raises — on
     failure returns ``{"ok": False, "error": "..."}``.
@@ -76,62 +74,14 @@ def call(
             return probe  # daemon unreachable / spawn failed
         result = probe.get("result") or {}
 
-        if expected_vault is not None:
-            daemon_vault_raw = result.get("vault")
-            if daemon_vault_raw is None:
-                return {"ok": False, "error": "daemon status returned no vault"}
-            try:
-                daemon_vault = Path(daemon_vault_raw).resolve()
-                want_vault = expected_vault.resolve()
-            except OSError as e:
-                return {"ok": False, "error": f"vault resolution failed: {e}"}
-            if daemon_vault != want_vault:
-                return {
-                    "ok": False,
-                    "error": (
-                        f"daemon is bound to vault {daemon_vault}, but caller "
-                        f"expects {want_vault} — restart the daemon or run "
-                        f"with --vault"
-                    ),
-                }
-
-        if expected_embedder is not None:
-            daemon_embedder = result.get("embedder")
-            if daemon_embedder != expected_embedder:
-                return {
-                    "ok": False,
-                    "error": (
-                        f"daemon embedder is {daemon_embedder!r}, caller "
-                        f"expects {expected_embedder!r} — restart the daemon"
-                    ),
-                }
-
-        if expected_reranker is not None:
-            daemon_reranker = result.get("reranker")
-            # Accept the daemon's self-disabled state even if we expected a
-            # real model name. `run_daemon()` downgrades the reranker to
-            # "disabled" at warmup time on platforms where mlx_lm cannot
-            # load (Linux, Intel macOS). Rejecting that would break the
-            # daemon-first workflow on those supported setups. But if the
-            # caller explicitly asked for "disabled" (via empty
-            # `SEEKLINK_RERANKER_MODEL`), a running daemon with a real
-            # reranker IS a mismatch — the user asked for raw RRF scores
-            # and would silently get reranked ones.
-            is_mismatch = (
-                daemon_reranker != expected_reranker
-                and not (
-                    expected_reranker != "disabled"
-                    and daemon_reranker == "disabled"
-                )
-            )
-            if is_mismatch:
-                return {
-                    "ok": False,
-                    "error": (
-                        f"daemon reranker is {daemon_reranker!r}, caller "
-                        f"expects {expected_reranker!r} — restart the daemon"
-                    ),
-                }
+        mismatch = _config_mismatch_error(
+            result,
+            expected_vault=expected_vault,
+            expected_embedder=expected_embedder,
+            expected_reranker=expected_reranker,
+        )
+        if mismatch is not None:
+            return _restart_and_retry(cmd, args, mismatch)
 
         # All checks passed — send the real command without re-spawning.
         try:
@@ -142,6 +92,98 @@ def call(
     return _call_once_with_spawn(cmd, args)
 
 
+def _config_mismatch_error(
+    status: dict[str, Any],
+    *,
+    expected_vault: Path | None,
+    expected_embedder: str | None,
+    expected_reranker: str | None,
+) -> str | None:
+    """Return a human-readable mismatch reason, or None if config matches."""
+    if expected_vault is not None:
+        daemon_vault_raw = status.get("vault")
+        if daemon_vault_raw is None:
+            return "daemon status returned no vault"
+        try:
+            daemon_vault = Path(daemon_vault_raw).resolve()
+            want_vault = expected_vault.resolve()
+        except OSError as e:
+            return f"vault resolution failed: {e}"
+        if daemon_vault != want_vault:
+            return (
+                f"daemon is bound to vault {daemon_vault}, but caller "
+                f"expects {want_vault}"
+            )
+
+    if expected_embedder is not None:
+        daemon_embedder = status.get("embedder")
+        if daemon_embedder != expected_embedder:
+            return (
+                f"daemon embedder is {daemon_embedder!r}, caller "
+                f"expects {expected_embedder!r}"
+            )
+
+    if expected_reranker is not None:
+        daemon_reranker = status.get("reranker")
+        # Accept the daemon's self-disabled state even if we expected a
+        # real model name. `run_daemon()` downgrades the reranker to
+        # "disabled" at warmup time on platforms where mlx_lm cannot
+        # load (Linux, Intel macOS). Rejecting that would break the
+        # daemon-first workflow on those supported setups. But if the
+        # caller explicitly asked for "disabled" (via empty
+        # `SEEKLINK_RERANKER_MODEL`), a running daemon with a real
+        # reranker IS a mismatch — the user asked for raw RRF scores
+        # and would silently get reranked ones.
+        is_mismatch = (
+            daemon_reranker != expected_reranker
+            and not (
+                expected_reranker != "disabled"
+                and daemon_reranker == "disabled"
+            )
+        )
+        if is_mismatch:
+            return (
+                f"daemon reranker is {daemon_reranker!r}, caller "
+                f"expects {expected_reranker!r}"
+            )
+
+    return None
+
+
+def _restart_and_retry(
+    cmd: str, args: dict[str, Any], mismatch: str
+) -> dict[str, Any]:
+    logger.info("Daemon config mismatch; restarting: %s", mismatch)
+    shutdown = _shutdown_daemon()
+    if not shutdown.get("ok"):
+        return {
+            "ok": False,
+            "error": (
+                f"{mismatch}; failed to shut down stale daemon: "
+                f"{shutdown.get('error', 'unknown error')}"
+            ),
+        }
+
+    if not _wait_for_socket_shutdown(SHUTDOWN_WAIT_SECONDS):
+        return {
+            "ok": False,
+            "error": (
+                f"{mismatch}; stale daemon did not stop within "
+                f"{SHUTDOWN_WAIT_SECONDS}s"
+            ),
+        }
+
+    return _call_once_with_spawn(cmd, args)
+
+
+def _shutdown_daemon() -> dict[str, Any]:
+    """Ask the currently running daemon to exit. Never spawns a daemon."""
+    try:
+        return _connect_and_send("shutdown", {})
+    except Exception as e:
+        return {"ok": False, "error": str(e)}
+
+
 def _call_once_with_spawn(cmd: str, args: dict[str, Any]) -> dict[str, Any]:
     """Try the daemon, spawn + retry once if unreachable."""
     try:
@@ -231,3 +273,20 @@ def _wait_for_socket(timeout: float) -> bool:
                 pass
         time.sleep(0.2)
     return False
+
+
+def _wait_for_socket_shutdown(timeout: float) -> bool:
+    """Wait until the socket is gone or no longer accepts connections."""
+    deadline = time.time() + timeout
+    while time.time() < deadline:
+        if not SOCKET_PATH.exists():
+            return True
+        try:
+            probe = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+            probe.settimeout(0.5)
+            probe.connect(str(SOCKET_PATH))
+            probe.close()
+        except (ConnectionRefusedError, OSError):
+            return True
+        time.sleep(0.1)
+    return False

seeklink/daemon.py

@@ -11,7 +11,7 @@
 length-prefixed JSON response before closing the connection.
 
 Request schema:
-    {"cmd": "search" | "status" | "index", "args": {...}}
+    {"cmd": "search" | "status" | "index" | "shutdown", "args": {...}}
 
 Response schema:
     {"ok": true,  "result": ...}   on success
@@ -27,7 +27,7 @@
 import socket
 import sys
 from pathlib import Path
-from typing import Any
+from typing import Any, Callable
 
 logger = logging.getLogger(__name__)
 
@@ -133,7 +133,16 @@ def _shutdown(signum: int, _frame: object) -> None:
             except OSError:
                 break  # socket closed during shutdown
             try:
-                _handle_connection(conn, db, embedder, reranker, vault_root)
+                _handle_connection(
+                    conn,
+                    db,
+                    embedder,
+                    reranker,
+                    vault_root,
+                    request_shutdown=lambda: shutdown_requested.__setitem__(
+                        "flag", True
+                    ),
+                )
             except Exception:
                 logger.exception("Error handling connection")
             finally:
@@ -166,6 +175,7 @@ def _handle_connection(
     embedder: Any,
     reranker: Any,
     vault_root: Path,
+    request_shutdown: Callable[[], None] | None = None,
 ) -> None:
     """Handle a single client connection: read request, execute, send response."""
     from seeklink.ingest import ingest_file, ingest_vault
@@ -183,6 +193,7 @@ def _handle_connection(
 
     cmd = req.get("cmd")
     args = req.get("args") or {}
+    should_shutdown = False
 
     try:
         if cmd == "search":
@@ -257,11 +268,17 @@ def _handle_connection(
                 stats = ingest_vault(db, vault_root, embedder)
                 response = {"ok": True, "result": stats}
 
+        elif cmd == "shutdown":
+            response = {"ok": True, "result": {"status": "shutting_down"}}
+            should_shutdown = True
+
         else:
             _send_error(conn, f"unknown command: {cmd}")
             return
 
         _send_framed(conn, json.dumps(response).encode("utf-8"))
+        if should_shutdown and request_shutdown is not None:
+            request_shutdown()
 
     except Exception as e:
         logger.exception("Command %r failed", cmd)