Skip to content

Commit 75df6f3

Browse files
wesmwesm
committed
docs: fold semantic-search design notes into internals page, drop working specs
1 parent 082da50 commit 75df6f3

5 files changed

Lines changed: 176 additions & 1811 deletions

File tree

‎docs/semantic-search-internals.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
---
2+
title: Semantic Search Internals
3+
description: Architecture and invariants behind the vector index — storage, generations, build pipeline, concurrency, and search path
4+
---
5+
6+
This page documents the internal design of [Semantic Search](/semantic-search/)
7+
for maintainers extending or debugging the vector index. It assumes the
8+
user-facing behavior described there and does not repeat configuration or CLI
9+
usage.
10+
11+
## Storage layout
12+
13+
`vectors.db` is a separate SQLite database beside the main archive
14+
(`sessions.db`), not a set of tables inside it. Two things follow from that:
15+
16+
- **It survives a parser-change resync.** A resync rebuilds and atomically swaps
17+
`sessions.db`; `vectors.db` is untouched by the swap. The next mirror
18+
refresh re-derives identities against the new archive, so unchanged messages
19+
keep their vectors and only genuinely changed content re-embeds.
20+
- **It's self-contained.** The mirror copies message content into `vectors.db`
21+
rather than joining back to the archive, so the vector store never needs the
22+
archive open to serve a query, and `vectors.db` can be deleted and rebuilt
23+
(`embeddings build --full-rebuild`) without touching `sessions.db`.
24+
25+
Tables inside the archive DB were rejected: that would tie vector writes to the
26+
archive's write path and lock, and complicate the resync-swap story with
27+
special-casing during the swap instead of a plain re-scan afterward.
28+
29+
### The `vector_messages` mirror table
30+
31+
One row per embeddable message (`role IN ('user','assistant')`, non-system,
32+
non-system-prefixed — the same universe FTS uses before `--exclude-system`).
33+
Columns: `doc_key` (primary key), `session_id`, `source_uuid`, `ordinal`,
34+
`content` (copied text), `content_hash` (sha256 of content, kit's revision
35+
column), `embed_gen`.
36+
37+
### `doc_key` scheme
38+
39+
`internal/vector/mirror.go` builds `doc_key` as:
40+
41+
- `u:<session_id>:<source_uuid>` when the message has a `source_uuid` (with a
42+
`#<n>` occurrence suffix when more than one message in a session shares the
43+
same `source_uuid` — `n` is a 1-based counter assigned in
44+
`(session_id, ordinal)` scan order, so it's deterministic across resyncs)
45+
- `o:<session_id>:<ordinal>` otherwise (legacy parsed data with no per-message
46+
UUID)
47+
48+
`session_id` and `source_uuid` are percent-escaped before joining — a custom
49+
escape that only encodes `%`, `:`, and `#` as `%XX`, not a general URL-encoder —
50+
so a literal colon, hash, or percent sign inside either component can't be
51+
mistaken for one of the key's own delimiters, and an occurrence-suffix-shaped
52+
`source_uuid` can't collide with a real occurrence suffix.
53+
54+
UUID-keyed rows survive ordinal renumbering (e.g. from a resync) as a cheap
55+
`ordinal`/`content_hash` update with no re-embed. Ordinal-keyed rows become a
56+
new document whenever their ordinal shifts, and re-embed — an accepted cost that
57+
only affects data parsed before per-message UUIDs existed.
58+
59+
## Generations and fingerprints
60+
61+
The vector index moves through kit's generation lifecycle: **building → active →
62+
retired**. A generation's fingerprint is derived from `model` + `dimension` +
63+
`max_input_chars` — changing any of them, including the chunking cap, produces a
64+
different fingerprint.
65+
66+
- `embeddings build` (incremental): mirror refresh, then fill whatever the
67+
active generation is missing.
68+
- `embeddings build --full-rebuild`: if the target fingerprint differs from the
69+
active generation's, cuts a new generation and fills it fully, activating on
70+
clean completion; if the fingerprint is unchanged (e.g. rebuilding after a
71+
content-only change), it resets and refills the *existing* active generation
72+
in place — clearing its vectors, chunks, and stamps but keeping the
73+
generation row — rather than cutting a new one.
74+
- The staleness gate checked at query time is exactly this: the active
75+
generation's stored fingerprint no longer matches the fingerprint computed
76+
from the current `[vector.embeddings]` config.
77+
78+
## Build pipeline
79+
80+
### Mirror refresh (scan)
81+
82+
Before a fill, the mirror is reconciled against the archive's embeddable
83+
universe: new identities are inserted, `ordinal`/`content_hash` updated on
84+
existing ones, and identities no longer present removed.
85+
86+
Removal is two-phase, and the ordering matters: deleting only the
87+
`vector_messages` row would leave its vectors occupying KNN slots (the query
88+
path filters them from hits, but the slots themselves are never reclaimed).
89+
Removal always deletes the document's vectors first, then the mirror row — and,
90+
within a single scan, a row that's merely displaced (for example a duplicate
91+
`source_uuid` shifting occurrence) is parked at a negative sentinel ordinal
92+
instead of being deleted outright, so a same-scan reinsert under the same
93+
`doc_key` survives via upsert and keeps its `embed_gen` rather than
94+
re-embedding.
95+
96+
### Fill and skip-and-stamp
97+
98+
Fill embeds every pending document (content changed, or never embedded, for the
99+
active generation). A document whose encode call fails with a permanent error —
100+
any 4xx except 429, e.g. token-window overflow or a content-policy rejection —
101+
is not retried in that fill or the next one: it's stamped for the generation
102+
with no vectors at its current `content_hash`, which marks it non-pending. It's
103+
logged (doc key plus the underlying error) and counted in the build summary's
104+
skipped count, but there is no separate poison list or periodic retry — the only
105+
way it embeds again is if the message's content itself changes later (a new
106+
`content_hash`, so a new pending row). All other failures (5xx, network errors,
107+
timeouts, 429) abort the fill and are retried on the next scheduled build.
108+
109+
### Scope (`include_automated`)
110+
111+
Whether automated sessions are in the embeddable universe is stored in
112+
`vector_meta` (`scope_include_automated`). Changing it — in config or via the
113+
one-off `--include-automated` flag — forces a full mirror *reconciliation*, not
114+
a re-embed: it inserts or removes rows to match the new scope, but documents
115+
that stay in scope and are unchanged keep their existing stamps.
116+
117+
## Concurrency and locking
118+
119+
`vectors.db` follows the archive's single-writer model, with its own lock file
120+
(`vectors.write.lock`) separate from the archive's `db.write.lock` so fills
121+
never contend with archive writes:
122+
123+
- With a writable daemon running, `embeddings build`/`activate`/`retire` proxy
124+
to it over HTTP; the daemon holds `vectors.write.lock` for its lifetime and
125+
serializes all builds through one in-process `Manager`.
126+
- Without a daemon, the CLI takes the same `vectors.write.lock` itself and runs
127+
the build in-process.
128+
129+
The after-sync scheduler debounces sync-completion signals about 30s before
130+
triggering a build, and never blocks sync on embedding. A build already in
131+
progress causes a new trigger to be dropped, not queued — the pending state is
132+
left set so the next debounce or backstop tick picks it up. A periodic backstop
133+
(`backstop_interval`, default 24h) runs a full reconciliation independent of
134+
sync activity, to catch stragglers from crashes or transient encode failures; if
135+
a backstop tick lands while a build is already running, it's remembered so the
136+
*next* build, not the next 24h tick, carries the full-reconciliation flag.
137+
138+
Generation activation always happens under the single writer. Search opens
139+
`vectors.db` read-only from any process, with no locking.
140+
141+
## Search path
142+
143+
- **Active generation only.** Search never falls back to a building or retired
144+
generation — if only a building generation exists, it hard-errors with a
145+
progress percentage rather than silently querying partial data.
146+
- **Hybrid is RRF over two legs, filtered before merge.** `--hybrid` runs the
147+
vector leg and the FTS leg over the same corpus (embeddable messages only),
148+
applies session and metadata filtering to each leg independently, and only
149+
then fuses them with reciprocal rank fusion — so RRF never spends its rank
150+
budget on candidates a filter would have dropped anyway.
151+
- **Metadata filters post-filter the vector leg, with over-fetch.** Vector KNN
152+
doesn't know about `--project`/`--agent`/`--date*`, so the vector leg
153+
over-fetches `max(limit × 4, 200)` candidates, then filters and truncates to
154+
the requested limit. At small corpora or narrow filters this can return
155+
fewer than `--limit` results even though more exist — a known v1 tradeoff
156+
(see [Limitations](/semantic-search/#limitations)).
157+
158+
## Error taxonomy
159+
160+
Two sentinel errors carry every semantic/hybrid failure across CLI, HTTP, and
161+
MCP:
162+
163+
| Sentinel | Meaning | HTTP |
164+
| ------------------------ | ----------------------------------------------------------------------------------------------------- | ---- |
165+
| `ErrSemanticUnavailable` | Not enabled/configured, index never finished a build, still building, or stale (fingerprint mismatch) | 501 |
166+
| `ErrSemanticTransient` | Embeddings endpoint unreachable or timed out at query time — retryable | 503 |
167+
168+
The distinction matters for callers: 501 means the feature will not work until
169+
something is configured or built; 503 means it should work and is worth
170+
retrying. CLI and MCP surface the same cause-specific remediation text described
171+
in the [user-facing error taxonomy](/semantic-search/#error-taxonomy).

‎docs/semantic-search.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ substring/regex/FTS5 content search. This is an opt-in feature backed by an
99
OpenAI-compatible embeddings endpoint — a local [Ollama](https://ollama.com)
1010
model or a hosted API.
1111

12+
For the architecture behind this page — storage layout, generations,
13+
concurrency, and the search path — see
14+
[Semantic Search Internals](/semantic-search-internals/).
15+
1216
!!! note "SQLite only"
1317

1418
Semantic and hybrid search require the local SQLite archive.

0 commit comments

Comments
 (0)