Zim uses one local SQLite database for full-text, popularity, and optional vector data. ZIM archives remain authoritative; deleting index.sqlite never removes archive content.
internal/indexstoreowns the unchanged SQLite schema and concreteSQLiteStore.internal/indexerowns archive scanning, extraction, embedding, bounded workers, batching, cancellation, and progress events.internal/searchowns per-archive FTS, semantic search, all-library search, and native-ZIM fallback.- CLI and web packages translate inputs and render results; they do not own search rules.
Narrow store and embedder interfaces live in the consuming indexer/search package. This keeps SQLiteStore concrete and prevents storage details from defining every consumer's contract.
Indexing stores URL, title, excerpt, body text, redirect state, and popularity rank. SQLite FTS5 ranks title and article-text matches. If a selected archive has no SQLite index, lexical search tries the original query and one UTF-8-safe uppercase-first-rune candidate through bounded native title-similarity and prefix lookups, in C-then-A order. It does not enumerate namespaces for substring matches. Other SQLite failures remain visible rather than being mistaken for a missing index, and all-library search is indexed-only.
Model2Vec generates local embeddings through ONNX Runtime. Long article text is inferred in bounded batches of eight chunks, then normalized and mean-pooled with the same output ordering as single-chunk inference. Vectors are stored as little-endian float32 SQLite blobs and ranked by exact cosine similarity. The query path filters rows by declared dimension and blob length, rejects malformed, zero-norm, and non-finite vectors, and retains only the requested top K in a worst-first heap. This bounds candidate memory without changing exact results.
Final ordering is deterministic: cosine score, then a nonzero popularity rank, title, and stable scan order. A semantic query without compatible stored vectors returns an actionable error instructing the user to rebuild the index.
The configured worker count, parallel flag, and batch size control the indexer. A single producer streams directory entries in C-then-A namespace order into the existing bounded worker channel; it does not materialize either namespace as a full slice. Positional archive read failures propagate, and every producer and worker observes the command context. Cancellation stops archive scanning, embedding, and pending batches without requiring process termination. Progress is emitted as data so the CLI may render it without coupling business rules to stdout.
The SQLite schema is versioned independently of ZIM archives. IndexStatus(ctx, path) delegates to the bulk IndexStatuses path, which looks up the latest record per archive in batches of at most 500. The (path, indexed_at DESC, id DESC) index supports that selection. CLI and web consumers receive whether each latest archive fingerprint has indexed articles and its article count. Rebuild removes the prior record before writing replacement batches; cancellation or failure then removes the incomplete replacement with a short cleanup context that outlives command cancellation, so generations never mix and restart begins cleanly.
The rankings importer streams a bounded TSV into a transaction-scoped WITHOUT ROWID temporary table in 250-row batches. Keys are trimmed and spaces become underscores; duplicate keys keep their smallest numeric rank. Set-based joins match each key against both article URLs and normalized titles. An article matched by more than one key receives the smallest rank, while matched counts distinct ranking keys that matched at least one article.
The same transaction builds matches, clears old ranks for only the target archive, and writes the replacements. Input, cancellation, and SQL failures roll back the entire import.