feat: semantic search with run-grouped embeddings and conversation-unit citations (#999)

Adds opt-in semantic/vector search over conversation content alongside the existing substring/regex/FTS modes, and gives every content-search match — in every mode, on all three backends — a conversation-unit citation.

## Semantic search

- `session search --semantic` and `--hybrid` (reciprocal-rank fusion of the vector and FTS legs), plus `--scope top|all|subordinate` to control whether subordinate evidence (sidechain runs, subagent/fork sessions) is shown; subordinate hits are rank-penalized and annotated, never silently hidden.
- Embedding documents are run-grouped: each user message is one document, and each unbroken run of assistant/tool messages between user turns is concatenated into one document (~25x fewer assistant-side documents than per-message embedding). Semantic hits anchor on the message containing the best-matching chunk's center and carry the run's ordinal span.
- `embeddings build/list/activate/retire` manage the index: through the daemon when one is running, directly (flock-guarded) otherwise. `serve` wires a debounced after-sync scheduler when `[vector]` is enabled.
- Embeddings come from any OpenAI-compatible endpoint (`[vector]` in config.toml; Ollama quickstart in the docs). An optional `input_suffix` appends a client-side terminator to every embedded text for models that need one (e.g. Qwen3-Embedding's `<|endoftext|>` under llama.cpp); it joins the generation fingerprint.
- `session messages --around N --before/--after --role` retrieves context windows around any hit on all three backends; `session search --context N` inlines them.

## Conversation-unit citations

Every match now carries `ordinal_range: [start, end]` — the conversation unit enclosing the anchor — plus `subordinate`, `relationship`, `parent_session_id`, and `is_sidechain`. `ordinal` remains the exact matched message in every mode.

- Row cardinality is mode-specific by design: lexical modes stay grep-like (one row per matching source row), semantic returns one row per embedded unit, hybrid one row per unit with exact-match anchors.
- Lexical and hybrid unit-less rows derive their unit structurally from the messages/sessions tables — deterministic, identical on SQLite/PostgreSQL/DuckDB, and independent of whether a vector index exists. A property test pins derivation to exact equivalence with the embedding reducer's unit spans.
- Derivation is post-scan and O(page): batched correlated point lookups with run sharing, ~0.4-0.65ms added per 50-hit page on the gated benchmarks.
- Surfaces: CLI renders `#start-end @anchor` with a `sub` marker; MCP `search_content` carries the same fields; the OpenAPI schema and generated client are updated.

## SQLite DSN hardening

Read-only connections were silently read-write: mattn/go-sqlite3 ignores `mode=ro` without a `file:` URI scheme. Fixed for sessions.db, vectors.db, and every foreign-app parser DB (which were also being converted to WAL on read). The fix exposed and fixed a WAL close-ordering bug in the resync swap flow. Paths are percent-escaped; read-only enforcement is pinned by tests.

## Architecture

Vectors live in a separate `vectors.db` (SQLite + sqlite-vec via go.kenn.io/kit), a mirror keyed by resync-stable doc keys with generations fingerprinted by model/dimension/unit-scheme config; a mirror schema version gates cross-version reads (rebuild-required surfaces as 501 with remediation) and resets stale mirrors on writable opens. Staleness, first-build progress, and endpoint outages surface as distinct errors across CLI/HTTP/MCP. See `docs/semantic-search.md` (usage) and `docs/semantic-search-internals.md` (unit model, doc keys, derivation invariants, fusion, error taxonomy).

## Where to look

- `internal/db/messages.go`, `internal/db/unit_range.go` — run reducer and shared unit-range derivation (the correctness core; reducer-equivalence property test).
- `internal/vector/` — mirror, generations, chunk-anchor resolution, build orchestration, encoder, search.
- `internal/db/search_content*.go` — semantic/hybrid modes, unit-granularity fusion, scope filtering, lexical citation enrichment.
- `internal/postgres/unit_range.go`, `internal/duckdb/unit_range.go` — SQL-only backend seams over the shared resolvers.
- `cmd/agentsview/embeddings.go`, `embed_scheduler.go` — CLI group, serve wiring, scheduler.
- `internal/vector/encoder.go`, `internal/vector/build.go` — build throughput: requests use `encoding_format: "base64"` (~4x smaller responses, with transparent float fallback for servers that reject or ignore the field), and a `[vector.embeddings] concurrency` key (default 4) embeds documents in parallel via kit's new `FillOptions.Concurrency` (kenn-io/kit#27; go.mod pins that PR's commit and should move to a tagged kit release before merge). Saves stay serialized, preserving the single-writer model. Sequential float-JSON requests left builds round-trip-bound against remote endpoints; measured on a slow WireGuard link, these two changes took a full-archive build from ~46 to ~700 chunks/min.
- Frontend diff is regenerated API-client output only; the web UI remains FTS-only in this release.

## Limitations

- Semantic/hybrid search is SQLite-archive only; `pg serve`/DuckDB validate and report it unavailable (citations work on all three).
- Metadata filters post-filter the vector leg (over-fetch mitigates recall loss); narrow scopes can under-fill a page past the batched FTS-leg cap.
- Citation derivation adds ~19-21% to content-search page latency (sub-millisecond absolute); levers (covering index, statement cache) documented but not pulled.
- Draft: kept open for real-world exercise before merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)



Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>

Author: wesm

AGENTS.md

@@ -120,6 +120,8 @@ CLI (agentsview) -> Config -> DB (SQLite/FTS5)
 - `internal/parser/` - Per-agent session file parsers and content extraction.
 - `internal/server/` - HTTP handlers, SSE, middleware, search, and export.
 - `internal/sync/` - Sync engine, file watcher, discovery, and hashing.
+- `internal/vector/` - Semantic search: embeddings encoder, `vectors.db`
+  mirror/index, build orchestration, and semantic/hybrid search.
 - `internal/timeutil/` - Time parsing utilities.
 - `internal/web/` - Embedded frontend copied from `frontend/dist/` at build
   time.
@@ -128,30 +130,33 @@ CLI (agentsview) -> Config -> DB (SQLite/FTS5)
 
 ## Key Files
 
-| Path                             | Purpose                                       |
-| -------------------------------- | --------------------------------------------- |
-| `cmd/agentsview/main.go`         | CLI entry point, server startup, file watcher |
-| `cmd/agentsview/pg.go`           | `pg` command group: push, status, serve       |
-| `internal/server/server.go`      | HTTP router and handler setup                 |
-| `internal/server/sessions.go`    | Session list/detail API handlers              |
-| `internal/server/search.go`      | Full-text search API                          |
-| `internal/server/events.go`      | SSE event streaming                           |
-| `internal/db/db.go`              | Database open, migrations, schema             |
-| `internal/db/sessions.go`        | Session CRUD queries                          |
-| `internal/db/search.go`          | FTS5 search queries                           |
-| `internal/sync/engine.go`        | Sync orchestration                            |
-| `internal/parser/types.go`       | Agent registry with one `AgentDef` per agent  |
-| `internal/parser/*.go`           | Per-agent session parsers                     |
-| `internal/postgres/connect.go`   | Connection setup, SSL checks, DSN helpers     |
-| `internal/postgres/schema.go`    | PG DDL and schema management                  |
-| `internal/postgres/push.go`      | Push logic and fingerprinting                 |
-| `internal/postgres/sync.go`      | Push sync lifecycle                           |
-| `internal/postgres/store.go`     | PostgreSQL read-only store                    |
-| `internal/postgres/sessions.go`  | PG session queries on the read side           |
-| `internal/postgres/messages.go`  | PG message queries and ILIKE search           |
-| `internal/postgres/analytics.go` | PG analytics queries                          |
-| `internal/postgres/time.go`      | Timestamp conversion helpers                  |
-| `internal/config/config.go`      | Config loading and flag registration          |
+| Path                             | Purpose                                                   |
+| -------------------------------- | --------------------------------------------------------- |
+| `cmd/agentsview/main.go`         | CLI entry point, server startup, file watcher             |
+| `cmd/agentsview/pg.go`           | `pg` command group: push, status, serve                   |
+| `cmd/agentsview/embeddings.go`   | `embeddings` command group: build, list, activate, retire |
+| `internal/server/server.go`      | HTTP router and handler setup                             |
+| `internal/server/sessions.go`    | Session list/detail API handlers                          |
+| `internal/server/search.go`      | Full-text search API                                      |
+| `internal/server/events.go`      | SSE event streaming                                       |
+| `internal/db/db.go`              | Database open, migrations, schema                         |
+| `internal/db/sessions.go`        | Session CRUD queries                                      |
+| `internal/db/search.go`          | FTS5 search queries                                       |
+| `internal/vector/index.go`       | `vectors.db` schema, generations, staleness gate          |
+| `internal/vector/search.go`      | Semantic + hybrid search, RRF merge                       |
+| `internal/sync/engine.go`        | Sync orchestration                                        |
+| `internal/parser/types.go`       | Agent registry with one `AgentDef` per agent              |
+| `internal/parser/*.go`           | Per-agent session parsers                                 |
+| `internal/postgres/connect.go`   | Connection setup, SSL checks, DSN helpers                 |
+| `internal/postgres/schema.go`    | PG DDL and schema management                              |
+| `internal/postgres/push.go`      | Push logic and fingerprinting                             |
+| `internal/postgres/sync.go`      | Push sync lifecycle                                       |
+| `internal/postgres/store.go`     | PostgreSQL read-only store                                |
+| `internal/postgres/sessions.go`  | PG session queries on the read side                       |
+| `internal/postgres/messages.go`  | PG message queries and ILIKE search                       |
+| `internal/postgres/analytics.go` | PG analytics queries                                      |
+| `internal/postgres/time.go`      | Timestamp conversion helpers                              |
+| `internal/config/config.go`      | Config loading and flag registration                      |
 
 ## Development
 

README.md

@@ -277,6 +277,11 @@ agentsview stats --include-git-outcomes
 | ![Search](https://agentsview.io/assets/generated/screenshots/search-results.png) | ![Heatmap](https://agentsview.io/assets/generated/screenshots/heatmap.png) |
 
 - **Full-text search** across all message content (FTS5)
+- **Semantic search** (opt-in) -- index session content with any
+  OpenAI-compatible embeddings endpoint and search by meaning with
+  `agentsview session search --semantic` or `--hybrid`; every content-search
+  match cites the conversation unit it came from
+  ([docs](https://agentsview.io/semantic-search/))
 - **Token usage and cost dashboard** -- per-session and per-model cost
   breakdowns, daily spend charts, all in the web UI
 - **Analytics dashboard** -- activity heatmaps, tool usage, velocity metrics,

cmd/agentsview/cli.go

@@ -115,12 +115,14 @@ func newRootCommand() *cobra.Command {
 	root.AddCommand(newActivityCommand())
 	root.AddCommand(newPGCommand())
 	root.AddCommand(newDuckDBCommand())
+	root.AddCommand(newEmbeddingsCommand())
 	root.AddCommand(newSessionCommand())
 	root.AddCommand(newMCPCommand())
 	root.AddCommand(newStatsCommand())
 	root.AddCommand(newParseDiffCommand())
 	root.AddCommand(newClassifierCommand())
 	root.AddCommand(newSecretsCommand())
+	root.AddCommand(newSkillsCommand())
 	root.AddCommand(newDoctorCommand())
 	root.AddCommand(newVersionCommand())
 	root.AddCommand(newOpenAPICommand())

cmd/agentsview/doctor.go

@@ -202,12 +202,17 @@ func inspectDoctorDB(path string) doctorDBInspection {
 	return insp
 }
 
+// doctorReadOnlyDSN builds a read-only sqlite3 DSN. The file: scheme is
+// required for mattn/go-sqlite3 to honor mode=ro (a bare path silently opens
+// read-write), and the path is percent-encoded so `%`, `?`, or `#` in a real
+// path cannot be misparsed as URI syntax.
 func doctorReadOnlyDSN(path string) string {
 	params := url.Values{}
 	params.Set("mode", "ro")
 	params.Set("_busy_timeout", "5000")
 	params.Set("_foreign_keys", "ON")
-	return path + "?" + params.Encode()
+	escaped := (&url.URL{Path: path}).EscapedPath()
+	return "file:" + escaped + "?" + params.Encode()
 }
 
 func listDoctorResyncTempFiles(dbPath string) []string {