Query- & search-derived datasets

docker-compose.yml

@@ -82,6 +82,12 @@ services:
       MINIO_SECURE: ${MINIO_SECURE:-false}
       MINIO_BUCKET: ${MINIO_BUCKET:-openrunner}
       REDIS_URL: ${REDIS_URL:-redis://redis:6379/0}
+      # Speech-catalog semantic search: TEI bge-m3 embed server on the host
+      # (docker-published, firewalled to internal only). host.docker.internal ->
+      # host gateway -> tei :8081. Used to embed the NL query for /catalog/semantic.
+      # Literal (not ${...}) so it wins over a stale value in env_file/.env.
+      CATALOG_EMBED_URL: "http://host.docker.internal:8081/v1/embeddings"
+      CATALOG_EMBED_MODEL: ${CATALOG_EMBED_MODEL:-BAAI/bge-m3}
       # SECRET_KEY / JWT_*_SECRET: validators in app/core/config.py refuse
       # weak/default values (>=32 chars, no `change-me*` placeholders) — the
       # entrypoint auto-generates and persists strong values to
@@ -125,6 +131,11 @@ services:
       - ./secrets:/secrets:ro
     ports:
       - "127.0.0.1:8000:8000"
+    extra_hosts:
+      # host.docker.internal -> host gateway, so the API can reach the TEI embed
+      # server published on the host (:8081) for /catalog/semantic. Not automatic
+      # on Linux compose.
+      - "host.docker.internal:host-gateway"
     depends_on:
       postgres:
         condition: service_healthy

docs/datasets/derived-datasets.md

@@ -0,0 +1,185 @@
+# Derived datasets (query- & search-derived)
+
+A **derived dataset** is a saved selection over a parent dataset's LanceDB table.
+You run a **word**, **full-text**, or **semantic** query and OpenRunner records the
+rows that matched — as a new dataset, or appended into an existing one.
+
+It is built for scale and correctness:
+
+- **No replication.** Only the matching row keys (`debug_id`) are stored, never a
+  copy of the rows. A derived dataset over 5.8M clips is a few thousand IDs.
+- **Inherits every parent field dynamically.** At read time the parent rows are
+  fetched live, so any column the parent gains later shows up automatically.
+- **Extendable.** Add your own columns on top (labels, notes, review status) —
+  stored only in the derived dataset, the parent is never touched.
+- **Auto de-duplication.** The selection is keyed on `debug_id`; re-running a
+  query or appending a second query never double-counts a row.
+- **Auto provenance.** Every derived row carries `derived_from_request` (the query
+  that pulled it in) and `derived_from_request_date` (when).
+
+A canonical example: derive a **free-user medical** dataset from the speech
+catalog by semantically searching *"a medical consultation between a doctor and a
+patient about symptoms, diagnosis and treatment"*.
+
+---
+
+## How it works
+
+```
+parent dataset (LanceDB table, e.g. transcriptions — 5.8M rows)
+        │   word / fts / semantic query
+        ▼
+   matching debug_ids ──►  members table  (debug_id + provenance + your extra cols)
+        │                        │  merge_insert(debug_id) → dedup
+        ▼                        ▼
+   read a page ── join by debug_id ──►  live parent fields  +  your extra columns
+```
+
+The members table lives in the **same** LanceDB as the parent (no new storage
+bucket). Only keys + your extra columns are written. Reads page the member keys
+and point-look-up exactly that page from the parent (BTREE on `debug_id`), so a
+derived dataset over millions of rows still lists instantly.
+
+---
+
+## Create a derived dataset
+
+=== "UI"
+
+    Open the parent dataset → **Indexes** tab → **🧬 Derive a dataset from this
+    query**. Pick a mode, type the query, name the dataset, **Create**.
+
+=== "API"
+
+    ```bash
+    curl -X POST /api/v1/datasets/{parent_id}/derive \
+      -H 'Authorization: Bearer $TOKEN' -H 'Content-Type: application/json' \
+      -d '{
+        "name": "free-user-medical",
+        "mode": "semantic",
+        "q": "a medical consultation between a doctor and a patient about symptoms, diagnosis and treatment",
+        "limit": 2000
+      }'
+    ```
+
+    Returns the new `DatasetRead`. `num_examples` is the de-duplicated member count.
+
+### Query modes
+
+| Mode | Backend | Use it for |
+|------|---------|-----------|
+| `word` | inverted `lexical_map` (O(1)-ish) | one exact term, cheapest |
+| `fts` | BM25 full-text | keyword queries, ranked |
+| `exact` | phrase full-text | adjacent-word phrase |
+| `semantic` | cosine ANN over embeddings | meaning / cross-lingual (needs a vector index) |
+
+`lang` (optional) restricts to a language; `limit` caps the matches (default 2000).
+
+---
+
+## Append another query (same or new dataset)
+
+Pass `target_dataset_id` to add a second query's matches **into an existing
+derived dataset**. De-dup is automatic — rows already present are left as-is
+(keeping their original `derived_from_request`).
+
+```bash
+curl -X POST /api/v1/datasets/{parent_id}/derive \
+  -H 'Authorization: Bearer $TOKEN' -H 'Content-Type: application/json' \
+  -d '{ "target_dataset_id": "<derived_id>", "mode": "word", "q": "prescription", "limit": 5000 }'
+```
+
+The derived dataset's metadata keeps a `queries[]` log (mode, query, matched,
+added, timestamp) for full provenance.
+
+---
+
+## Read derived rows
+
+```bash
+curl '/api/v1/datasets/{derived_id}/derived-rows?offset=0&limit=100'
+```
+
+```jsonc
+{
+  "rows": [
+    {
+      "debug_id": "G-f0ab…",
+      "language": "en",
+      "full_transcript": "…doctor: what symptoms…",   // inherited from parent, live
+      "duration_seconds": 12.4,                          // inherited
+      "derived_from_request": "semantic:a medical consultation…",  // auto
+      "derived_from_request_date": "2026-07-22T06:37:19",          // auto
+      "my_label": "confirmed"                            // your extra column
+    }
+  ],
+  "total": 1873,
+  "extra_columns": ["my_label"],
+  "queries": [ { "mode": "semantic", "q": "…", "matched": 1873, "added": 1873, "at": "…" } ]
+}
+```
+
+Every parent column is present without being copied; add a column to the parent
+later and it simply appears here on the next read.
+
+---
+
+## Add & edit your own columns
+
+Extra columns live only in the derived dataset.
+
+```bash
+# add a column (all rows default to the given value / NULL)
+curl -X POST /api/v1/datasets/{derived_id}/columns \
+  -H 'Content-Type: application/json' -d '{ "name": "my_label", "default": "unlabeled" }'
+
+# set one row's value (creates the column if new)
+curl -X PATCH /api/v1/datasets/{derived_id}/cell \
+  -H 'Content-Type: application/json' \
+  -d '{ "debug_id": "G-f0ab…", "column": "my_label", "value": "confirmed" }'
+```
+
+---
+
+## API reference
+
+| Method & path | Purpose |
+|---------------|---------|
+| `POST /datasets/{parent_id}/derive` | Create (with `name`) or append (with `target_dataset_id`) from a query |
+| `GET /datasets/{derived_id}/derived-rows` | Page of inherited parent fields + auto + extra columns |
+| `POST /datasets/{derived_id}/columns` | Add an extra column |
+| `PATCH /datasets/{derived_id}/cell` | Set one row's extra-column value |
+
+Lineage: a derived dataset records a `derived-from` edge to its parent, so it
+appears in the dataset lineage graph.
+
+## Enabling semantic search (optimize embedding search)
+
+Semantic mode (and `GET /datasets/{id}/semantic`) needs an **embedding column +
+ANN index** on the table. Any indexed dataset can be optimized for it:
+
+```bash
+# embed rows missing a vector + build/rebuild the IVF index (background job)
+curl -X POST /api/v1/datasets/{id}/optimize-embedding-search \
+  -H 'Content-Type: application/json' -d '{ "text_column": "full_transcript", "max_rows": 200000 }'
+# poll dataset_metadata.lance.semantic.status -> running | ready | error
+
+# then search by meaning
+curl '/api/v1/datasets/{id}/semantic?q=a doctor discussing symptoms&limit=20'
+```
+
+Or in the UI: dataset → **Indexes** tab → **⚡ Optimize embedding search**.
+
+It embeds only rows whose text column is non-empty, and — critically — uses a
+**stage-then-flush** write (embed to a staging table via cheap appends, then a
+single `merge_insert`). Per-chunk `merge_insert` on a wide table rewrites the
+whole embedding column each call (minutes per chunk), so it is never used.
+
+## Notes & limits
+
+- The parent must be **indexed** (have a LanceDB table). Semantic mode also needs
+  a vector index on the embedding column (run *optimize embedding search*) and a
+  reachable embed server.
+- Provenance columns `derived_from_request` / `derived_from_request_date` are
+  reserved and written automatically; they are not counted as your extra columns.
+- Extra columns are stored as strings.

mkdocs.yml

@@ -65,7 +65,9 @@ nav:
     - Quick Start: getting-started/quickstart.md
     - Troubleshooting (First Run): troubleshooting-first-run.md
   - Skills: skills/index.md
-  - Datasets: datasets/index.md
+  - Datasets:
+    - Overview: datasets/index.md
+    - Derived datasets: datasets/derived-datasets.md
   - SDK Reference:
     - init(): sdk/init.md
     - Logging: sdk/logging.md