Query- & search-derived datasets#602
Merged
Merged
Conversation
Reference datasets (e.g. the speech catalog) index audio that physically
lives in a different bucket than the dataset's own connection, stored per
row as an s3://bucket/key uri. The generic /files/{path}/download route
presigns against the dataset's own connection and requires a DatasetFile
manifest, so these s3:// audio refs 404.
Add GET /datasets/{id}/audio?uri=s3://... which matches the uri's bucket to
one of the org's storage connections and returns a presigned GET url —
played with server-held credentials, never a raw/public s3 url. Scoped: the
bucket must be an org connection and the key must sit under the dataset's
declared audio_prefix (dataset_metadata).
Frontend: detect s3:// audio values (any container extension) and route them
to the new endpoint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…LanceDB uri
The dataset Indexes/Preview/Search/facets tabs resolve the LanceDB table at the
hardcoded {storage_prefix}/_lancedb location. A reference dataset whose data is a
shared table (the Speech Catalog `transcriptions` at s3://dataset-database/lancedb)
therefore couldn't be searched/faceted from the dataset UI — the path didn't match.
Let dataset_metadata["lance"] carry an explicit `uri`; search_lance_sync and
facet_values_sync (and the live row-fetch) use it when present, else fall back to
{storage_prefix}/_lancedb. So a reference dataset can be backed by an existing
catalog table: Indexes search, Preview live rows, and Analytics facets all hit the
real transcriptions table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The live Preview fetch for a catalog-backed reference dataset selected every column, including the 1024-float `embedding` vector and `raw_input_json` blob. On the 5.8M-row transcriptions table that made the Preview tab crawl. Project away embedding/raw_input_json/vector/_text before fetching the head rows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… S3) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Returning the 1024-float embedding + raw json blobs per hit made catalog FTS search crawl. Project them out in python (an explicit .select on the 160-col table is itself slow). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…/mixed mode
Search returns light hits (debug_id + snippet + a few fields), not full rows —
the client fetches a full row on demand via /datasets/{id}/row?debug_id=
(BTREE-indexed single-row lookup). Adds mode=exact (phrase) | mixed (ranked
tokens). Removes the 1024-float embedding / json blobs from search payloads.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GET /datasets/{id}/word-lookup?term=X -> {df, languages, debug_ids} from the
lexical_map table (word -> transcripts). A pure map lookup (one row/term),
cheaper than FTS and unaffected by concurrent enrichment writes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Checkbox reflecting lance.word_lookup + a per-word lookup box hitting
/datasets/{id}/word-lookup (df + affected transcript ids from lexical_map).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ndex-align # Conflicts: # src/api/app/api/v1/datasets.py # src/api/app/services/dataset_index.py
…, inherit + extend)
Derive a new dataset from a word / full-text / semantic query over a parent
LanceDB table. Stores only matching debug_ids (no row copy), inherits every
parent field dynamically at read time, and auto-de-duplicates on debug_id.
- POST /datasets/{parent}/derive — create (name) or append (target_dataset_id)
- GET /datasets/{id}/derived-rows — inherited parent fields + auto + extra cols
- POST /datasets/{id}/columns, PATCH /datasets/{id}/cell — add/edit own columns
- Each derived row auto-tagged derived_from_request + derived_from_request_date
- Members table lives in the same LanceDB; reads page keys + point-lookup parent
- UI: "Derive a dataset from this query" panel in the Indexes tab
- Docs: docs/datasets/derived-datasets.md
Also point catalog_embed_url at the host TEI embed (:8081) so /catalog/semantic
and semantic-mode derive can embed the query (compose CATALOG_EMBED_URL +
host.docker.internal extra_hosts).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generalizes the speech-catalog embed backfill into a reusable dataset action:
embed rows missing a vector (stage-then-flush, never per-chunk merge_insert) and
build/rebuild an IVF ANN index for sub-second semantic search — on any indexed
dataset. Also powers semantic-mode dataset derivation.
- POST /datasets/{id}/optimize-embedding-search — background job; status in
dataset_metadata.lance.semantic (running/ready/error)
- GET /datasets/{id}/semantic?q= — cosine ANN search over the table
- service dataset_embed_optimize.py: stage-then-flush embed + IVF build; skips
empty-text rows; single merge (O(column) not O(chunks))
- UI: "⚡ Optimize embedding search" card (optimize + semantic box) in Indexes tab
- docs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # src/api/app/api/v1/datasets.py # src/web/src/pages/DatasetDetail.tsx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds derived datasets: run a word / full-text / semantic query over a parent dataset's LanceDB table and save the matches as a new dataset — or append into an existing one.
Built for scale + correctness:
debug_ids, not a copy of the rows.debug_id; re-running or appending a query never double-counts.derived_from_request+derived_from_request_date.Example: derive a free-user medical dataset from the 5.8M-clip speech catalog with one semantic query.
Endpoints
POST /datasets/{parent}/derive— create (name) or append (target_dataset_id)GET /datasets/{id}/derived-rows— inherited fields + auto + extra columnsPOST /datasets/{id}/columns·PATCH /datasets/{id}/cell— add/edit own columnsUI
"🧬 Derive a dataset from this query" panel in the dataset Indexes tab.
Tested
Validated end-to-end against the live
transcriptionstable (5.8M rows): word/fts/semantic member selection, dedup (re-add adds 0), inheritedfull_transcript/language, auto provenance columns, add-column + set-cell.Docs
docs/datasets/derived-datasets.md(+ nav).Also fixes
catalog_embed_urlto the host TEI embed (:8081) so semantic search / derive can embed the query.🤖 Generated with Claude Code