fix: complete BigQuery cutover for SEC search and query interface (#130)

[C00ldudeNoonan]

Summary

The warehouse migration was incomplete: registered code and docs still assumed DuckDB/MotherDuck while BigQueryWarehouseResource returns a BigQuery client. On BigQuery the SEC search assets failed outright (PRAGMA, list_cosine_similarity, match_bm25, ?::FLOAT[768], conn.execute(...).pl()), NL-to-SQL emitted/validated DuckDB SQL, and a clean checkout advertised the retired MotherDuck interface. This completes the cutover.

SEC search — BigQuery-native rewrite

• semantic_search.py — vector_search / keyword_search / hybrid_search now take the BigQueryWarehouseResource and emit Standard SQL:
  • Vector: 1 - ML.DISTANCE(embedding, @query_embedding, 'COSINE') over the ARRAY<FLOAT64> embeddings already written by sec_filing_search_index (brute-force scan — exact, no VECTOR INDEX min-row/consistency constraints).
  • Keyword: SEARCH(content_text, @query_text) to match, ranked by distinct query-term overlap via CONTAINS_SUBSTR (BigQuery exposes no BM25 score), recency tie-break.
• fts.py — drops INSTALL/LOAD fts and PRAGMA create/drop_fts_index; populates the denormalized FTS table and ensures a best-effort BigQuery SEARCH INDEX (SEARCH() still works via full scan if the index can't be created).
• tables.py — adds ensure_sec_filing_fts_content_table (BigQuery DDL).

NL-to-SQL

• nl_to_sql_module.py — SQLValidator parses with dialect="bigquery"; the DSPy output field now asks the model for BigQuery Standard SQL (not "DuckDB-compatible").

Cleanup

• Replaced stale kinds={"duckdb"} → {"bigquery"} across 40 asset modules (72 tags).
• Removed the retired MotherDuck MCP server from .mcp.json; the dbt-index MCP's warehouse tool is the clean-checkout BigQuery query path.
• README, .env.example, .env.dagster.example, macro_agents/.env.example: describe BigQuery instead of MotherDuck; corrected stale BIGQUERY_PROJECT_ID → BIGQUERY_PROJECT (the var the resource actually reads).

Design validation (no live BQ in CI)

Every generated search statement is run through the same prepare_query_parameters contract the resource uses at runtime (sqlglot BigQuery parse + named-param matching), so the SQL is proven to be valid BigQuery and DML-safe at the unit level.

Tests

• test_sec_search_queries.py — builders emit valid BigQuery (contract-parsed), hybrid RRF fuses vector + keyword hits, and a guard asserts no DuckDB primitives remain in the SEC search modules.
• test_nl_to_sql_bigquery.py — validator accepts BigQuery-only syntax (backtick-quoted table, which DuckDB rejects) and the prompt targets BigQuery.
• Removed test_sec_search_functional.py (tested the retired in-memory DuckDB backend; imported a now-deleted helper).
• 479 tests collect; ruff check + format clean.

Acceptance criteria

• No registered production asset sends DuckDB-only SQL to BigQuery
• SEC keyword + semantic search have a documented BigQuery backend
• NL-to-SQL emits and validates BigQuery SQL
• README / env examples / MCP config describe the actual architecture
• Clean checkout can initialize the documented query interface (dbt-index MCP)
• Integration coverage against live BigQuery for representative keyword/vector/hybrid searches — needs a warehouse target + embeddings; cannot run in CI without creds. Flagged for a live validation run.

Closes #130

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.