Assay is an autonomous data analyst. The engineering goal is not "generate SQL" — it is to make an agent's answers trustworthy: recoverable when wrong, verifiable before shown, and provable after. This document describes how the system is built to deliver that.
Every analysis is an append-only log of typed events (run.started, plan.proposed,
sql.generated, query.failed, repair.attempted, verification.checked, answer.rendered, …).
Derived state — status, cost, confidence, the answer — is a pure fold over that log, never
stored as the source of truth.
This single decision buys almost everything else:
- Crash-safe, resumable runs — state is rebuilt by replaying events.
- Deterministic replay — re-folding a recorded log reproduces a run exactly (property-tested: any log folds identically twice, and is order-independent by sequence number).
- Provenance — the log is the record of how an answer was produced.
- Evaluation — an eval case is just a recorded run, graded.
The event taxonomy lives in @assay/contracts as a Zod discriminated union; the fold and replay
live in @assay/kernel. Events are immutable and additive — new capabilities add new event
types, never mutate old ones.
Understand → Plan → Generate → [ Safety-gate → Execute → on failure: classify → Repair → retry ]
→ Verify → Explain → (follow-up)
- Understand / Plan (
@assay/agent,@assay/semantic) — the question is resolved against a semantic layer: an embedding index retrieves the handful of relevant tables from a wide schema, expands them along the foreign-key join graph, and surfaces curated/learned metric definitions. If a term is genuinely ambiguous, the planner asks one precise question rather than guessing. - Generate (
@assay/agent) — the model emits a single dialect-correctSELECT. - Safety gate (
@assay/forge) — the query is parsed to an AST and rejected unless it is a single read-onlySELECT(no DDL/DML, no multiple statements, no semicolon injection). It fails closed: an unparseable query is refused, not trusted. - Execute (
@assay/warehouse) — the query runs read-only, with a row cap, a statement timeout, and anEXPLAIN-based cost governor that rejects runaway queries before they run. Results are cached by normalized SQL. - Repair — a failed query is classified (unknown column, type mismatch, ambiguous reference, syntax, …) and regenerated with the real database error fed back, within a bounded budget.
- Verify (
@assay/verifier) — independent checks run against the result: reconciliation, join fan-out detection, row-count plausibility, all-null measures, duplicate rows, and a check that period/segment-labeled aggregates are actually filtered to their period. A transparent confidence score is a function of these checks — not a number the model invents. - Explain (
@assay/narrator) — a plain-language finding, a chart chosen from the shape of the data (not the model's guess), and a provenance bundle (plan + exact SQL + checks).
Writing to a user's database is the one catastrophic failure this product cannot have. Two independent, structural controls prevent it (either alone suffices):
- Connection level — the agent connects with a role that has no write privileges (a
GRANT SELECT-only Postgres role;readOnlySQLite handles). Verified at onboarding by a probe write that must fail. - AST level — every generated query must parse to a single read statement or it is refused.
Column-level data policy is enforced structurally, two ways: restricted columns are removed
from the schema the planner ever sees (so it cannot select them) and redacted from every result
(so even a SELECT * cannot leak them). The result: a restricted column's data never reaches the
model — proven by a test that drives the full analyst loop and asserts the value appears in no
prompt. Row-level policy is modeled and enforced by database RLS / per-principal views, not by
unsound post-hoc filtering.
A monitor (@assay/watch) watches a metric on a schedule (@assay/chronos: a cron engine
plus a natural-language parser that refuses to guess). When the metric moves anomalously (z-score
or percent-change vs a rolling baseline), Watchtower launches the full analyst loop to
investigate why — so the explanation is verified and provenance-backed — and delivers an alert.
A worker ticks each minute and runs the monitors that are due.
A workflow (@assay/workflows) is a typed node graph (agent / tool / branch / wait / end),
serialized as diffable YAML, validated as a DAG (no cycles, all reachable), and executed on the
same analyst loop. The Console ships a visual editor with live validation and YAML export.
@assay/model-gateway is the single chokepoint for the LLM: tier routing (a cheap tier for
classification/repair, a stronger tier for planning), integer micro-USD cost accounting per run,
structured-output helpers, and a record/replay mode so the entire test suite runs deterministically
with no API key. Concrete model ids live in exactly one file.
A pnpm + Turborepo monorepo, TypeScript strict, ESM, Node 22+. Three apps (web, cli, worker)
over a set of focused packages (see the README for the map). Tests are part of every feature:
property tests on the kernel, recorded-fixture tests on the model path, integration tests against
Postgres and SQLite, and a live eval harness.
- No LLM framework. The agent loop, memory of the run, and the trust machinery are the product; a framework would hide exactly what needs to be explicit.
- One database. Postgres (with
pgvectoravailable) serves relational, adjacency-graph, and vector needs — one datastore, deliberately. - Deferred connectors are deferred, not stubbed. MySQL / BigQuery / Snowflake need live
services to test; they slot in behind the same
Connectorinterface when that infrastructure exists, rather than shipping untested.