test: rebuild the testing architecture — Vitest everywhere, self-provisioning integration DB, five-layer contract

.github/workflows/ci.yml

@@ -1,7 +1,7 @@
 # Test CI (#70) — gate PRs on the checks that already pass locally:
-# lint / build / unit tests across the workspace, plus apps/api/scripts/rls-test.sh
-# (cross-tenant RLS isolation + the auth/chat e2e over real HTTP against a
-# throwaway Postgres — same script as local, GitHub-hosted runners have Docker).
+# lint / build / unit tests across the workspace, plus the api integration suite
+# (cross-tenant RLS isolation + the HTTP-boundary suites against a throwaway
+# Postgres self-provisioned by Testcontainers — same command as local).
 #
 # Actions are pinned to commit SHAs (comment = the release tag they resolve to).
 
@@ -67,9 +67,8 @@ jobs:
       - run: git diff --exit-code
 
       # Unit tests via the turbo task graph (cached like every other task).
-      # DB-backed suites self-skip without POSTGRES_URL / TEST_DATABASE_URL;
-      # model evals stay opt-in behind RUN_MODEL_EVALS — all three are declared
-      # in apps/api/turbo.json's test.env so their presence busts the cache.
+      # Strictly dependency-free: DB-backed suites live in the separate
+      # `test:integration` task (the rls job), evals behind `test:evals`.
       - run: pnpm exec turbo run test
 
   rls:
@@ -90,10 +89,13 @@ jobs:
 
       - run: pnpm install --frozen-lockfile
 
-      # Proves FORCE RLS cross-tenant isolation and runs the HTTP e2e suites
-      # against a throwaway Postgres owned by a non-superuser role — the same
-      # worst-case single-role deployment the script exercises locally.
-      - run: bash apps/api/scripts/rls-test.sh
+      # Self-provisioning (Testcontainers in the project's globalSetup): a
+      # throwaway Postgres owned by a NON-superuser role that runs the
+      # migrations — the worst-case single-role self-hosted deployment, so a
+      # green RLS suite proves FORCE constrains even the table owner. The
+      # same command runs the HTTP-boundary suites; locally it is identical:
+      # `pnpm --filter api test:integration` and nothing else.
+      - run: pnpm --filter api test:integration
 
   browser-e2e:
     name: browser e2e (Playwright, worker mode)

AGENTS.md

@@ -16,6 +16,7 @@ The product overview (what llame is) is short and always relevant, so it is impo
 - [CHANGELOG.md](CHANGELOG.md) — shipped chronology
 - [`docs/research`](docs/research) — noncanonical evidence, alternatives, and decision provenance
 - [DESIGN.md](DESIGN.md) — design system reference (visual language, OKLCH tokens, component stylings); consult before building or restyling any UI
+- [docs/testing.md](docs/testing.md) — the test pyramid, suffix-is-runner naming contract, placement rules, and CI mapping
 - [docs/scaling.md](docs/scaling.md) — horizontal-scaling topology, invariants, and the design constraints for the durable-run worker split (#48/#50)
 
 ## Monorepo layout
@@ -48,7 +49,7 @@ pnpm test:e2e:report     # playwright show-report
 ```
 
 Scope to one workspace with `pnpm --filter web <script>` (or `--filter api`).
-Install Playwright browsers once with `pnpm exec playwright install chromium` if the local browser cache is missing. For E2E, Playwright starts a throwaway Docker Postgres, applies migrations, then starts `apps/api` and `apps/web`; set `POSTGRES_URL` only to use an already-migrated external database instead. Authenticated E2E tests should use the worker-scoped fixture from `e2e/fixtures.ts`, which writes per-worker storage state under `.auth/`; destructive session tests should request `freshAccount`. Override `E2E_WEB_PORT`, `E2E_API_PORT`, `E2E_DB_PORT`, or `E2E_DB_READY_PORT` only when the default E2E ports (`4300`/`4301`/`55433`/`4302`) conflict. Next.js 16 enforces one dev instance **per project directory**, not per port — `pnpm test:e2e`'s own `next dev --port 4300` refuses to start (and the whole run fails) while a manual `pnpm dev`/`next dev` is running anywhere against `apps/web`, even on a different port; stop the manual dev server first.
+Install Playwright browsers once with `pnpm exec playwright install chromium` if the local browser cache is missing. For E2E, Playwright starts a throwaway Docker Postgres, applies migrations, then starts `apps/api` and `apps/web`; set `POSTGRES_URL` only to use an already-migrated external database instead. Authenticated E2E tests should use the worker-scoped fixture from `e2e/support/fixtures.ts`, which writes per-worker storage state under `.auth/`; destructive session tests should request `freshAccount`. Override `E2E_WEB_PORT`, `E2E_API_PORT`, `E2E_DB_PORT`, or `E2E_DB_READY_PORT` only when the default E2E ports (`4300`/`4301`/`55433`/`4302`) conflict. Next.js 16 enforces one dev instance **per project directory**, not per port — `pnpm test:e2e`'s own `next dev --port 4300` refuses to start (and the whole run fails) while a manual `pnpm dev`/`next dev` is running anywhere against `apps/web`, even on a different port; stop the manual dev server first.
 
 ## Local database (docker)
 
@@ -61,13 +62,14 @@ pnpm db:provision-rls # assign privileged RLS helper ownership after migrations
 pnpm db:studio    ·   pnpm db:psql   ·   pnpm db:logs
 ```
 
-Dev provisions a non-superuser role so RLS (incl. `FORCE`) is exercised as in production — the role model, the per-request `app.current_user_id` requirement, and `scripts/rls-test.sh` are documented in [apps/api/AGENTS.md](apps/api/AGENTS.md). `apps/api` is the sole DB owner; `apps/web` holds no database connection and reads/writes only through `apps/api` (SPEC.md §22.0).
+Dev provisions a non-superuser role so RLS (incl. `FORCE`) is exercised as in production — the role model, the per-request `app.current_user_id` requirement, and the self-provisioning `test:integration` gate are documented in [apps/api/AGENTS.md](apps/api/AGENTS.md). `apps/api` is the sole DB owner; `apps/web` holds no database connection and reads/writes only through `apps/api` (SPEC.md §22.0).
 
 ## Conventions
 
 - TypeScript only across web/api/worker — no second backend language (SPEC.md §23).
 - Drizzle ORM for all DB access; generate migrations with `drizzle-kit`, never hand-write migration SQL.
 - Conventional commits (e.g. `feat(api):`, `docs(spec):`).
+- Tests follow [docs/testing.md](docs/testing.md): `*.test.ts(x)` is Vitest everywhere (`.integration` infix = needs real Postgres; root `e2e/` is Playwright's island); component behavior belongs in Storybook stories, not jsdom render tests; DB-backed suites fail loudly, never skip silently.
 - UI work follows the design language in [DESIGN.md](DESIGN.md) — compose `@workspace/ui` primitives and the semantic tokens; no ad-hoc colors or a brand hue (see its §10 Do/Don't).
 
 ## Storybook MCP tools

CHANGELOG.md

@@ -1,5 +1,9 @@
 _Reverse-chronological record of shipped work — features, fixes, and chores. Newest first._
 
+# 2026-07-31
+
+- Rebuilt the testing architecture around one explicit contract, documented in [docs/testing.md](docs/testing.md) and validated by a three-model review panel against twelve placement scenarios before implementation. The old state had four runners and three different meanings for `.spec.ts` depending on directory: `apps/api` ran Jest (ts-jest) with unit `.spec.ts`, DB-backed `.integration.spec.ts` that **silently self-skipped to zero tests** without `TEST_DATABASE_URL` in the same `pnpm test` invocation, and a second Jest config for `test/*.e2e-spec.ts` held together by `forceExit`; `apps/web` ran Vitest `.test.ts(x)`; Playwright owned root `e2e/*.spec.ts`; the RLS proof lived in a 150-line bash provisioner (`rls-test.sh`). Now the pyramid is five layers with stock tools at every level: **`*.test.ts(x)` is Vitest everywhere** (root `e2e/` is Playwright's island with its own `.spec.ts` convention, unreachable by any Vitest glob; `nest g` no longer scaffolds `.spec.ts` files), and the `.integration` infix is the only other marker. `apps/api` migrated Jest→Vitest (80 files; `unplugin-swc` emits the decorator metadata Nest DI silently loses under esbuild) with three projects: `unit` (`pnpm test`, zero external deps), `integration`, and `evals` (opt-in via `RUN_MODEL_EVALS`). The **integration project is self-provisioning**: a Testcontainers globalSetup starts a throwaway Postgres and reproduces the worst-case self-hosted topology — a non-superuser `app` role that owns the schema and runs the migrations, plus the `app_rls` BYPASSRLS function owner — so `pnpm --filter api test:integration` is the entire story locally and in CI, `rls-test.sh` is deleted, and a green RLS suite still proves `FORCE` constrains even the table owner (`TEST_DATABASE_URL` overrides for docker-less machines). The api's separate HTTP-e2e layer was **dissolved into integration**: supertest suites are integration tests of the HTTP boundary, moved into `src/` feature modules as `*.integration.test.ts` (shared helpers in `src/testing/`), with the per-suite pg-boss schema isolation now covering every integration file; `worker.module` was likewise reclassified — it required a real Postgres while hiding in the unit suite. Nothing silently skips (no database = loud failure, not a green zero-test run) and the DB-backed turbo tasks are `cache: false`, because a stale cached PASS against live state turbo cannot hash would be worse than the silent skip this kills. Root `e2e/` reorganized per product surface (`e2e/web/` suites, `e2e/support/` boot infra), so a future non-browser surface adds `e2e/<surface>/` instead of competing for the root. The migration surfaced and fixed a real latent bug the silent-skip era had buried: `worker-harness.ts` only set `POSTGRES_URL` from `TEST_DATABASE_URL` when unset, but `ConfigModule.forRoot` had already leaked the developer's `.env.local` dev-database URL into `process.env` — so on a dev machine the durable-run worker suites would have run against the dev database; the override is now unconditional, and the globalSetup setting both URLs closes the same trap for the HTTP suites. Verified against a provisioned worst-case-owner Postgres: 507 unit and 256 integration tests green (including 8 formerly-skipped worker tests and the 62 former e2e tests); Playwright collects all 19 browser tests post-reorg. Ts-jest, the ESM `transformIgnorePatterns` hacks, `jest-e2e.json`, `rls-test.sh`, and six Jest-toolchain devDependencies are gone. Deliberate follow-ups are listed at the end of docs/testing.md: migrating story-eligible `apps/web` component tests into Storybook play tests per the ≤2-mocks rubric, replacing the source-regex `chat-page.hydration.test.ts` with behavioral coverage, removing the now-dead in-file `describeIfDb` guards, and ratcheting the four temporarily-disabled `vitest/*` oxlint style rules back on.
+
 # 2026-07-28
 
 - **BREAKING (operators with a custom `systemPromptFile`)**: model system-prompt files are now **Handlebars templates** instead of the bespoke `${model.id}` / `${model.name}` / `$${model.name}` grammar. Hard cutover, no dual-syntax window and no compatibility shim — llame is not in production use, so `${...}` simply carries no meaning now and is ordinary prose. The motivation is **absence, not syntax**: the old grammar could substitute a value but not express that one was missing, so an unset value left its label, heading, or enclosing sentence stranded — and per-user context, project instructions, and later memory surfaces all need exactly that. Three designs were tried to avoid conditionals and each failed on a concrete case (an llame-owned block operators could not reshape; a "drop the line whose expressions all render empty" rule that silently deleted operator instructions sharing the line; absence markers that put llame's prose inside operator sentences). Renderable paths remain exactly `{{model.id}}` and `{{model.name}}` — **no new data is exposed** — with `if`/`unless` conditionals, comments, and whitespace control now available to operators. Validation is **deny-by-default at boot** over the parsed AST: permitted node kinds are literal content, value expressions, block expressions, and comments; everything else aborts startup naming the model id and the construct, which covers partials in all three syntactic forms (`{{> x}}`, `{{#> x}}…{{/x}}`, and inline partials via `{{#*inline}}` — verified to render if permitted) that would otherwise reintroduce the prompt composition `model-system-prompts` forbids. One deliberate behavior change: an **unknown path aborts boot** (typos fail loudly) but an **absent value renders empty**, so `{{#if model.name}}…{{model.name}}…{{/if}}` is expressible. Three implementation facts were verified against handlebars 4.7.9 rather than assumed, and are documented in `apps/api/AGENTS.md` because each is a trap: `Handlebars.create()` shares `Utils` **by reference** with the global export, so patching `escapeExpression` on a "isolated" environment changes escaping process-wide — llame instead escapes when building the context and emits `SafeString`s; the default escaping converts `'`, `"`, `=`, and backticks into character references, mangling prose and code fragments, so escaping is narrowed to exactly `&`, `<`, `>`; and a `SafeString` is truthy **even when it wraps an empty string**, so absent and empty values are omitted from the render context entirely rather than passed as empty, or every `{{#if}}` over them would silently evaluate true (values are trimmed, whitespace-only counts as absent). The render context is a hand-built projection and never a database row — `users` carries a `password` column, and a row passed as context would put a credential hash into a system prompt, an immutable snapshot, and the owner-visible receipt.

apps/api/.oxlintrc.json

@@ -3,15 +3,28 @@
   "options": {
     "typeAware": true
   },
+  "plugins": ["typescript", "unicorn", "oxc", "vitest"],
   "categories": {
     "correctness": "error"
   },
   "env": {
     "node": true,
-    "jest": true
+    "vitest": true
   },
   "ignorePatterns": ["dist/**", "src/db/migrations/**"],
   "rules": {
+    // Requires a type parameter on every `vi.fn()` — noisy against our test
+    // style, which relies on inference/assertion at the call site instead.
+    "vitest/require-mock-type-parameters": "off",
+    // Pre-existing test style predating the vitest plugin (expects inside
+    // rejects-catch blocks, shared beforeAll assertions, helper-asserting
+    // tests). Tightening these is follow-up cleanup, not part of the Jest →
+    // Vitest migration — flip one to "error" and fix its findings when
+    // touching the affected suites.
+    "vitest/no-conditional-expect": "off",
+    "vitest/no-standalone-expect": "off",
+    "vitest/require-to-throw-message": "off",
+    "vitest/expect-expect": "off",
     "typescript/await-thenable": "error",
     "typescript/no-array-delete": "error",
     "typescript/no-base-to-string": "error",