Merge pull request #10 from BjornMelin/feat/neon-postgres-drizzle-orm

feat: align architecture and env config with Neon Postgres and Fluid compute stack

Author: BjornMelin

.env.example

@@ -25,6 +25,9 @@ AUTH_ALLOWED_EMAILS=
 #   Vercel OAuth callback URL allowlisting (Neon Auth uses a branch-specific callback URL).
 NEXT_PUBLIC_AUTH_SOCIAL_PROVIDERS=vercel
 
+# App base URL (used for server-to-server callbacks like QStash)
+APP_BASE_URL=
+
 # Database (Neon Postgres)
 DATABASE_URL=
 
@@ -42,9 +45,12 @@ QSTASH_CURRENT_SIGNING_KEY=
 QSTASH_NEXT_SIGNING_KEY=
 QSTASH_TOKEN=
 
-# Vercel AI Gateway (OpenAI-compatible)
+# Vercel AI Gateway
 AI_GATEWAY_API_KEY=
-AI_GATEWAY_BASE_URL=https://ai-gateway.vercel.sh/v1
+AI_GATEWAY_BASE_URL=https://ai-gateway.vercel.sh/v3/ai
+# Optional: default model IDs for the app (AI Gateway model IDs)
+AI_GATEWAY_CHAT_MODEL=xai/grok-4.1-fast-reasoning
+AI_GATEWAY_EMBEDDING_MODEL=alibaba/qwen3-embedding-4b
 
 # Vercel Blob (uploads)
 BLOB_READ_WRITE_TOKEN=

AGENTS.md

@@ -37,8 +37,12 @@ bun run fetch:models           # Update AI model catalog (requires AI_GATEWAY_AP
 
 ## Drizzle + database
 
-- `drizzle.config.ts` reads `DATABASE_URL` at runtime; `bun run db:*` commands will fail fast if it's missing.
+- `drizzle.config.ts` loads Next.js `.env*` files and requires `DATABASE_URL` for
+  commands that need a live DB connection (e.g. `db:migrate`, `db:studio`).
 - Schema lives in `src/db/schema.ts` and migrations are generated into `src/db/migrations`.
+- Runtime DB access uses Postgres TCP with pooling (`pg`) and Drizzle
+  (`drizzle-orm/node-postgres`). On Vercel Fluid compute, the pool is attached
+  with `attachDatabasePool` (`@vercel/functions`) in `src/db/client.ts`.
 
 ## Database
 

README.md

@@ -46,6 +46,10 @@ flowchart LR
 - Auth: Neon Auth (managed auth + UI components)
 - AI: Vercel AI SDK v6 + AI Gateway
 - DB: Neon Postgres + Drizzle ORM
+- DB connectivity (Vercel): Postgres TCP + connection pooling (`pg`) with
+  `attachDatabasePool` (`@vercel/functions`) for Fluid compute
+  ([Vercel Functions package](https://vercel.com/docs/functions/functions-api-reference/vercel-functions-package),
+  [Vercel KB: Connection Pooling with Vercel Functions](https://vercel.com/kb/guide/connection-pooling-with-functions))
 - Infra helpers: Upstash (Redis, QStash, Vector)
 - Quality: Biome (format/lint) + ESLint (TSDoc/JSDoc enforcement) + Vitest
 - Typing/Schema: Zod v4
@@ -101,6 +105,23 @@ bun install
 bun run dev
 ```
 
+### Database migrations
+
+Generate migrations (does not require a live DB connection):
+
+```bash
+bun run db:generate
+```
+
+Apply migrations (requires `DATABASE_URL`):
+
+```bash
+bun run db:migrate
+```
+
+Integration tests in `tests/integration/db.test.ts` run only when `DATABASE_URL`
+is set; otherwise they are skipped.
+
 Optional: implementation/deploy automation variables (GitHub/Vercel/Neon/Upstash)
 are documented in [`docs/ops/env.md`](./docs/ops/env.md).
 
@@ -149,6 +170,10 @@ bun run test
 bun run build
 ```
 
+Tests are colocated under `src/**` (including route handler tests in
+`src/app/api/**/__tests__`) with integration tests under
+`tests/integration`.
+
 ## Releases and versioning
 
 This repo uses Release Please and Conventional Commits.

bun.lock

@@ -1,18 +1,21 @@
 ---
 ADR: 0003
-Title: Database: Neon Postgres + Drizzle ORM
+Title: Database - Neon Postgres + Drizzle ORM
 Status: Accepted
-Version: 0.2
-Date: 2026-01-30
+Version: 0.4
+Date: 2026-02-03
 Supersedes: []
 Superseded-by: []
 Related: [ADR-0004, ADR-0005, ADR-0016]
 Tags: [architecture, data]
 References:
-  - [Neon serverless driver](https://neon.com/docs/serverless/serverless-driver)
+  - [Neon: Connecting to Neon from Vercel](https://neon.com/docs/guides/vercel-connection-methods)
+  - [Vercel: attachDatabasePool](https://vercel.com/docs/functions/functions-api-reference/vercel-functions-package)
   - [Neon + Drizzle guide](https://neon.com/docs/guides/drizzle)
   - [Drizzle connect Neon](https://orm.drizzle.team/docs/connect-neon)
+  - [Drizzle connect overview (node-postgres)](https://orm.drizzle.team/docs/connect-overview)
   - [Neon on Vercel](https://vercel.com/marketplace/neon)
+  - [Neon serverless driver (HTTP/WebSocket)](https://neon.com/docs/serverless/serverless-driver)
 ---
 
 ## Status
@@ -23,6 +26,10 @@ Accepted — 2026-01-30.
 
 Use Neon Postgres for relational persistence and Drizzle for schema/migrations.
 
+See [SPEC-0021](../spec/SPEC-0021-full-stack-finalization-fluid-compute-neon-upstash-ai-elements.md)
+for the cross-cutting “finalization” plan that ties DB, ingestion, retrieval,
+durable runs, and UI together.
+
 ## Context
 
 We need durable storage for projects, files, runs, steps, artifacts, and chat state. Neon provides a serverless-friendly driver for Vercel deployments and strong DX via marketplace integration. Drizzle provides type-safe schema and migrations with low boilerplate. The repo is already configured via `drizzle.config.ts` to use `src/db/schema.ts`.
@@ -37,7 +44,7 @@ We need durable storage for projects, files, runs, steps, artifacts, and chat st
 
 ## Alternatives
 
-- A: Neon + Drizzle — Pros: strong DX; serverless driver. Cons: Drizzle learning curve.
+- A: Neon + Drizzle — Pros: strong DX; Vercel-aligned pooling; serverless driver available where pooling isn’t safe. Cons: Drizzle learning curve.
 - B: Neon + Prisma — Pros: popular. Cons: heavier generated layer.
 - C: SQLite — Pros: local simplicity. Cons: poor cloud/serverless fit.
 
@@ -54,7 +61,15 @@ We need durable storage for projects, files, runs, steps, artifacts, and chat st
 
 ## Decision
 
-We will adopt **Neon Postgres** with the **Neon serverless driver** and **Drizzle ORM** for schema and migrations.
+We will adopt **Neon Postgres** with **Drizzle ORM** for schema and migrations.
+
+Connection method policy:
+
+- On Vercel **Fluid compute**, use Postgres TCP with a connection pool (`pg`)
+  and attach the pool via `attachDatabasePool`.
+  - Implementation: `src/db/client.ts`
+- In classic serverless environments where safe pooling is unavailable, Neon’s
+  HTTP/WebSocket serverless driver is an acceptable alternative.
 
 ## Constraints
 
@@ -98,22 +113,35 @@ flowchart LR
 
 - `src/db/schema.ts`: schema definitions.
 - `src/db/migrations/`: generated migrations.
-- `src/db/client.ts`: Neon driver + Drizzle instance (to add).
+- `src/db/client.ts`: Neon driver + Drizzle instance (implemented).
+- `src/lib/data/*.server.ts`: server-only Data Access Layer (DAL) modules that
+  encapsulate all DB reads/writes (implemented).
 
 ### Implementation Details
 
 - UUID PKs, timestamps.
 - JSONB for citations and usage.
 - Index hot fields (`project_id`, `run_id`, `step_name`).
+- Runtime driver: Postgres TCP with pooling (Vercel Fluid compute recommendation).
+  - `pg` (node-postgres) + `drizzle-orm/node-postgres`.
+  - On Vercel, we attach the pool using `@vercel/functions` to align with
+    Vercel’s pooling semantics.
+  - In classic serverless environments where connection pooling is unsafe,
+    `@neondatabase/serverless` (HTTP/WebSocket) remains a viable alternative.
 
 ## Testing
 
 - Integration: migration apply + CRUD.
 - Regression: idempotent step updates maintain consistency.
+  - Integration tests live in `tests/integration/db.test.ts` and run only when
+    `DATABASE_URL` is available.
 
 ## Implementation Notes
 
-- Prefer short-lived connections via serverless driver; avoid long pooled TCP connections in serverless.
+- Prefer pooled Postgres TCP connections on Vercel Fluid compute (recommended).
+- Keep pool sizes conservative to avoid exhausting Neon connection limits.
+- If deploying to a classic serverless platform without safe pooling, switch to
+  the Neon serverless driver (HTTP/WebSocket).
 
 ## Consequences
 
@@ -134,9 +162,11 @@ flowchart LR
 
 ### Dependencies
 
-- **Added**: drizzle-orm, drizzle-kit, @neondatabase/serverless
+- **Added**: drizzle-orm, drizzle-kit, pg, @vercel/functions
 
 ## Changelog
 
 - **0.1 (2026-01-29)**: Initial version.
 - **0.2 (2026-01-30)**: Updated for current repo baseline (Bun, `src/` layout, CI).
+- **0.3 (2026-02-03)**: Updated connection method policy for Vercel Fluid compute (TCP pooling + `attachDatabasePool`).
+- **0.4 (2026-02-03)**: Linked to SPEC-0021 as the cross-cutting finalization spec.

docs/architecture/adr/ADR-0003-database-neon-postgres-drizzle-orm.md

@@ -2,8 +2,8 @@
 ADR: 0004
 Title: Retrieval: Upstash Vector for semantic + hybrid search
 Status: Accepted
-Version: 0.3
-Date: 2026-02-01
+Version: 0.4
+Date: 2026-02-03
 Supersedes: []
 Superseded-by: []
 Related: [ADR-0003, ADR-0006, ADR-0024]
@@ -31,6 +31,10 @@ Prefer **HYBRID** vector indexes (dense + lexical) when provisioning new indexes
 to improve exact-token recall (important for code and identifiers), while still
 retaining semantic retrieval quality.
 
+See [SPEC-0021](../spec/SPEC-0021-full-stack-finalization-fluid-compute-neon-upstash-ai-elements.md)
+for the cross-cutting “finalization” plan that ties Vector retrieval into
+ingestion, chat, search, and durable runs end-to-end.
+
 ## Context
 
 System quality depends on grounding responses in uploaded sources, previously
@@ -122,9 +126,9 @@ flowchart LR
 
 ### Architecture Overview
 
-Planned implementation module:
+Implementation module:
 
-- `src/lib/upstash/vector.ts`: client + helper functions (namespace, upsert,
+- `src/lib/upstash/vector.server.ts`: client + helper functions (namespace, upsert,
   query, delete).
 
 ### Implementation Details
@@ -179,3 +183,4 @@ Planned implementation module:
 - **0.1 (2026-01-29)**: Initial version.
 - **0.2 (2026-01-30)**: Updated for current repo baseline (Bun, `src/` layout, CI).
 - **0.3 (2026-02-01)**: Updated for hybrid retrieval guidance.
+- **0.4 (2026-02-03)**: Updated file path references for server-only Vector client module.

docs/architecture/adr/ADR-0004-retrieval-upstash-vector-for-semantic-search.md

@@ -2,8 +2,8 @@
 ADR: 0005
 Title: Orchestration: Upstash QStash for durable workflows
 Status: Accepted
-Version: 0.2
-Date: 2026-01-30
+Version: 0.3
+Date: 2026-02-03
 Supersedes: []
 Superseded-by: []
 Related: [ADR-0013, ADR-0012]
@@ -21,6 +21,10 @@ Accepted — 2026-01-30.
 
 Use QStash to execute multi-step runs durably and idempotently.
 
+See [SPEC-0021](../spec/SPEC-0021-full-stack-finalization-fluid-compute-neon-upstash-ai-elements.md)
+for the cross-cutting “finalization” plan that ties QStash orchestration into
+run persistence, UI, and step execution patterns.
+
 ## Context
 
 Full spec generation and research can exceed route execution limits. Durable orchestration ensures work completes even if the user closes the browser. QStash provides serverless-friendly HTTP-based queueing, signature verification, and retries.
@@ -146,3 +150,4 @@ flowchart LR
 
 - **0.1 (2026-01-29)**: Initial version.
 - **0.2 (2026-01-30)**: Updated for current repo baseline (Bun, `src/` layout, CI).
+- **0.3 (2026-02-03)**: Linked to SPEC-0021 as the cross-cutting finalization spec.

docs/architecture/adr/ADR-0005-orchestration-upstash-qstash-for-durable-workflows.md

@@ -2,8 +2,8 @@
 ADR: 0006
 Title: Agent runtime: AI SDK v6 ToolLoopAgent + streaming UI responses
 Status: Accepted
-Version: 0.2
-Date: 2026-01-30
+Version: 0.3
+Date: 2026-02-03
 Supersedes: []
 Superseded-by: []
 Related: [ADR-0007, ADR-0011, ADR-0012]
@@ -22,6 +22,10 @@ Accepted — 2026-01-30.
 
 Use AI SDK v6 agents for multi-step tool loops and streaming chat UX.
 
+See [SPEC-0021](../spec/SPEC-0021-full-stack-finalization-fluid-compute-neon-upstash-ai-elements.md)
+for the cross-cutting “finalization” plan that ties agent streaming into the
+workspace UI, retrieval, caching, and durable orchestration.
+
 ## Context
 
 The system requires multi-step reasoning and tool usage. AI SDK v6 provides ToolLoopAgent for iterative tool calls and createAgentUIStreamResponse for streaming message parts to the UI in Next.js Route Handlers.
@@ -141,3 +145,4 @@ flowchart LR
 
 - **0.1 (2026-01-29)**: Initial version.
 - **0.2 (2026-01-30)**: Updated for current repo baseline (Bun, `src/` layout, CI).
+- **0.3 (2026-02-03)**: Linked to SPEC-0021 as the cross-cutting finalization spec.

docs/architecture/adr/ADR-0006-agent-runtime-ai-sdk-v6-toolloopagent-streaming-ui-responses.md

@@ -2,8 +2,8 @@
 ADR: 0007
 Title: Model access: Vercel AI Gateway exclusively
 Status: Accepted
-Version: 0.2
-Date: 2026-01-30
+Version: 0.3
+Date: 2026-02-03
 Supersedes: []
 Superseded-by: []
 Related: [ADR-0013, ADR-0021]
@@ -94,13 +94,17 @@ flowchart LR
 
 ### Architecture Overview
 
-- `src/lib/ai/gateway.ts` exposes model factories and model listing for UI.
+- `src/lib/ai/gateway.server.ts` exposes AI Gateway provider wiring and model configuration.
 - `scripts/fetch-models.sh` fetches `/models` into `docs/ai-gateway-models.json`.
 
 ### Implementation Details
 
 - Persist `model`, `provider`, `token_usage`, `latency_ms` in run steps.
 - Provide run-level budget checks before expensive steps.
+- Default model IDs are config-driven via env vars (see `docs/ops/env.md` and
+  SPEC-0021):
+  - `AI_GATEWAY_CHAT_MODEL` (default: `xai/grok-4.1-fast-reasoning`)
+  - `AI_GATEWAY_EMBEDDING_MODEL` (default: `alibaba/qwen3-embedding-4b`)
 
 ## Testing
 
@@ -138,3 +142,4 @@ flowchart LR
 
 - **0.1 (2026-01-29)**: Initial version.
 - **0.2 (2026-01-30)**: Updated for current repo baseline (Bun, `src/` layout, CI).
+- **0.3 (2026-02-03)**: Updated file path references and recorded final default model IDs (SPEC-0021).

docs/architecture/adr/ADR-0007-model-access-vercel-ai-gateway-exclusively.md

@@ -99,6 +99,10 @@ flowchart LR
 ### Implementation Details
 
 - `src/app/api/upload/route.ts`: streams to Blob.
+- Upload processing runs per-file work in parallel; async ingestion enqueues a
+  QStash job per file with deduplication ids and labels.
+  ([QStash deduplication](https://upstash.com/docs/qstash/features/deduplication),
+  [QStash publish API](https://upstash.com/docs/qstash/api-reference/messages/publish-a-message))
 - `src/lib/blob/client.ts`: wrapper with typed metadata.
 - Persist `blobUrl`, `sha256`, `sizeBytes`, `mimeType`.