Merge pull request #13 from BjornMelin/feat/workflow-ai-sdk-streamdown-ai-elements

feat(workflow): integrate Vercel Workflow + AI Elements

Author: BjornMelin

.env.example

@@ -30,6 +30,8 @@ APP_BASE_URL=
 
 # Database (Neon Postgres)
 DATABASE_URL=
+# Optional: unpooled connection string (recommended for migrations/DDL).
+DATABASE_URL_UNPOOLED=
 
 # Upstash Redis (cache + rate limit)
 UPSTASH_REDIS_REST_TOKEN=

.gitignore

@@ -107,6 +107,7 @@ repomix-output.md
 .gemini/
 .claude/
 opensrc/
+config/mcporter.json
 # next-agents-md
 .next-docs/
 .env*.local

AGENTS.md

@@ -35,6 +35,12 @@ bun run fetch:models           # Update AI model catalog (requires AI_GATEWAY_AP
   - TSDoc syntax enforcement (`tsdoc/syntax`)
   - JSDoc policy on exported APIs (warn-level for iteration speed)
 
+## Import policy
+
+- Do **not** switch to package barrel imports by default.
+- Barrel imports are allowed only for packages listed in `next.config.ts` `experimental.optimizePackageImports` (currently `radix-ui` and `lucide-react`).
+- For all other packages, use explicit/non-barrel imports.
+
 ## Drizzle + database
 
 - `drizzle.config.ts` loads Next.js `.env*` files and requires `DATABASE_URL` for
@@ -58,8 +64,9 @@ bun run fetch:models           # Update AI model catalog (requires AI_GATEWAY_AP
 
 ## AI Gateway
 
-- Prefer `ai` package `gateway(...)` usage for model routing.
-- OpenAI-compatible base URL is `https://ai-gateway.vercel.sh/v1`.
+- Prefer the AI SDK's `gateway(...)` / `createGateway(...)` provider for model routing.
+- AI SDK Gateway provider base URL is `https://ai-gateway.vercel.sh/v3/ai` (default; configurable via `AI_GATEWAY_BASE_URL`).
+- OpenAI-compatible base URL is `https://ai-gateway.vercel.sh/v1` (used for OpenAI SDK compatibility and `GET /models`).
 - `bun run fetch:models` writes a model catalog JSON (default: `docs/ai-gateway-models.json`).
 
 ## Documentation standards

bun.lock

@@ -0,0 +1,23 @@
+{
+  "$schema": "https://ui.shadcn.com/schema.json",
+  "aliases": {
+    "components": "@/components",
+    "hooks": "@/hooks",
+    "lib": "@/lib",
+    "ui": "@/components/ui",
+    "utils": "@/lib/utils"
+  },
+  "iconLibrary": "lucide",
+  "registries": {},
+  "rsc": true,
+  "rtl": false,
+  "style": "new-york",
+  "tailwind": {
+    "baseColor": "zinc",
+    "config": "",
+    "css": "src/app/globals.css",
+    "cssVariables": true,
+    "prefix": ""
+  },
+  "tsx": true
+}

components.json

@@ -1,12 +1,12 @@
 ---
 ADR: 0005
-Title: Orchestration: Upstash QStash for durable workflows
+Title: Orchestration: Upstash QStash for background jobs (ingestion + fanout)
 Status: Accepted
-Version: 0.3
+Version: 0.4
 Date: 2026-02-03
 Supersedes: []
 Superseded-by: []
-Related: [ADR-0013, ADR-0012]
+Related: [ADR-0013, ADR-0012, ADR-0026]
 Tags: [architecture, reliability]
 References:
   - [QStash Next.js quickstart](https://upstash.com/docs/qstash/quickstarts/vercel-nextjs)
@@ -16,18 +16,26 @@ References:
 ## Status
 
 Accepted — 2026-01-30.
+Updated — 2026-02-03: re-scoped to background jobs (ingestion + fanout); interactive runs moved to ADR-0026.
 
 ## Description
 
-Use QStash to execute multi-step runs durably and idempotently.
+Use QStash to execute **background jobs** durably and idempotently, with signed
+delivery, retries, and rate limiting.
+
+This ADR no longer governs **interactive runs or chat streaming**. Those are
+handled by Vercel Workflow DevKit per [ADR-0026](./ADR-0026-orchestration-vercel-workflow-devkit-for-interactive-runs.md).
 
 See [SPEC-0021](../spec/SPEC-0021-full-stack-finalization-fluid-compute-neon-upstash-ai-elements.md)
 for the cross-cutting “finalization” plan that ties QStash orchestration into
-run persistence, UI, and step execution patterns.
+ingestion, UI, and background step execution patterns.
 
 ## Context
 
-Full spec generation and research can exceed route execution limits. Durable orchestration ensures work completes even if the user closes the browser. QStash provides serverless-friendly HTTP-based queueing, signature verification, and retries.
+Ingestion and other background jobs (extract/chunk/embed/index) can exceed route
+execution limits and should continue even if the user closes the browser. QStash
+provides serverless-friendly HTTP-based queueing, signature verification, and
+retries.
 
 ## Decision Drivers
 
@@ -55,43 +63,46 @@ Full spec generation and research can exceed route execution limits. Durable orc
 
 ## Decision
 
-We will use **Upstash QStash** to orchestrate run steps, verifying signatures and enforcing step idempotency.
+We will use **Upstash QStash** to orchestrate background jobs (primarily the ingestion pipeline), verifying signatures and enforcing idempotency.
 
 ## Constraints
 
-- Step endpoints must verify QStash signatures.
+- Background job endpoints must verify QStash signatures.
 - Verification requires `QSTASH_CURRENT_SIGNING_KEY` and `QSTASH_NEXT_SIGNING_KEY`.
-- Steps must be idempotent per (runId, stepName).
+- Jobs must be idempotent (e.g., by `(fileId, stage)` and/or content hash).
 - Ensure at-least-once delivery does not duplicate artifacts.
 
 ## High-Level Architecture
 
 ```mermaid
 flowchart LR
-  UI --> Runs[/src/app/api/runs/]
-  Runs --> DB[(Neon)]
-  Runs --> Q[(QStash)]
-  Q --> Step[/src/app/api/jobs/run-step/]
-  Step --> DB
-  Step --> Tools[AI Gateway / Exa / Firecrawl / Vector]
+  UI --> Upload[/src/app/api/upload/]
+  Upload --> DB[(Neon)]
+  Upload --> Q[(QStash)]
+  Q --> Ingest[/src/app/api/jobs/ingest-file/]
+  Ingest --> DB
+  Ingest --> Tools[AI Gateway / Vector / Blob]
 ```
 
 ## Related Requirements
 
 ### Functional Requirements
 
-- **FR-010:** Durable run execution.
-- **FR-011:** Persist step status and outputs.
+- **FR-003:** Upload files to a project.
+- **FR-004:** Store original files durably.
+- **FR-005:** Extract text + structural metadata.
+- **FR-006:** Chunk extracted content.
+- **FR-007:** Embed + index chunks in vector store.
 
 ### Non-Functional Requirements
 
 - **NFR-004:** Observability for each step.
-- **NFR-005:** Deterministic artifacts per step.
+- **NFR-006:** Cost controls for background processing.
 
 ### Performance Requirements
 
-- **PR-004:** Runs survive disconnects.
 - **PR-005:** Idempotent steps.
+- **PR-003:** Ingest 10 MB PDF within target (excluding queue delay).
 
 ### Integration Requirements
 
@@ -101,25 +112,34 @@ flowchart LR
 
 ### Architecture Overview
 
-- A run is a DB record with ordered steps.
-- Each step handler acquires a DB lease, executes, writes outputs, and marks status.
+QStash is used to deliver background processing jobs (ingestion and related
+fanout tasks) reliably:
+
+- upload endpoint publishes a QStash message referencing the file/project
+- ingestion worker verifies the signature, loads state from DB, and advances the pipeline
+- retries are expected; all operations must be idempotent
+
+The ingestion pipeline details are defined in:
+
+- [SPEC-0003](../spec/SPEC-0003-upload-ingestion-pipeline.md)
+- [SPEC-0021](../spec/SPEC-0021-full-stack-finalization-fluid-compute-neon-upstash-ai-elements.md)
 
 ### Implementation Details
 
-- Store `attempt` and `error` in `run_steps`.
-- Provide a UI action to retry a failed step.
-- Cap maximum retries and enforce budgets.
+- Verify QStash signatures on every background job route.
+- Enforce idempotency at the file/content hash boundary (avoid double indexing).
+- Cap retries and enforce budgets to avoid runaway costs.
 
 ### File locations (target)
 
-- `src/app/api/runs/route.ts`
-- `src/app/api/jobs/run-step/route.ts`
+- `src/app/api/upload/route.ts`
+- `src/app/api/jobs/ingest-file/route.ts`
 
 ## Testing
 
-- Integration: replay same step twice yields identical artifacts.
 - Contract: unsigned requests rejected.
-- E2E: run completes after client disconnect.
+- Integration: replay same job twice yields the same DB/vector state (idempotent).
+- E2E: ingestion completes after client disconnect.
 
 ## Implementation Notes
 
@@ -151,3 +171,4 @@ flowchart LR
 - **0.1 (2026-01-29)**: Initial version.
 - **0.2 (2026-01-30)**: Updated for current repo baseline (Bun, `src/` layout, CI).
 - **0.3 (2026-02-03)**: Linked to SPEC-0021 as the cross-cutting finalization spec.
+- **0.4 (2026-02-03)**: Re-scoped to background jobs (ingestion + fanout); interactive runs moved to ADR-0026.

docs/architecture/adr/ADR-0005-orchestration-upstash-qstash-for-durable-workflows.md

@@ -1,22 +1,25 @@
 ---
 ADR: 0006
-Title: Agent runtime: AI SDK v6 ToolLoopAgent + streaming UI responses
+Title: Agent runtime: AI SDK v6 agents + streaming UI responses (Workflow DevKit durable sessions)
 Status: Accepted
-Version: 0.3
+Version: 0.4
 Date: 2026-02-03
 Supersedes: []
 Superseded-by: []
-Related: [ADR-0007, ADR-0011, ADR-0012]
+Related: [ADR-0007, ADR-0011, ADR-0012, ADR-0026]
 Tags: [agents, architecture]
 References:
   - [AI SDK agents](https://ai-sdk.dev/docs/agents/overview)
   - [ToolLoopAgent](https://ai-sdk.dev/docs/reference/ai-sdk-core/tool-loop-agent)
-  - [createAgentUIStreamResponse](https://ai-sdk.dev/docs/reference/ai-sdk-core/create-agent-ui-stream-response)
+  - [createUIMessageStreamResponse](https://ai-sdk.dev/docs/reference/ai-sdk-ui/create-ui-message-stream-response)
+  - [Workflow DevKit: DurableAgent](https://useworkflow.dev/docs/api-reference/workflow-ai/durable-agent)
+  - [Workflow DevKit: Resumable streams](https://useworkflow.dev/docs/ai/resumable-streams)
 ---
 
 ## Status
 
 Accepted — 2026-01-30.
+Updated — 2026-02-03: aligned with Workflow DevKit durable sessions and resumable streaming (ADR-0026).
 
 ## Description
 
@@ -26,9 +29,17 @@ See [SPEC-0021](../spec/SPEC-0021-full-stack-finalization-fluid-compute-neon-ups
 for the cross-cutting “finalization” plan that ties agent streaming into the
 workspace UI, retrieval, caching, and durable orchestration.
 
+See [SPEC-0022](../spec/SPEC-0022-vercel-workflow-durable-runs-and-streaming-contracts.md)
+for the canonical Workflow DevKit integration (durable multi-turn sessions,
+resumable streams, hook endpoints).
+
 ## Context
 
-The system requires multi-step reasoning and tool usage. AI SDK v6 provides ToolLoopAgent for iterative tool calls and createAgentUIStreamResponse for streaming message parts to the UI in Next.js Route Handlers.
+The system requires multi-step reasoning and tool usage. AI SDK v6 provides
+agent loops (ToolLoopAgent semantics) and UI message streaming primitives.
+For production-grade resumable streaming and multi-turn durability, we run the
+agent loop inside Workflow DevKit using `@workflow/ai` `DurableAgent` and stream
+via Workflow run streams (see [ADR-0026](./ADR-0026-orchestration-vercel-workflow-devkit-for-interactive-runs.md) for orchestration rationale).
 
 ## Decision Drivers
 
@@ -56,7 +67,10 @@ The system requires multi-step reasoning and tool usage. AI SDK v6 provides Tool
 
 ## Decision
 
-We will implement agents using **ToolLoopAgent** and stream UI output using **createAgentUIStreamResponse** in Route Handlers.
+We will implement agent loops using **AI SDK v6 agents** (ToolLoopAgent-style
+tool loops) and stream UI output using **AI SDK UI message streams**. In
+interactive/durable paths, we will execute these loops inside Workflow DevKit
+via `@workflow/ai` `DurableAgent` and expose resumable streams to clients.
 
 ## Constraints
 
@@ -92,7 +106,7 @@ flowchart LR
 ### Performance Requirements
 
 - **PR-001:** fast streaming start.
-- **PR-004:** durable runs via QStash.
+- **PR-004:** durable runs and resumable streams (Workflow DevKit).
 
 ### Integration Requirements
 
@@ -146,3 +160,4 @@ flowchart LR
 - **0.1 (2026-01-29)**: Initial version.
 - **0.2 (2026-01-30)**: Updated for current repo baseline (Bun, `src/` layout, CI).
 - **0.3 (2026-02-03)**: Linked to SPEC-0021 as the cross-cutting finalization spec.
+- **0.4 (2026-02-03)**: Updated for Workflow DevKit durable sessions + resumable streaming alignment.

docs/architecture/adr/ADR-0006-agent-runtime-ai-sdk-v6-toolloopagent-streaming-ui-responses.md

@@ -28,6 +28,10 @@ See [SPEC-0021](../spec/SPEC-0021-full-stack-finalization-fluid-compute-neon-ups
 for the cross-cutting “finalization” plan that specifies the workspace
 information architecture and the API contracts required by the UI.
 
+See [SPEC-0023](../spec/SPEC-0023-ai-elements-workspace-ui-and-interaction-model.md)
+for the UI information architecture, component-vendoring plan, and interaction
+model (chat/runs/search/uploads).
+
 ## Context
 
 Building chat and streaming UI primitives from scratch is slow and error-prone. AI Elements provides ready-made components designed for AI SDK message parts. Streamdown improves streaming markdown rendering. shadcn/ui provides accessible UI components aligned with Tailwind v4.
@@ -141,4 +145,4 @@ flowchart LR
 
 - **0.1 (2026-01-29)**: Initial version.
 - **0.2 (2026-01-30)**: Updated for current repo baseline (Bun, `src/` layout, CI).
-- **0.3 (2026-02-03)**: Linked to SPEC-0021 as the cross-cutting finalization spec.
+- **0.3 (2026-02-03)**: Linked to SPEC-0021 as the cross-cutting finalization spec; added SPEC-0023 reference for UI/interaction model.