Fix prod schema drift: pgvector dim, missing escrow/portfolio columns, agent_ratings types

Audit found that several columns declared in 0002_fresh_george_stacy.sql
never landed on prod because that migration uses CREATE TABLE IF NOT EXISTS
for tables already created by 0000. Side effects observed:

- escrow_transactions was missing retry_count/last_error in prod, so the
  Phase-2 on-chain-failure rollback in releaseEscrow tried to UPDATE
  nonexistent columns and itself errored, leaving escrows stuck at
  RELEASING with no retry metadata.
- portfolio_items was missing category, completion_time, requester_rating,
  is_pinned, display_order — surfaces silently coerced to NULL.
- agent_ratings score columns were integer in prod, real in schema. The
  weighted overallScore = q*0.3 + s*0.2 + ... computation got truncated
  on every rating insert.
- pgvector dim mismatch: nomic-embed-text-v1.5 produces 768-dim vectors
  but schema/code declared vector(1536). Embedding writes are wrapped in
  .catch(console.error), so every embed silently 500'd, leaving every
  description_embedding NULL and degrading invitation matching to a no-op.

Changes:
- packages/api/src/services/embeddings.ts: drop padTo1536, return 768
  native; export EMBEDDING_DIM constant.
- packages/api/src/db/schema.ts: vector(1536) → vector(768) on agents,
  agent_skills, tasks.
- packages/api/drizzle/0005_next_dragon_lord.sql: corrective migration.
  ADD COLUMN IF NOT EXISTS for the missing escrow/portfolio columns,
  ALTER COLUMN ... USING ::real for the score widenings, DROP+ADD for
  the vector dim change (pgvector cannot ALTER dimensions in place).
  Idempotent against fresh databases that already have the columns.
- CLAUDE.md: rewrite the schema-change workflow with the lessons —
  review generated SQL for IF NOT EXISTS pitfalls, audit prod drift
  before assuming migrations match prod state, document destructive-op
  guards and the integration-test gate. Add Observability section
  covering traceOp + initTelemetry wiring.

Verified: tracked migrations 0000-0005 apply cleanly to a fresh DB
(escrow_transactions has retry_count/last_error, vectors are 768,
agent_ratings scores are real). All 130 unit tests + 8 integration
tests green post-migration.

Author: waydelyle

CLAUDE.md

@@ -30,25 +30,45 @@ Dashboard: http://localhost:3200
 
 ## Database
 
-PostgreSQL 16 with pgvector. Schema defined in `packages/api/src/db/schema.ts` using Drizzle ORM.
+PostgreSQL 16 with pgvector. Schema defined in `packages/api/src/db/schema.ts` using Drizzle ORM. Embeddings use the nomic-embed-text-v1.5 model — pgvector columns are `vector(768)` and must match exactly.
 
-Core tables: `agents`, `agent_skills`, `tasks`, `task_bids`, `escrow_transactions`, `agent_ratings`, `challenges`, `agent_wallets`, `anomaly_events`, `disputes`, `transactions`, `audit_log`.
+Core tables: `agents`, `agent_skills`, `tasks`, `task_bids`, `escrow_transactions`, `agent_ratings`, `challenges`, `agent_wallets`, `anomaly_events`, `disputes`, `transactions`, `audit_log`, `event_outbox`, `agent_messages`, `agent_reputation`, `portfolio_items`, `task_invitations`.
 
-v2 tables: `quality_evaluations`, `quality_metrics`, `agent_activity`, `agent_endorsements`, `agent_following`, `agent_guilds`, `guild_members`, `mcp_services`, `mcp_tool_calls`, `mcp_subscriptions`.
+v2 tables: `quality_evaluations`, `quality_metrics`, `agent_activity`, `agent_endorsements`, `agent_following`, `agent_guilds`, `guild_members`.
 
 Drizzle commands:
-- `pnpm db:generate` — generate SQL migration from schema changes
-- `pnpm db:migrate` — apply pending tracked migrations
-- `pnpm db:push` — push schema directly (dev only, no migration file)
-- `pnpm db:studio` — open Drizzle Studio
-
-**Schema change workflow:**
-1. Edit `packages/api/src/db/schema.ts`
-2. Run `pnpm --filter @swarmdock/api db:generate` to create a SQL migration in `packages/api/drizzle/`
-3. Review the generated SQL, then commit it alongside the schema change
-4. On deploy, `start.sh` runs the tracked migrations automatically via `db:migrate`
-
-**IMPORTANT: Migrations auto-deploy.** The Dockerfile copies `packages/api/drizzle/` into the production image and `start.sh` runs `db:migrate` on every container start. Any new migration files pushed to `main` will be applied to production on next deploy. Always verify schema changes compile (`pnpm type-check`) and review generated SQL before pushing.
+- `pnpm --filter @swarmdock/api db:generate` — generate SQL migration from schema changes
+- `pnpm --filter @swarmdock/api db:migrate` — apply pending tracked migrations
+- `pnpm --filter @swarmdock/api db:push` — push schema directly (dev/test, no migration file)
+- `pnpm --filter @swarmdock/api db:studio` — open Drizzle Studio
+
+### Schema change workflow
+
+1. Edit `packages/api/src/db/schema.ts`.
+2. Run `pnpm --filter @swarmdock/api db:generate` to create a SQL file in `packages/api/drizzle/`.
+3. **Review the generated SQL.** drizzle-kit's defaults can be wrong:
+   - It emits `CREATE TABLE IF NOT EXISTS` for new tables. If a same-named table already exists in any environment, the new columns silently do not apply — replace with `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` for any column added to an existing table.
+   - `ALTER COLUMN ... SET DATA TYPE vector(N)` between different dimensions is a no-op or errors in pgvector. Replace with `DROP COLUMN IF EXISTS` + `ADD COLUMN ... vector(N)` (data loss is real — guard or backfill).
+   - Type widenings (e.g. integer → real) need an explicit `USING <expr>::real` cast or Postgres rejects the change.
+4. **Audit prod drift before assuming migrations match prod state.** drizzle generates from the snapshot in `drizzle/meta/`, which is whatever the schema looked like at the previous generate — not what is actually in the prod DB. To check, dump prod and a fresh schema-derived DB, then `comm -23` the column lists:
+   ```bash
+   render psql swarmdock-db -c "COPY (SELECT table_name||'|'||column_name||'|'||udt_name FROM information_schema.columns WHERE table_schema='public' ORDER BY 1,2) TO STDOUT" > /tmp/prod-cols.txt
+   docker compose exec -T postgres psql -U swarmdock -d schema_check -c "..."  # same query
+   diff <(sort /tmp/prod-cols.txt) <(sort /tmp/schema-cols.txt)
+   ```
+   If prod is missing columns the schema expects, hand-write a corrective `ALTER TABLE` migration before shipping any new feature that depends on them.
+5. Verify the migration end-to-end against a fresh DB: `DROP DATABASE`, run `db:migrate` on it, then run integration tests.
+6. Commit the migration alongside the schema change.
+
+### Production deploy
+
+**Migrations auto-apply on every container restart.** The Dockerfile copies `packages/api/drizzle/` into the image and `start.sh` runs `db:migrate` before booting the API. Any new migration file merged to `main` will execute against prod within minutes of the next Render deploy. Before pushing:
+
+- `pnpm type-check` and `pnpm --filter @swarmdock/api test` must be green.
+- `pnpm --filter @swarmdock/api test:integration` must be green (real-Postgres exercise of the schema and escrow state machine).
+- Migration is reviewed for destructive operations (`DROP COLUMN`, `ALTER TYPE` without `USING`, `TRUNCATE`).
+
+If a migration needs to backfill or run in a long-running step, it must be split: ship the schema change first (idempotent, additive), backfill from a worker or one-off script, then ship the constraint/cleanup in a follow-up migration.
 
 ## Code Conventions
 
@@ -82,10 +102,34 @@ open → bidding → assigned → in_progress → review → completed
 # Type check all packages
 pnpm type-check
 
-# Test API with curl
+# Lint
+pnpm lint
+
+# Unit tests (fast, no DB)
+pnpm --filter @swarmdock/api test
+pnpm --filter @swarmdock/sdk test
+pnpm --filter @swarmdock/shared test
+pnpm --filter @swarmdock/cli test
+
+# Real-Postgres integration tests (escrow state machine, FOR UPDATE locks,
+# 3-phase commit). Creates `swarmdock_test` DB and applies the live schema
+# via drizzle-kit push; truncates between tests. Requires `docker compose
+# up -d postgres`.
+pnpm --filter @swarmdock/api test:integration
+
+# Smoke test API
 curl http://localhost:3100/api/v1/health
 ```
 
+## Observability
+
+OpenTelemetry traces are wired via `packages/api/src/lib/telemetry.ts`:
+
+- HTTP middleware spans on every request (`packages/api/src/middleware/otel.ts`).
+- `traceOp(name, attrs, fn)` helper wraps high-value business ops: `escrow.fund`, `escrow.release`, `escrow.refund`, `identity.issueAAT`, `quality.verify`. Add new spans by importing `traceOp` and wrapping the function body.
+- Both `index.ts` and `worker.ts` call `initTelemetry()` at the very top before any other import — required for the auto-instrumentation to monkey-patch correctly.
+- Export is gated on `OTEL_EXPORTER_OTLP_ENDPOINT`. Local default points at the `jaeger` service in docker-compose; UI at http://localhost:16686. Production env var is currently unset, so spans are dropped — set it on the Render web + worker services to enable.
+
 ## Related Projects
 
 - **SwarmClaw** (`../swarmclaw`) — Agent runtime and control plane. See `SWARMDOCK.md` for integration.

packages/api/drizzle/0005_next_dragon_lord.sql

@@ -0,0 +1,61 @@
+-- 0005: corrective migration to align prod with schema.ts.
+--
+-- Background: migrations 0000–0004 declared the canonical schema, but
+-- 0002 used `CREATE TABLE IF NOT EXISTS` for tables that already existed
+-- (created by 0000). That silently skipped the new columns and types.
+-- This migration brings any environment whose schema followed the tracked
+-- migration path back into alignment with schema.ts.
+--
+-- Idempotent: every statement uses IF EXISTS / IF NOT EXISTS so it is safe
+-- to re-run and safe against fresh databases that already have the columns.
+
+-- escrow_transactions: missing retry/error tracking (escrow.ts:209-210
+-- writes these on Phase-2 on-chain failure; without them the rollback path
+-- itself errored out, leaving escrows stuck at RELEASING).
+ALTER TABLE "escrow_transactions" ADD COLUMN IF NOT EXISTS "retry_count" integer DEFAULT 0 NOT NULL;
+--> statement-breakpoint
+ALTER TABLE "escrow_transactions" ADD COLUMN IF NOT EXISTS "last_error" text;
+--> statement-breakpoint
+
+-- portfolio_items: schema added category, completion_time, requester_rating,
+-- is_pinned, display_order in 0002. Prod is missing them all.
+ALTER TABLE "portfolio_items" ADD COLUMN IF NOT EXISTS "category" text;
+--> statement-breakpoint
+ALTER TABLE "portfolio_items" ADD COLUMN IF NOT EXISTS "completion_time" text;
+--> statement-breakpoint
+ALTER TABLE "portfolio_items" ADD COLUMN IF NOT EXISTS "requester_rating" real;
+--> statement-breakpoint
+ALTER TABLE "portfolio_items" ADD COLUMN IF NOT EXISTS "is_pinned" boolean DEFAULT false NOT NULL;
+--> statement-breakpoint
+ALTER TABLE "portfolio_items" ADD COLUMN IF NOT EXISTS "display_order" integer;
+--> statement-breakpoint
+
+-- agent_ratings: 0002 widened the score columns from integer to real so
+-- weighted-average overall scores stop truncating. Cast preserves existing
+-- values (any 1-5 int rounds correctly to a float).
+ALTER TABLE "agent_ratings" ALTER COLUMN "quality_score" SET DATA TYPE real USING "quality_score"::real;
+--> statement-breakpoint
+ALTER TABLE "agent_ratings" ALTER COLUMN "speed_score" SET DATA TYPE real USING "speed_score"::real;
+--> statement-breakpoint
+ALTER TABLE "agent_ratings" ALTER COLUMN "communication_score" SET DATA TYPE real USING "communication_score"::real;
+--> statement-breakpoint
+ALTER TABLE "agent_ratings" ALTER COLUMN "reliability_score" SET DATA TYPE real USING "reliability_score"::real;
+--> statement-breakpoint
+
+-- pgvector dim alignment: nomic-embed-text-v1.5 produces 768-dim vectors,
+-- but the schema (and historical migrations) declared vector(1536), so
+-- every embedding insert silently 500'd in the .catch(console.error)
+-- path. Drop + re-add since pgvector cannot ALTER the dimension of an
+-- existing column. No data loss in practice — embedding writes have been
+-- failing for the lifetime of the bug, so columns are uniformly NULL.
+ALTER TABLE "agents" DROP COLUMN IF EXISTS "description_embedding";
+--> statement-breakpoint
+ALTER TABLE "agents" ADD COLUMN "description_embedding" vector(768);
+--> statement-breakpoint
+ALTER TABLE "agent_skills" DROP COLUMN IF EXISTS "skill_embedding";
+--> statement-breakpoint
+ALTER TABLE "agent_skills" ADD COLUMN "skill_embedding" vector(768);
+--> statement-breakpoint
+ALTER TABLE "tasks" DROP COLUMN IF EXISTS "description_embedding";
+--> statement-breakpoint
+ALTER TABLE "tasks" ADD COLUMN "description_embedding" vector(768);