cloud: scaffold multi-tenant build (Postgres adapter + register-cloud DI + k8s job executor stub + auth middleware + CI). See CLOUD.md punchlist.

Author: Ariel Shadkhan

.github/workflows/cloud-build.yml

@@ -0,0 +1,42 @@
+name: cloud-build
+on:
+  push:
+    branches: [cloud]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  packages: write
+
+jobs:
+  image:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: docker/setup-buildx-action@v3
+
+      - uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: meta
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: ghcr.io/shep-ai/shep
+          tags: |
+            type=sha,prefix=cloud-,format=short
+            type=raw,value=cloud-latest
+
+      - uses: docker/build-push-action@v6
+        with:
+          context: .
+          file: Dockerfile.cloud
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          cache-from: type=gha,scope=cloud
+          cache-to: type=gha,mode=max,scope=cloud

CLOUD.md

@@ -0,0 +1,78 @@
+# Shep Cloud — `cloud` branch
+
+Multi-tenant SaaS build of shep that ships at **app.shep.bot**. Same web UI you'd see at `localhost:4050`, but every user is logged in via Keycloak, every row is scoped to their org, and every "run agent" spawns an isolated k8s Job in the cluster instead of a local subprocess.
+
+This branch is **structural scaffolding, not yet a working build**. The skeleton lays down the swap points; the punchlist below is what's left.
+
+## Mental model
+
+The CLI build (`main`) is unchanged: SQLite, local subprocesses, single user. The cloud build is selected by `SHEP_MODE=cloud` at boot — that swaps:
+
+| Concern        | `main` (CLI)               | `cloud`                                                |
+|----------------|----------------------------|--------------------------------------------------------|
+| Persistence    | SQLite (`~/.shep/shep.db`) | Postgres on the cluster, RLS + `org_id` on every table |
+| Auth           | none (local user)          | Keycloak OIDC, NextAuth session, request-scoped tenant |
+| Agent execution| `child_process.spawn`      | k8s `batch/v1.Job` per run, ephemeral PVC, NetworkPol  |
+| Worktree FS    | `~/.shep/worktrees/...`    | Per-Job ephemeral `emptyDir` or PVC; git remote is SoT |
+| Settings store | `~/.shep/settings.json`    | Postgres `org_settings` table                          |
+
+**No fork from main:** all cloud-mode code is additive (new files under `*/postgres/`, `register-cloud.ts`, etc.). The only edits to existing files are conditional (`if (SHEP_MODE === 'cloud')`). Rebases from `main` should be clean.
+
+## What's scaffolded already
+
+- `packages/core/src/infrastructure/persistence/postgres/` — connection, tenant context, RLS-based migration baseline
+- `packages/core/src/infrastructure/repositories/postgres-feature.repository.ts` — proof-of-pattern; matches `IFeatureRepository`
+- `packages/core/src/infrastructure/di/modules/register-cloud.ts` — DI module that swaps adapters when `SHEP_MODE=cloud`
+- `packages/core/src/infrastructure/di/container.ts` — branches on `SHEP_MODE` to call `registerCloud()` instead of (some of) the SQLite registrations
+- `packages/core/src/infrastructure/services/agent-executor/k8s-job.executor.ts` — stub implementing `IInteractiveAgentExecutor`
+- `src/presentation/web/middleware.ts` — Keycloak gate, request-scoped tenant context
+- `.github/workflows/cloud-build.yml` — CI builds `ghcr.io/shep-ai/shep:cloud-<sha>` on push to this branch
+
+## Punchlist (in order of dependency)
+
+### Persistence layer
+- [ ] **Port the remaining ~30 repositories** to Postgres. The pattern is `postgres-feature.repository.ts` — copy, swap query syntax, rely on RLS for tenant filter.
+- [ ] **Settings repository** is the next gating one (the UI hits it on every page load). Then `repository-repository`, `application-repository`, the PM family.
+- [ ] **Migrations** — flesh out `0001_init.sql` to cover every table currently in SQLite (`migrations/sqlite/*.sql` is the source). Use a per-table `org_id uuid not null` column + RLS policy `org_id = current_setting('app.org_id')::uuid`.
+- [ ] **`org_id` propagation in domain code** — the domain entities (`Feature`, `Application`, etc.) don't carry `org_id`. We have two options: (a) thread it through the domain (invasive), (b) keep RLS-only and let Postgres enforce. Going with (b). Repos take tenant context from a request-scoped tsyringe child container.
+
+### Auth & tenancy
+- [ ] **Org/Member model** — currently no `orgs` or `org_members` tables. Add them in migrations. UI for create org / invite member.
+- [ ] **Keycloak token → org_id** — middleware reads JWT, looks up the user's default org, sets `app.org_id` on the connection. Multi-org users get an org switcher.
+- [ ] **Per-route guards** — every server action and API route currently assumes a single user. Add a higher-order wrapper that asserts `getServerSession()` and resolves tenant before the handler runs.
+
+### Agent execution
+- [ ] **k8s-job.executor.ts** is a stub. Real impl needs to:
+  - render a `batch/v1.Job` with the agent CLI (Claude Code / Cursor / Gemini) in the image
+  - mount a fresh `emptyDir` at `/workspace`, clone the repo (using a per-org GitHub deploy token from `org_secrets`)
+  - inject the org's AI provider API key via `secretRef`
+  - stream pod logs back via SSE — reuse the existing `agent-events` API route, swap its source
+  - report exit/result, post-process to update `agent_runs` row + open PR
+- [ ] **NetworkPolicy** scoped to the agent's namespace allowing egress only to `api.github.com` + the chosen AI provider host.
+- [ ] **Sandboxing tier** — for v1 use default runtime + restrictive securityContext. Migrate to gVisor (`runtimeClassName: gvisor`) once we have one untrusted real user.
+
+### Filesystem
+- [ ] **No local worktrees.** Every Job clones afresh. State lives in git. Future: persistent worktrees via per-org PVCs if performance requires.
+- [ ] **Settings/uploads/attachments** — currently disk-based. Move to MinIO or skip until needed.
+
+### Deployment
+- [ ] **shep-infra:** swap the placeholder Deployment at `app.shep.bot` to pull `ghcr.io/shep-ai/shep:cloud-<sha>`. Wire `SHEP_MODE=cloud`, `DATABASE_URL`, Keycloak env, k8s SA token.
+- [ ] **Run drizzle/postgres migrations** as initContainer (or built-in app boot, idempotent).
+- [ ] **Stop CLI bits** — at boot in cloud mode, don't initialize local SQLite, daemon, notification watcher, IDE integration, etc. `register-cloud.ts` should branch.
+
+### Hardening / nice-to-have
+- [ ] Org-secrets vault (encrypt-at-rest, ideally via Infisical when we wire ESO back on).
+- [ ] Rate-limit per org on agent runs (so one tenant can't burn the cluster).
+- [ ] Audit log on tenant-impacting actions.
+- [ ] Idle-org reaper that pauses long-unused orgs' resources.
+
+## How to work this branch
+
+1. Pick a repository from the punchlist.
+2. Copy `postgres-feature.repository.ts` to `postgres-<name>.repository.ts`. Implement the interface methods.
+3. Add the registration to `register-cloud.ts`.
+4. Add migrations for any new tables in `migrations/postgres/`.
+5. Run locally with `SHEP_MODE=cloud DATABASE_URL=postgres://... pnpm dev` against a local Postgres.
+6. Push. CI builds `ghcr.io/shep-ai/shep:cloud-<sha>`.
+
+Stay strict about additive-only changes to files that exist on `main`. Anything more is a recipe for merge hell.

Dockerfile.cloud

@@ -0,0 +1,37 @@
+# Shep Cloud image — built from the `cloud` branch.
+#
+# Differences from the existing CLI Dockerfile:
+#   - boots the Next.js web server in standalone mode (not the CLI)
+#   - sets SHEP_MODE=cloud so DI swaps to Postgres + k8s Job executor
+#   - migrations run at app boot
+FROM node:22-alpine AS deps
+RUN corepack enable && corepack prepare pnpm@9.15.0 --activate
+WORKDIR /app
+COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
+COPY src/presentation/web/package.json ./src/presentation/web/package.json
+COPY packages/core/package.json ./packages/core/package.json
+RUN pnpm install --frozen-lockfile
+
+FROM node:22-alpine AS builder
+RUN corepack enable && corepack prepare pnpm@9.15.0 --activate
+WORKDIR /app
+COPY --from=deps /app/node_modules ./node_modules
+COPY . .
+ENV NEXT_TELEMETRY_DISABLED=1
+RUN pnpm run build:release
+
+FROM node:22-alpine AS runtime
+RUN apk upgrade --no-cache && apk add --no-cache curl
+WORKDIR /app
+ENV NODE_ENV=production NEXT_TELEMETRY_DISABLED=1 SHEP_MODE=cloud
+RUN addgroup --system --gid 1001 nodejs && adduser --system --uid 1001 shep
+COPY --from=builder --chown=shep:nodejs /app/web ./web
+COPY --from=builder --chown=shep:nodejs /app/dist ./dist
+COPY --from=builder --chown=shep:nodejs /app/package.json ./
+USER shep
+EXPOSE 4050
+ENV HOSTNAME=0.0.0.0 PORT=4050
+# The shep CLI's `_serve` command boots the Next.js standalone server using
+# the same DI container that powers the CLI, just with SHEP_MODE=cloud the
+# repositories and executor are swapped out.
+CMD ["node", "dist/presentation/cli/index.js", "_serve", "--port", "4050"]

packages/core/src/infrastructure/di/modules/register-cloud.ts

@@ -0,0 +1,39 @@
+/**
+ * Cloud-mode DI registrations.
+ *
+ * Called from `container.ts` instead of (or in addition to) the SQLite/CLI
+ * registrations when `SHEP_MODE=cloud`.
+ *
+ * Strategy:
+ *   - Postgres-backed repositories replace SQLite ones (see CLOUD.md punchlist
+ *     for the full list to port).
+ *   - Agent execution swaps the local subprocess executor for a k8s Job one.
+ *   - The local daemon, notification watcher, IDE integration, etc. don't
+ *     register at all in cloud mode.
+ */
+
+import type { DependencyContainer } from 'tsyringe';
+import type { IFeatureRepository } from '../../../application/ports/output/repositories/feature-repository.interface.js';
+import { PostgresFeatureRepository } from '../../repositories/postgres-feature.repository.js';
+import type { IInteractiveAgentExecutor } from '../../../application/ports/output/agents/interactive-agent-executor.interface.js';
+import { K8sJobAgentExecutor } from '../../services/agent-executor/k8s-job.executor.js';
+
+export function registerCloud(container: DependencyContainer): void {
+  // Repositories — start with feature; remaining repos are punchlisted.
+  container.register<IFeatureRepository>('IFeatureRepository', {
+    useClass: PostgresFeatureRepository,
+  });
+
+  // Agent execution — k8s Job per run instead of local subprocess.
+  container.register<IInteractiveAgentExecutor>('IInteractiveAgentExecutor', {
+    useClass: K8sJobAgentExecutor,
+  });
+
+  // TODO(CLOUD.md): once 30 repos are ported, remove the corresponding
+  // sqlite registrations from registerRepositories rather than letting
+  // them be silently overridden.
+}
+
+export function isCloudMode(): boolean {
+  return process.env.SHEP_MODE === 'cloud';
+}

packages/core/src/infrastructure/persistence/postgres/connection.ts

@@ -0,0 +1,31 @@
+/**
+ * Postgres Connection
+ *
+ * Single connection pool for cloud-mode shep. Equivalent of `sqlite/connection.ts`
+ * but multi-tenant: each request must `SET app.org_id` (see tenant-context.ts)
+ * before running queries — RLS policies in the schema do the actual filtering.
+ *
+ * Lazy-initialized; reads `DATABASE_URL` from env on first use.
+ */
+
+import postgres, { type Sql } from 'postgres';
+
+let _sql: Sql | null = null;
+
+export function getPostgresConnection(): Sql {
+  if (_sql) return _sql;
+  const url = process.env.DATABASE_URL;
+  if (!url) throw new Error('cloud mode: DATABASE_URL is required');
+  _sql = postgres(url, {
+    max: Number(process.env.PG_POOL_MAX ?? 10),
+    idle_timeout: 20,
+    prepare: false, // postgres-js + Node 22 + cluster pool: prepared statements break under churn
+  });
+  return _sql;
+}
+
+export async function closePostgresConnection(): Promise<void> {
+  if (!_sql) return;
+  await _sql.end();
+  _sql = null;
+}

packages/core/src/infrastructure/persistence/postgres/migrations.ts

@@ -0,0 +1,45 @@
+/**
+ * Postgres migrations runner.
+ *
+ * Reads SQL files from ./migrations/, applies any not yet recorded in the
+ * `_migrations` table. Idempotent — safe to run on every boot.
+ */
+
+import { readFile, readdir } from 'node:fs/promises';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { withoutTenant } from './tenant-context.js';
+
+const MIGRATIONS_DIR = path.resolve(
+  fileURLToPath(import.meta.url),
+  '..',
+  'migrations'
+);
+
+export async function runPostgresMigrations(): Promise<void> {
+  await withoutTenant(async (sql) => {
+    await sql`
+      create table if not exists _migrations (
+        name        text primary key,
+        applied_at  timestamptz not null default now()
+      )
+    `;
+
+    const files = (await readdir(MIGRATIONS_DIR))
+      .filter((f) => f.endsWith('.sql'))
+      .sort();
+
+    for (const f of files) {
+      const applied = await sql`select 1 from _migrations where name = ${f}`;
+      if (applied.length > 0) continue;
+
+      const body = await readFile(path.join(MIGRATIONS_DIR, f), 'utf8');
+      // postgres-js's .unsafe() runs raw SQL (multi-statement OK) — required
+      // for migration files containing multiple `create table` etc.
+      await sql.unsafe(body);
+      await sql`insert into _migrations (name) values (${f})`;
+      // eslint-disable-next-line no-console
+      console.log(`[pg] applied migration ${f}`);
+    }
+  });
+}

packages/core/src/infrastructure/persistence/postgres/migrations/0001_init.sql

@@ -0,0 +1,70 @@
+-- Cloud-mode initial schema baseline.
+--
+-- Goal: tenant-isolated copies of the SQLite schema in shep main, with RLS
+-- enforcing org_id filtering at the database layer.
+--
+-- This is a baseline. Subsequent migrations add the rest of shep's tables
+-- (every SQLite table needs an org_id column + RLS policy here).
+
+create extension if not exists "pgcrypto";
+
+create table if not exists orgs (
+  id          uuid primary key default gen_random_uuid(),
+  slug        text not null unique,
+  name        text not null,
+  created_at  timestamptz not null default now()
+);
+
+create table if not exists users (
+  id          text primary key,                         -- Keycloak `sub`
+  email       text not null unique,
+  name        text,
+  created_at  timestamptz not null default now()
+);
+
+create table if not exists org_members (
+  org_id      uuid not null references orgs(id) on delete cascade,
+  user_id     text not null references users(id) on delete cascade,
+  role        text not null check (role in ('owner','admin','member')),
+  joined_at   timestamptz not null default now(),
+  primary key (org_id, user_id)
+);
+
+-- ---------------------------------------------------------------------------
+-- features (proof-of-pattern; mirrors `features` from sqlite migrations)
+-- ---------------------------------------------------------------------------
+create table if not exists features (
+  id              uuid primary key default gen_random_uuid(),
+  org_id          uuid not null references orgs(id) on delete cascade,
+  repository_path text not null,
+  slug            text not null,
+  title           text not null,
+  description     text,
+  branch          text,
+  lifecycle       text not null,
+  status          text,
+  pr_url          text,
+  metadata        jsonb,
+  created_by      text references users(id),
+  created_at      timestamptz not null default now(),
+  updated_at      timestamptz not null default now(),
+  archived_at     timestamptz,
+  deleted_at      timestamptz,
+  unique (org_id, repository_path, slug)
+);
+
+create index if not exists features_org_lifecycle_idx
+  on features (org_id, lifecycle) where deleted_at is null;
+
+-- ---------------------------------------------------------------------------
+-- RLS — every tenant table follows this exact pattern.
+-- ---------------------------------------------------------------------------
+alter table features enable row level security;
+
+create policy features_tenant_isolation on features
+  using  (org_id = current_setting('app.org_id', true)::uuid)
+  with check (org_id = current_setting('app.org_id', true)::uuid);
+
+-- Helper for migrations: bypass RLS as table owner. App connections never
+-- get the bypass role.
+-- (App connects as a less-privileged role created in a later migration.)

packages/core/src/infrastructure/persistence/postgres/tenant-context.ts

@@ -0,0 +1,43 @@
+/**
+ * Tenant Context (cloud mode)
+ *
+ * Every Postgres query in cloud mode must carry an org_id. We propagate it via
+ * Postgres session settings + RLS, so repository code stays single-tenant
+ * shaped and the DB enforces isolation.
+ *
+ * Pattern:
+ *   await withTenant(orgId, async (sql) => {
+ *     return sql<Feature[]>`SELECT * FROM features WHERE id = ${id}`;
+ *   })
+ *
+ * Inside the callback every query runs in a transaction with
+ * `SET LOCAL app.org_id = ...`, which the RLS policy reads via
+ * `current_setting('app.org_id')::uuid`.
+ *
+ * Server actions and API routes resolve the org from the Keycloak session
+ * (see src/presentation/web/middleware.ts) and call `withTenant`.
+ */
+
+import type { Sql, TransactionSql } from 'postgres';
+import { getPostgresConnection } from './connection.js';
+
+export async function withTenant<T>(
+  orgId: string,
+  fn: (sql: TransactionSql) => Promise<T>
+): Promise<T> {
+  const sql = getPostgresConnection();
+  return sql.begin(async (tx) => {
+    await tx`SELECT set_config('app.org_id', ${orgId}, true)`;
+    return fn(tx);
+  });
+}
+
+/**
+ * For admin-only operations (migrations, org provisioning) that need to bypass
+ * RLS. Use sparingly — anything user-facing must go through `withTenant`.
+ */
+export async function withoutTenant<T>(
+  fn: (sql: Sql) => Promise<T>
+): Promise<T> {
+  return fn(getPostgresConnection());
+}