Merge pull request #1 from SSheppDev/feat/multi-org-schemas

prepare 1.1.0 multi-org per-schema release

Author: SSheppDev

CLAUDE.md

@@ -105,18 +105,22 @@ sf-db/
 - All database queries go through the pg pool — never create ad-hoc connections
 
 ### Postgres
-- Synced Salesforce data → `salesforce` schema
 - Internal app tables → `sfdb` schema
+- Synced Salesforce data → one schema per registered org named `org_<lowercased orgid>` (e.g. `org_00d5g000001abcdeaa`)
+- All `sfdb.*` per-object/per-field tables (`sync_config`, `field_config`, `field_metadata`, `sync_log`, `sync_lock`) are keyed by `(org_id, ...)` with `ON DELETE CASCADE` from `sfdb.orgs`
+- The active UI/sync context is stored in `sfdb.active_org` (single row); the API resolves it from `X-Org-Id` request header first, falling back to that pointer
 - Every synced table must have: `id`, `sf_created_at`, `sf_updated_at`, `sf_deleted_at`, `synced_at`
 - Field names are lowercase snake_case versions of SF API names
-- DDL is always idempotent (`IF NOT EXISTS` / `IF EXISTS`)
+- DDL is always idempotent (`IF NOT EXISTS` / `IF EXISTS`); identifiers are always quoted (objects like `Order` / `User` collide with PG reserved words)
 
 ### Sync engine
-- Always acquire `sfdb.sync_lock` before running any sync
-- Always release the lock in a `finally` block — never leave it held on error
+- Every sync entry point takes `orgId` as its primary key; alias is only used to look up an `~/.sfdx` token via `sfdb.orgs`
+- `sfdb.sync_lock` is per-org (one row per registered org). Acquire before any sync; always release in a `finally` block
+- Different orgs sync in parallel; one sync per org is serialized via that org's lock
 - If `last_delta_sync` is NULL → initial full load (no SystemModstamp WHERE clause)
 - Stale lock threshold: 30 minutes
 - Log purge runs at the start of every sync (delete rows older than `LOG_RETENTION_DAYS`)
+- The cron scheduler runs as one process with two ticks (delta per minute, full daily 02:00) that iterate every registered org
 
 ### API
 - All routes under `/api/` prefix
@@ -151,9 +155,10 @@ All runtime config (active org alias, sync intervals, enabled objects/fields) li
 
 ## Key Design Decisions (do not revisit without good reason)
 
-- **sf CLI binary is NOT in the Docker image.** Auth tokens are read directly from the `~/.sf/` JSON files mounted into the container. No `sf org display` command.
+- **sf CLI binary is NOT in the Docker image.** Auth tokens are read directly from the `~/.sfdx/` JSON files mounted into the container. No `sf org display` command.
 - **The API is not a data API.** It serves the UI and orchestrates syncs only. External tools connect directly to Postgres.
 - **Deletions are soft.** `sf_deleted_at` is set — records are never hard-deleted from the local DB.
 - **Bulk API 2.0 by default.** REST query fallback only for objects under 2,000 records.
-- **Config in DB, not `.env`.** `.env` is infrastructure only. Org alias, object selection, field selection, and schedule config all live in `sfdb.app_config` / `sfdb.sync_config` / `sfdb.field_config`.
-- **One active org at a time.** Multi-org simultaneous sync is out of scope for v1.
+- **Config in DB, not `.env`.** `.env` is infrastructure only. Org registry, object selection, field selection, and schedule config all live in `sfdb.orgs` / `sfdb.sync_config` / `sfdb.field_config` / `sfdb.app_config`.
+- **Multi-org by schema.** Every registered org gets its own `org_<orgid>` schema. Removing an org drops the schema and cascades through `sfdb.*` via the FKs on `sfdb.orgs(org_id)`.
+- **Schema name is derived from the immutable Salesforce org id**, not the user-editable alias — aliases can be renamed without affecting where the data lives.

README.md

@@ -31,21 +31,26 @@ A self-hosted Salesforce-to-PostgreSQL sync pipeline. Run it with `docker compos
 # 1. Authenticate a Salesforce org (skip if already done)
 sf org login web --alias my-org
 
-# 2. Configure environment
+# 2. Export decrypted Salesforce tokens for Docker to use
+npm run export-tokens
+
+# 3. Configure environment
 cp .env.example .env
 # Edit .env — set POSTGRES_PASSWORD at minimum
 
-# 3. Start
+# 4. Start
 docker compose up -d
 
-# 4. Open the UI
+# 5. Open the UI
 open http://localhost:7743
 ```
 
 First start takes ~30 seconds while Postgres initializes and the API container builds.
 
 The onboarding screen will detect your authenticated orgs and ask you to pick one. After that, go to the Objects page and enable the Salesforce objects you want to sync.
 
+`npm run export-tokens` writes plaintext access tokens to `data/tokens.json` so the Docker container can authenticate to Salesforce. This file is local-only secret material, is git-ignored, and should never be committed or shared.
+
 ## Connect a BI tool or SQL client
 
 Once data is syncing, connect any Postgres-compatible tool directly:
@@ -54,12 +59,12 @@ Once data is syncing, connect any Postgres-compatible tool directly:
 |----------|-----------------------|
 | Host     | `localhost`           |
 | Port     | `7745`                |
-| Database | `sfdb`               |
-| Schema   | `salesforce`          |
-| User     | `sfdb`               |
+| Database | `sfdb`                |
+| Schema   | `org_<orgid>`         |
+| User     | `sfdb`                |
 | Password | *(your `.env` value)* |
 
-The Settings page in the UI shows a copyable connection string.
+Each registered Salesforce org gets its own schema named `org_<lowercased 18-char Salesforce org id>`. The Settings page in the UI shows the schema name for every registered org and a copyable connection string.
 
 A read-only role is also available — set `READONLY_PASSWORD` in `.env` and connect as user `sfdb_readonly`.
 
@@ -103,11 +108,11 @@ Queries `SELECT Id FROM <Object>` for the full live ID set, diffs against local
 
 ### Concurrency
 
-Only one sync runs at a time. A single-row lock table (`sfdb.sync_lock`) prevents overlap. Stale locks (> 30 min) are automatically reclaimed on startup.
+Sync is serialized per org via `sfdb.sync_lock`, with one lock row per registered org. Different orgs can sync in parallel; overlapping syncs for the same org are blocked. Stale locks (> 30 min) are automatically reclaimed on startup.
 
 ## Database schema
 
-**`salesforce` schema** — one table per enabled Salesforce object, e.g. `salesforce.account`
+**One schema per registered org** — named `org_<lowercased orgid>`, one table per enabled Salesforce object (e.g. `org_00d5g000001abcdeaa.account`)
 
 | Column | Type | Notes |
 |---|---|---|
@@ -118,7 +123,7 @@ Only one sync runs at a time. A single-row lock table (`sfdb.sync_lock`) prevent
 | `sf_deleted_at` | `timestamptz NULL` | NULL = live; set when deletion detected |
 | `synced_at` | `timestamptz` | Last written by this tool |
 
-**`sfdb` schema** — internal app tables (sync config, logs, lock, field metadata)
+**`sfdb` schema** — internal app tables (`orgs` registry, `active_org` pointer, sync config, logs, per-org lock, field metadata). Per-object tables are keyed by `(org_id, ...)`.
 
 ## Tech stack
 
@@ -127,7 +132,7 @@ Only one sync runs at a time. A single-row lock table (`sfdb.sync_lock`) prevent
 | Database | PostgreSQL 16 |
 | Backend | Node.js + TypeScript + Express |
 | Frontend | React + TypeScript + shadcn/ui + Tailwind |
-| Salesforce auth | `~/.sfdx` files read directly via Node `fs` (no `sf` binary in container) |
+| Salesforce auth | `~/.sfdx` files read directly via Node `fs` (no `sf` binary in container) — multiple orgs supported, each gets its own Postgres schema |
 | Salesforce data | jsforce + Bulk API 2.0 |
 | Scheduling | node-cron |
 | Containers | Docker + Docker Compose |

docker-compose.yml

@@ -36,6 +36,10 @@ services:
     environment:
       POSTGRES_HOST: postgres
       POSTGRES_PORT_INTERNAL: 5432
+      # Cap V8 old-space below the cgroup limit so the GC can recover before the
+      # OS kills the process. Large bulk-API result pages (700k+ records) can
+      # otherwise blow past the default heap before streaming releases memory.
+      NODE_OPTIONS: --max-old-space-size=1536
     ports:
       - "127.0.0.1:${APP_PORT:-7743}:7743"
     volumes:
@@ -49,7 +53,7 @@ services:
     deploy:
       resources:
         limits:
-          memory: 512m
+          memory: 2g
           cpus: '1.0'
 
 networks:

docs/first-run.md

@@ -17,7 +17,15 @@ Verify it worked:
 sf org list
 ```
 
-## 2. Configure environment
+## 2. Export decrypted Salesforce tokens for Docker
+
+```bash
+npm run export-tokens
+```
+
+This writes `data/tokens.json`, a local-only secret file consumed by the API container. It is git-ignored and should never be committed.
+
+## 3. Configure environment
 
 ```bash
 cp .env.example .env
@@ -27,7 +35,7 @@ Edit `.env` if you need to change ports or the DB password. Defaults:
 - UI + API: `http://localhost:7743`
 - PostgreSQL: `localhost:7745`
 
-## 3. Start the app
+## 4. Start the app
 
 ```bash
 docker compose up -d
@@ -41,15 +49,15 @@ docker compose ps
 docker compose logs -f api
 ```
 
-## 4. Open the UI
+## 5. Open the UI
 
 ```
 http://localhost:7743
 ```
 
 The onboarding screen will detect your authenticated orgs and ask you to pick one.
 
-## 5. Connect your BI tool / SQL client
+## 6. Connect your BI tool / SQL client
 
 Once data is syncing, connect directly to Postgres:
 
@@ -58,9 +66,9 @@ Once data is syncing, connect directly to Postgres:
 | Host | `localhost` |
 | Port | `7745` (or `$POSTGRES_PORT` from `.env`) |
 | Database | `sfdb` |
-| Schema | `salesforce` |
+| Schema | `org_<orgid>` |
 | User | `sfdb` (or `$POSTGRES_USER`) |
-| Password | `changeme` (or `$POSTGRES_PASSWORD`) |
+| Password | your `.env` `POSTGRES_PASSWORD` value |
 
 The Settings page in the UI shows a copyable connection string.