feat: auto-indexing, custom indexes, CA cert injection, and auth token fix (#4)

* fix: restore execute bit on 004_readonly_user.sh init script

macOS Git strips execute permissions on checkout, causing Postgres
container init to fail with "bad interpreter: Permission denied" on
first boot. Records the bit in the index so clones on Mac work correctly.

Bumps patch version to 1.1.1.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: inject US Signal corporate CA cert into Docker image

The corporate SSL inspection proxy re-signs outbound HTTPS traffic with
the internal US Signal CA chain, causing Node.js inside the container to
reject Salesforce API calls with UNABLE_TO_GET_ISSUER_CERT. Bakes the
full three-cert chain (edge trust → issue CA → root CA) into the image
via update-ca-certificates and points NODE_EXTRA_CA_CERTS at it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(db): auto-index reference fields and add operator-defined custom indexes

Two indexing features:
1. Any field with sfType 'reference' is automatically indexed at table creation
   and when a column is added. Index names are derived from object+field names
   and hashed if they exceed Postgres's 63-char identifier limit.

2. sfdb.custom_indexes table stores operator-defined indexes (composite, partial,
   or any arbitrary expression) that are applied at startup and whenever a table
   is created or recreated. CRUD exposed at /api/indexes.

All index creation uses CREATE INDEX CONCURRENTLY IF NOT EXISTS — no table locks,
idempotent, safe to run against a live database.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: document reference-field auto-indexing and custom indexes feature

- README: add Indexing section with curl examples for the /api/indexes API
- docs/scope.md: update DDL table, add sfdb.custom_indexes to schema reference,
  add Indexing section with API reference and behavior description
- CLAUDE.md: add both indexing behaviors to Key Design Decisions

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(ui): add custom indexes card to Settings page

Read-only table showing all registered custom indexes for the active org
with a per-row delete button. Fetches on mount and on org switch.
Also fixes GET /api/indexes to return camelCase keys consistent with
other API routes.

Co-Authored-By: Claude <noreply@anthropic.com>

* chore(build): sync package-lock.json version to 1.1.1

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(auth): add --verbose flag to sf org display to get real access tokens

sf org display --json redacts the accessToken unless --verbose is passed.
Without it, tokens.json was storing the literal "[REDACTED]..." string,
causing Bulk API 401 INVALID_AUTH_HEADER errors.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(auth): use sf org auth show-access-token to get real access tokens

sf org display --json now permanently redacts accessToken regardless of
flags, pointing to the new dedicated command. Switch to a two-call approach:
sf org display for instanceUrl/username metadata, then sf org auth
show-access-token for the actual token value.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Seth Sheppard <Seth.Sheppard@setshe-mbp-1.rvp.local>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: 3 people

CLAUDE.md

@@ -107,7 +107,7 @@ sf-db/
 ### Postgres
 - Internal app tables → `sfdb` schema
 - Synced Salesforce data → one schema per registered org named `org_<lowercased orgid>` (e.g. `org_00d5g000001abcdeaa`)
-- All `sfdb.*` per-object/per-field tables (`sync_config`, `field_config`, `field_metadata`, `sync_log`, `sync_lock`) are keyed by `(org_id, ...)` with `ON DELETE CASCADE` from `sfdb.orgs`
+- All `sfdb.*` per-object/per-field tables (`sync_config`, `field_config`, `field_metadata`, `sync_log`, `sync_lock`, `custom_indexes`) are keyed by `(org_id, ...)` with `ON DELETE CASCADE` from `sfdb.orgs`
 - The active UI/sync context is stored in `sfdb.active_org` (single row); the API resolves it from `X-Org-Id` request header first, falling back to that pointer
 - Every synced table must have: `id`, `sf_created_at`, `sf_updated_at`, `sf_deleted_at`, `synced_at`
 - Field names are lowercase snake_case versions of SF API names
@@ -162,3 +162,5 @@ All runtime config (active org alias, sync intervals, enabled objects/fields) li
 - **Config in DB, not `.env`.** `.env` is infrastructure only. Org registry, object selection, field selection, and schedule config all live in `sfdb.orgs` / `sfdb.sync_config` / `sfdb.field_config` / `sfdb.app_config`.
 - **Multi-org by schema.** Every registered org gets its own `org_<orgid>` schema. Removing an org drops the schema and cascades through `sfdb.*` via the FKs on `sfdb.orgs(org_id)`.
 - **Schema name is derived from the immutable Salesforce org id**, not the user-editable alias — aliases can be renamed without affecting where the data lives.
+- **Reference fields are auto-indexed.** `createObjectTable` and `addColumn` automatically issue `CREATE INDEX CONCURRENTLY IF NOT EXISTS` for every `sfType === 'reference'` column. Index names are `idx_ref_<object>_<field>`, md5-hashed if > 63 chars.
+- **Custom indexes live in `sfdb.custom_indexes`.** Operator-defined indexes (composite, partial, etc.) are registered via `POST /api/indexes` and re-applied at startup and on table creation. Deleting a registration does not drop the underlying PG index.

README.md

@@ -110,6 +110,47 @@ Queries `SELECT Id FROM <Object>` for the full live ID set, diffs against local
 
 Sync is serialized per org via `sfdb.sync_lock`, with one lock row per registered org. Different orgs can sync in parallel; overlapping syncs for the same org are blocked. Stale locks (> 30 min) are automatically reclaimed on startup.
 
+## Indexing
+
+sfetch maintains two layers of indexes on every synced table.
+
+### Automatic — reference fields
+
+Any Salesforce field of type `reference` (lookup / master-detail) is indexed automatically when the column is created. No configuration needed — sfetch infers this from the field type.
+
+### Custom indexes
+
+For queries with composite filters, date ranges, or partial index predicates, register an index via the API:
+
+```bash
+curl -s -X POST http://localhost:7743/api/indexes \
+  -H "Content-Type: application/json" \
+  -H "X-Org-Id: 00d0a0000025groeam" \
+  -d '{
+    "object_api_name": "fferpcore__billingdocument__c",
+    "index_name": "idx_bd_date_status",
+    "columns": ["fferpcore__documentdate__c", "fferpcore__documentstatus__c"],
+    "where_clause": "sf_deleted_at IS NULL"
+  }'
+```
+
+| Field | Required | Notes |
+|---|---|---|
+| `object_api_name` | yes | Salesforce API name of the object |
+| `index_name` | yes | Postgres index name — max 63 characters |
+| `columns` | yes | Array of column names to index |
+| `where_clause` | no | Raw SQL predicate for a partial index |
+
+The index is applied immediately on POST and re-applied every time the API starts or the table is recreated. All index creation uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS` — no table locks.
+
+```bash
+# List registered custom indexes for the active org
+curl http://localhost:7743/api/indexes
+
+# Remove a registration (does not drop the underlying PG index)
+curl -X DELETE http://localhost:7743/api/indexes/1
+```
+
 ## Database schema
 
 **One schema per registered org** — named `org_<lowercased orgid>`, one table per enabled Salesforce object (e.g. `org_00d5g000001abcdeaa.account`)

docs/scope.md

@@ -280,11 +280,13 @@ The sync engine manages schema changes directly with raw SQL — no migration fr
 | Event | DDL executed |
 |---|---|
 | Object enabled | `CREATE TABLE IF NOT EXISTS org_<orgid>.<object> (id text PRIMARY KEY, ...)` |
+| Object enabled | Auto-index every `reference`-type field; apply any registered custom indexes |
 | Object disabled + drop | `DROP TABLE org_<orgid>.<object>` |
 | Field re-enabled | `ALTER TABLE org_<orgid>.<object> ADD COLUMN <field> <type>` |
+| Field re-enabled (reference) | Auto-index the new column |
 | Field disabled | `ALTER TABLE org_<orgid>.<object> DROP COLUMN <field>` |
 
-DDL is idempotent where possible (`IF NOT EXISTS`, `IF EXISTS`).
+DDL is idempotent where possible (`IF NOT EXISTS`, `IF EXISTS`). All index creation uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS` — no table locks.
 
 ---
 
@@ -363,6 +365,55 @@ All per-object/per-field tables include `org_id` and have `ON DELETE CASCADE` fr
 - `org_id` text PRIMARY KEY
 - `locked` boolean, `locked_at` timestamptz, `job_type` text
 
+**`sfdb.custom_indexes`** — operator-defined indexes applied to org object tables
+- `id` serial PRIMARY KEY
+- `org_id` text (CASCADE from sfdb.orgs)
+- `object_api_name` text
+- `index_name` text (≤ 63 chars, unique per org)
+- `columns` text[] — columns to index
+- `where_clause` text NULL — optional partial index predicate (raw SQL)
+- Applied at startup and whenever the target table is created or recreated
+- CRUD via `GET/POST/DELETE /api/indexes`
+
+---
+
+## Indexing
+
+sfetch maintains two layers of indexes on org object tables.
+
+### Auto-indexes (reference fields)
+
+Every Salesforce field of type `reference` (lookup / master-detail) is automatically indexed when its column is created. Reference fields are JOIN columns by definition — indexing them requires no configuration.
+
+Index names follow the pattern `idx_ref_<object>_<field>`. If the combined name exceeds Postgres's 63-character identifier limit, the name is replaced with `idx_ref_<8-char md5 hash>`.
+
+These indexes are created with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so they impose no table lock.
+
+### Custom indexes (`sfdb.custom_indexes`)
+
+For application-specific query patterns (composite filters, date ranges, status columns) that the auto-rule cannot derive, operators register indexes via the API or directly in the table:
+
+```http
+POST /api/indexes
+Content-Type: application/json
+X-Org-Id: 00d0a0000025groeam
+
+{
+  "object_api_name": "fferpcore__billingdocument__c",
+  "index_name": "idx_bd_date_status",
+  "columns": ["fferpcore__documentdate__c", "fferpcore__documentstatus__c"],
+  "where_clause": "sf_deleted_at IS NULL"
+}
+```
+
+Custom indexes are applied immediately on POST and re-applied at every API startup and whenever the target table is created or recreated. They survive container restarts. Removing an entry via `DELETE /api/indexes/:id` removes the registration but does not drop the underlying Postgres index.
+
+| Endpoint | Purpose |
+|---|---|
+| `GET /api/indexes` | List all custom indexes for the active org |
+| `POST /api/indexes` | Register and immediately apply a new index |
+| `DELETE /api/indexes/:id` | Remove a registration |
+
 ---
 
 ## Deletion Tracking

package-lock.json

@@ -75,17 +75,32 @@ for (const org of orgs) {
   const target = org.alias || org.username
   process.stdout.write(`  ${target}... `)
 
-  const raw = run('sf', ['org', 'display', '--target-org', target, '--json'])
+  const displayRaw = run('sf', ['org', 'display', '--target-org', target, '--json'])
   let result
   try {
-    result = JSON.parse(raw)?.result
+    result = JSON.parse(displayRaw)?.result
   } catch {
     console.log('parse error')
     fail++
     continue
   }
 
-  if (!result?.accessToken || !result?.instanceUrl) {
+  if (!result?.instanceUrl) {
+    console.log('no instanceUrl')
+    fail++
+    continue
+  }
+
+  // sf CLI v2 redacts accessToken from org display; use dedicated command instead
+  const tokenRaw = run('sf', ['org', 'auth', 'show-access-token', '--target-org', target, '--json'])
+  let accessToken
+  try {
+    accessToken = JSON.parse(tokenRaw)?.result?.accessToken
+  } catch {
+    // ignore parse errors
+  }
+
+  if (!accessToken || accessToken.startsWith('[REDACTED]')) {
     console.log('no token')
     fail++
     continue
@@ -95,7 +110,7 @@ for (const org of orgs) {
   const entry = {
     alias: org.alias ?? null,
     username: result.username ?? org.username,
-    accessToken: result.accessToken,
+    accessToken,
     instanceUrl: result.instanceUrl,
     exportedAt: new Date().toISOString(),
   }

scripts/export-tokens.js

@@ -8,7 +8,9 @@ import {
   migrateToMultiOrg,
   ensureOrgObjectPrimaryKeys,
   ensureSyncLockRows,
+  ensureCustomIndexesTable,
 } from './db/migrate'
+import { applyAllCustomIndexes } from './sync/ddlManager'
 import { startScheduler } from './sync/scheduler'
 
 // ---------------------------------------------------------------------------
@@ -21,6 +23,7 @@ import logsRouter from './routes/logs'
 import schedulesRouter from './routes/schedules'
 import settingsRouter from './routes/settings'
 import syncOrderRouter from './routes/syncOrder'
+import indexesRouter from './routes/indexes'
 
 // ---------------------------------------------------------------------------
 // CORS helper — allow localhost origins only
@@ -97,6 +100,7 @@ export function createApp() {
   app.use('/api/schedules', schedulesRouter)
   app.use('/api/settings', settingsRouter)
   app.use('/api/sync-order', syncOrderRouter)
+  app.use('/api/indexes', indexesRouter)
 
   // ---------------------------------------------------------------------------
   // SPA static file serving — must come after all /api routes
@@ -155,10 +159,18 @@ export async function initApp(): Promise<void> {
     await migrateToMultiOrg()
     await ensureSyncLockRows()
     await ensureOrgObjectPrimaryKeys()
+    await ensureCustomIndexesTable()
   } catch (err) {
     console.error('[startup] Migration failed:', (err as Error).message)
   }
 
+  try {
+    await applyAllCustomIndexes()
+    console.log('[startup] Custom indexes applied')
+  } catch (err) {
+    console.warn('[startup] Could not apply custom indexes:', (err as Error).message)
+  }
+
   try {
     // Release any stale locks left by a process killed mid-sync (works on
     // both legacy single-row and new per-org sync_lock shapes).

src/api/src/app.ts

@@ -0,0 +1,14 @@
+-- 004_custom_indexes.sql
+-- Operator-defined indexes applied to org object tables at startup and on table creation.
+-- Runs once on first Postgres container start via docker-entrypoint-initdb.d.
+
+CREATE TABLE IF NOT EXISTS sfdb.custom_indexes (
+    id               serial      PRIMARY KEY,
+    org_id           text        NOT NULL REFERENCES sfdb.orgs(org_id) ON DELETE CASCADE,
+    object_api_name  text        NOT NULL,
+    index_name       text        NOT NULL CHECK (char_length(index_name) <= 63),
+    columns          text[]      NOT NULL,
+    where_clause     text,
+    created_at       timestamptz NOT NULL DEFAULT NOW(),
+    UNIQUE (org_id, index_name)
+);

src/api/src/db/init/004_custom_indexes.sql

@@ -323,6 +323,26 @@ export async function migrateToMultiOrg(): Promise<void> {
   }
 }
 
+// ---------------------------------------------------------------------------
+// Ensure sfdb.custom_indexes exists — added after initial release so existing
+// installs won't have it from the init scripts.
+// ---------------------------------------------------------------------------
+
+export async function ensureCustomIndexesTable(): Promise<void> {
+  await pool.query(`
+    CREATE TABLE IF NOT EXISTS sfdb.custom_indexes (
+      id               serial      PRIMARY KEY,
+      org_id           text        NOT NULL REFERENCES sfdb.orgs(org_id) ON DELETE CASCADE,
+      object_api_name  text        NOT NULL,
+      index_name       text        NOT NULL CHECK (char_length(index_name) <= 63),
+      columns          text[]      NOT NULL,
+      where_clause     text,
+      created_at       timestamptz NOT NULL DEFAULT NOW(),
+      UNIQUE (org_id, index_name)
+    )
+  `)
+}
+
 // ---------------------------------------------------------------------------
 // Convenience: ensure a sync_lock row exists for every registered org.
 // Called on every startup so a row is present even if registration predates this code.