Branched databases for application isolation (ephemeral, per-process)

[kriszyp]

Summary

Allow a deployed application to be configured to use a branched copy of one or more databases. A branch is a writable, isolated, ephemeral fork of a base database that lives for the lifetime of the process and is invisible to other applications and to replication. The app's code is unchanged — it imports databases.data from harper exactly as it does today, but the binding resolves to the branch.

Motivation

The driving use case is ephemeral application versions on a QA server: deploy multiple variants of an app to the same Harper instance, each with its own isolated view of data, so writes don't collide and tests don't pollute the shared database. Today there is no way to give an app a private writable copy of an existing database short of running a separate instance.

Proposed UX

In an application's config:

package: my-app
branchedDatabases:
  - data

The app then imports from harper normally:

import { databases } from 'harper';
await databases.data.MyTable.put({ id: 'x', ... });

…and databases.data transparently resolves to a branch named my-app:data. Other apps (and the base instance) see the unchanged base data.

Approach

RocksDB Checkpoint + per-application databases proxy + shared blob store.

1. On app load, for each entry in branchedDatabases, call baseDb.createCheckpoint(branchPath) (depends on Expose RocksDB Checkpoint API as createCheckpoint(path) rocksdb-js#577) to hardlink-clone the base DB.
2. Open the checkpoint via the existing database() factory (core/resources/databases.ts:740), registered under a namespaced key <appName>:<baseName>.
3. In getHarperExports(scope) (core/security/jsLoader.ts:678), wrap the global databases object in a Proxy so the app sees the branch where its config requests one. Mirror the same treatment for the tables flat alias when the default database is branched.
4. Blobs share the base's directory and ID allocator. The checkpoint only clones RocksDB; blob references in cloned records point at files under the base's blob dir. Branches must read/write the base's blob path and increment the base's blob ID counter so new blobs never collide. Branch sweep-deletes are gated by a high-water mark recorded at checkpoint time so the branch never removes a blob the base still references. See Branched databases: blob store sharing, allocator, and GC safety #644 for the full design.
5. On app unload / process exit, close the branch DB and fs.rm its directory. Add a startup sweep for orphaned branch directories from prior crashes.

Branches must not participate in cluster replication.

Subtasks

• Expose RocksDB Checkpoint API as createCheckpoint(path) rocksdb-js#577 — Expose createCheckpoint() in the native binding
• Branched databases: app config, scope proxy, lifecycle wiring #643 — App config, scope proxy, lifecycle wiring
• Branched databases: blob store sharing, allocator, and GC safety #644 — Blob store sharing, allocator, and GC safety
• Integration test: two apps with branched data, divergent writes (including blob writes), isolation assertions, base unchanged

Key files

• core/components/Application.ts (config schema lines 28–38, validation lines 72–109, prepareApplication lines 429–480)
• core/components/ApplicationScope.ts (add branches: Map<string, { branchName: string; hwm: bigint }>)
• core/components/componentLoader.ts (lines 307–315 — config plumbing into scope)
• core/security/jsLoader.ts (getHarperExports lines 678–716)
• core/resources/databases.ts (database() factory lines 740–794, cache line 771; startup sweep)
• core/resources/blob.ts (lines 614, 681–699, 775–815, 1218–1337 — shared path, shared allocator, gated delete, GC safety)

Explicit non-goals

• Persistence of branches across process restarts.
• Merge / promote branch writes back to base.
• Per-table (rather than per-database) branching.
• Branch sharing across applications.
• LMDB backend — RocksDB only.

🤖 Filed by Claude on behalf of Kris