Skip to content

Branched databases: app config, scope proxy, lifecycle wiring #643

Description

@kriszyp

Summary

Harper-side implementation of branched databases. Subtask of #642; depends on HarperFast/rocksdb-js#577 for the underlying checkpoint primitive.

Scope

Everything inside `core/` that wires the feature together. No native code changes.

1. Application config schema

`core/components/Application.ts` — extend `ApplicationConfig` (lines 28–38) with:

```ts
branchedDatabases?: string[];
```

Validate in `assertApplicationConfig()` (lines 72–109): array of strings; each must name an existing database; reject duplicates.

2. ApplicationScope field

`core/components/ApplicationScope.ts` — add:

```ts
branches: Map<string, string>; // baseDbName → branchDbName
```

3. Branch creation on app load

Extend `prepareApplication()` in `core/components/Application.ts` (lines 429–480). After the `ApplicationScope` is constructed but before any of the app's modules import `harper`:

For each `baseName` in `config.branchedDatabases`:

  • Branch DB name: `${scope.appName}:${baseName}` (colon-namespaced in the `databases` map).
  • Branch path: `<storage_root>/database/${scope.appName}${baseName}` (use `` on disk; colon would be illegal on some filesystems).
  • Resolve the base `RocksDatabase` via `database({ database: baseName })` (`core/resources/databases.ts:740`).
  • `await baseDb.createCheckpoint(branchPath)`.
  • Open the checkpoint as a database via the same factory, override path; register under the namespaced key.
  • Record `scope.branches.set(baseName, '${scope.appName}:${baseName}')`.

The checkpoint is byte-identical to the base, so `internal_dbis/` metadata is correct and `makeTable()` (`core/resources/Table.ts:146`) works unchanged — Table classes get `primaryStore` pointing at the branch.

4. Scoped `databases` proxy in `getHarperExports`

`core/security/jsLoader.ts` (`getHarperExports` lines 678–716). Replace the static `databases` reference with a `Proxy` that consults `scope.branches`:

```ts
const scopedDatabases = scope.branches.size === 0
? databases
: new Proxy(databases, {
get(target, prop, recv) {
if (typeof prop === 'string' && scope.branches.has(prop)) {
return Reflect.get(target, scope.branches.get(prop)!, recv);
}
return Reflect.get(target, prop, recv);
},
has(target, prop) {
return (typeof prop === 'string' && scope.branches.has(prop)) || Reflect.has(target, prop);
},
});
```

Mirror the proxy treatment for the `tables` flat alias when the default database is branched.

5. Branch teardown

  • On `ApplicationScope` dispose / app unload: `branchDb.close()` then `fs.rm(branchPath, { recursive: true })`. Hardlinks mean the base's SST files remain intact.
  • Startup sweep in `core/resources/databases.ts` initialization: remove any directory matching the `__` pattern that doesn't correspond to a currently-loaded app, to clean up after crashes.

6. Replication exclusion

Branch DBs must not participate in cluster replication. Plumb through the existing non-replicating mechanism (cf. `NON_REPLICATING_SYSTEM_TABLES` pattern in `databases.ts:78`) so replication code skips them. Verify there is no code path that auto-discovers databases on disk and replicates them.

Acceptance criteria

  • App with `branchedDatabases: ['data']` boots; its `databases.data` is the branch.
  • Writes from one branched app are invisible to a second branched app and to the base.
  • Reads from an app without `branchedDatabases` see the unchanged base.
  • On clean shutdown, branch directories are removed.
  • On unclean shutdown, startup sweep removes orphaned branch directories.
  • Replication does not propagate branch writes.

Verification

Integration test under the component test suite:

  1. Define base `data` with table `Item` containing 3 rows.
  2. Deploy two test applications `appA` and `appB`, both with `branchedDatabases: ['data']`.
  3. From `appA` write `{id: 'a-only'}`; from `appB` write `{id: 'b-only'}`.
  4. Assert: `appA` reads 3 + `a-only`; `appB` reads 3 + `b-only`; a third unbranched app reads exactly 3.
  5. Restart process; assert branch directories cleaned up and base `data` still has exactly 3 rows.

🤖 Filed by Claude on behalf of Kris

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:componentsComponents / applications subsystemenhancementNew feature or request

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions