Skip to content

DESIGN.md

Latest commit

 

History

History
165 lines (123 loc) · 32.5 KB

DESIGN.md

File metadata and controls

165 lines (123 loc) · 32.5 KB
Raw

resources/ — Navigation Guide

The Resource layer is Harper's universal abstraction: all queryable/mutable things (tables, caches, message topics, custom endpoints) extend Resource. Inbound protocols (REST, GraphQL, MQTT, NATS, WebSockets) all converge on this interface.

Read this when: you're touching the read/write path, authorization, subscriptions, or table CRUD semantics.

See also: ../DESIGN.md for cross-cutting non-obvious internals (RecordObject prototype, getFromSource timing, blob orphan cleanup).

Navigation convention. This guide references code by symbol name (e.g. _writeUpdate) and by section marker (e.g. // #section: write-path-internals). Jump in your editor via go-to-symbol, or grep for the section marker. Line numbers drift; symbols and section markers don't.

File overview

File Purpose
Resource.ts Base class; transactional() wrapper; method routing
Table.ts Table-as-Resource implementation. Factory makeTable() returns a TableResource subclass per table. See section markers below.
Resources.ts Registry mapping URL paths → Resource classes
RequestTarget.ts Parses path/query into a structured target
ResourceInterface.ts Type definitions (Context, Record, etc.)
RecordEncoder.ts msgpack encoding + entryMap (record → storage entry)
IterableEventQueue.ts Async iterable used for subscriptions and streaming responses
transaction.ts Per-request transaction object stored in contextStorage
auditStore.ts Append-only audit log records
nodeIdMapping.ts Maps node IDs ↔ timestamps for replication ordering
openApi.ts Generates OpenAPI/JSON Schema from @export schemas
defineTable.ts Code-first table authoring (defineTable + types) — a TS front-end to the canonical table() model
defineResource.ts Per-method request contract (defineResource / Resource.withSchema, t, schemaOf) — typed handlers + edge validation
jsonSchemaTypes.ts Shared JsonSchemaFragment IR + attributeToFragment projector (one vocabulary for validation/OpenAPI/MCP)
analytics/ Telemetry recording (separate from monitoring)

Resource.ts — base class

Static methods are protocol entry points (each wrapped in transactional()); instance methods are the per-resource behavior hooks subclasses override.

Member Notes
class Resource Generic over Record extends object
constructor(identifier, source)
Static CRUD entry points get, put, patch, delete, post, update, create, invalidate
Static pub/sub entry points connect, subscribe, publish
Static query entry points search, query
Static path helpers parsePath (URL → RequestTarget), getResource (path → class)
Other statics getNewId, copy, move
Authorization hooks allowRead / allowUpdate / allowCreate / allowDelete — default impls; override per resource
Instance helpers getId, getContext, getCurrentUser
transactional() wrapper Do not remove from static methods — owns transaction context lifetime
missingMethod / allowedMethods 405 response helpers
transformForSelect Select-clause expansion

Table.ts — section map

One giant makeTable() factory that returns a TableResource extends Resource class. The file is divided into the sections below; each is anchored by a // #section: <name> marker — grep for the marker (or use VS Code's go-to-symbol within the section) to land directly.

Section marker Contents
#section: setup-and-factory makeTable(options) entry, attribute parsing & primary-key detection, replication wiring, class Updatable (RecordObject prototype: getUpdatedTime, getExpiresAt, addTo, subtractFrom). Ends where class TableResource opens.
#section: static-config Static configuration properties: name, primaryStore, auditStore, primaryKey, indices, audit, databasePath, attributes, replicate, sealed, splitSegments, getResidencyById, dbisDB, schemaDefined, expirationMS.
#section: resource-registry sourcedFrom() (cache/source hierarchy — the largest static), isCaching, shouldRevalidateEvents, getResource(), _updateResource, ensureLoaded().
#section: lifecycle-admin getNewId() (UUID / autoincrement / prefix / time-based strategies), setTTLExpiration, residency (getResidencyRecord, setResidency, setResidencyById, getResidency), enableAuditing, coerceId, dropTable.
#section: read-path get() overloads & impl.
#section: authz-hooks allowRead, allowUpdate, allowCreate, allowDelete.
#section: write-path-public update(), save(), addTo(), subtractFrom(), getMetadata, getRecord, getChanges, _setChanges, setRecord, invalidate(), operation(), put(), create(), patch().
#section: write-path-internals _writeUpdate() — the central write routine (versioning, conflict resolution, audit, residency, replication metadata, blob orphan tracking). The write.skipped flag mentioned in ../DESIGN.md is set in this method's early-return paths. Also _writeInvalidate, _writeRelocate, _recordRelocate, evict(), lock(), delete(), _writeDelete.
#section: search-query search() (the query engine — index selection, filter evaluation), transformToOrderedSelect (select-clause ordering), transformEntryForSelect (record → response shape).
#section: pub-sub subscribe() (subscription request handling, replay, cursor management), subscribeOnThisThread, doesExist(), publish(), _writePublish().
#section: validation validate(record, patch?) — schema enforcement, computed attributes, attribute coercion.
#section: stats-admin getUpdatedTime, addAttributes, removeAttributes, getSize, getAuditSize, getStorageStats, getRecordCount, updatedAttributes (schema diff machinery).
#section: computed-history setComputedAttribute, deleteHistory, getHistory (generator), getHistoryOfRecord, clear, cleanup, _readTxnForContext.
(after the class) getFromSource() — cache miss → source load (see ../DESIGN.md for the resolve-before-commit timing trap); local helpers (coerceType, isDescendantId, etc.).

"Where is X" cheat sheet

Question Where
How is a CRUD request authorized? Table.ts → #section: authz-hooks; defaults in Resource.ts (allowRead etc.)
Where does versioning / conflict resolution happen? Table.ts → _writeUpdate (#section: write-path-internals)
How does search() choose an index? Table.ts → search (#section: search-query)
How are subscriptions replayed? Table.ts → subscribe (#section: pub-sub)
How is the response body shaped (select clause)? Table.ts → transformEntryForSelect (#section: search-query)
Where is record-level TTL evaluated? Table.ts → setTTLExpiration (#section: lifecycle-admin); Updatable.getExpiresAt (#section: setup-and-factory). Stored expiry metadata is resolved in the _writeUpdate commit closure: options.expiresAt ?? context.expiresAt ?? (record @expiresAt field, if finite &amp; ≥ 0) ?? table default. This metadata drives read-hiding + the cleanup sweep. The @expiresAt attribute is authoritative for direct put/patch only; cache/source fills persist via recordUpdater and derive expiry from sourceContext.expiresAt (source freshness / table default), not the field.
Why does search() hide a row that's past its TTL but not yet swept? Table.ts → transformEntryForSelect unconditionally treats entry.expiresAt < Date.now() as gone (lazy eviction on read) — correct for a SELECT, but a mutation locating rows to overwrite needs the opposite: pass target.includeExpired = true (read by the SQL engine's runUpdate/runDelete via SqlEngineContext.includeExpiredRows) to treat such a row as a live match, matching the leniency a direct by-id put/patch already has (they skip this check entirely, since Resource.patch's static options don't request ensureLoaded).
How are residencies enforced (replication)? Table.ts → #section: lifecycle-admin (residency block: getResidencyRecord, setResidency, setResidencyById, getResidency)
How is the RecordObject prototype applied? RecordEncoder.ts (see ../DESIGN.md)
Where is the per-request transaction stored? transaction.ts + contextStorage (AsyncLocalStorage)
How does a query opt out of a read snapshot? Pass snapshot: false on the search request (e.g. get_analytics). Table.ts → search calls txn.useReadTxn(snapshot === false); on RocksDB DatabaseTransaction.getReadTxn then builds the read txn with { disableSnapshot: true } so a long scan reads latest without pinning a snapshot. No-op on LMDB (LMDBTransaction.useReadTxn).
How does a URL path map to a Resource? Resources.ts → getMatch (exact/prefix fast path) then matchParamRoute (parameterised routes); see "Path routing" below
How does HNSW keep the graph connected on delete? indexes/HierarchicalNavigableSmallWorld.ts → index() delete path: zero-degree orphans reindexed via needsReindexing; severed multi-node islands detected and reconnected by repairSeveredNeighbors (#1712)
How is a filter applied during a vector search? Predicate-aware traversal (#1241): search.ts → executeConditions composes companion AND conditions + a request vectorFilter + a record-scoped allowRead override into one (primaryKey) => boolean (composeRecordFilter) and passes it to HierarchicalNavigableSmallWorld.search(cond, ctx, filter). The filter gates result admission at layer 0 only (routing ignores it, ACORN-style); a visit budget (filterExpansion) bounds the under-filled/selective case. Very selective condition filters are instead diverted to the exact brute-force path by the query planner's estimateCountAsSort ordering.
How is row-level read access control enforced? Unified allowRead (#1422 gap 2 / #1241): an application-OVERRIDDEN allowRead (detected via the isDefaultAllowRead marker on the framework defaults) is record-scoped — evaluated once per record with this = the (frozen) record during query execution, fail-closed on throw, dispatched via the resolved method (never record.allowRead lookup — data shadowing). The authorize wrapper (Resource.ts → authorizeActionOnResource) defers collection reads on tables (supportsRowLevelAllowRead) to this per-record path; single-record get keeps the entry check (record loaded, proxied reads work). Records also expose a non-enumerable allowRead delegate on the per-table structPrototype.

Path routing & parameterised routes

Resources.ts is the registry that maps URL paths to Resource classes. Resources are registered (jsResource.ts) from a component's exports:

  • Default path (convention): the export name, resolved relative to the component's directory — export class Widget → <dir>/Widget.
  • Declared path (static path): a static path field overrides the convention. A leading / makes it root-relative (top-level); ./ or a bare name is relative to the component directory.
  • Export-name-as-path: export { Widget as '/widget/:id' } — the export name is the path (also honors the leading-slash root rule).

A path is parameterised if any segment begins with : (named param) or * (wildcard/catch-all):

export class Widget extends Resource {
	static path = '/widget/:id/action/:action';
	get(target) {
		// GET /widget/10/action/jump → target.id === '10', target.action === 'jump'
	}
}

export class Files extends Resource {
	static path = '/files/*rest'; // GET /files/a/b/c.txt → target.rest === 'a/b/c.txt'
}

Mechanics:

  • Registration (Resources.set): parameterised paths are compiled into paramRoutes (kept out of the base Map) so the exact/prefix matching fast path is untouched. Routes are ordered most-specific-first (more leading static segments, then longer patterns; wildcards rank last).
  • Matching (Resources.getMatch): exact and prefix matches are tried first and win ("static wins"); only when no static resource matches — and only if paramRoutes is non-empty — does matchParamRoute run. Matched segment values are decoded and stored on entry.params.
  • Binding to the target: request handlers (server/REST.ts, server/DurableSubscriptionsSession.ts) Object.assign(target, entry.params) after building the RequestTarget, so :id lands on target.id, *rest on target.rest, etc.
  • Named params match exactly one segment; a wildcard captures the remainder (zero or more segments) and must be the final segment.
  • Discovery surfaces: because parameterised routes live outside the base Map, the enumerators read resources.paramRoutes explicitly — openApi.ts emits them as templated paths (:id → {id}) with path parameters, and components/mcp/resources.ts lists them via resources/templates/list as {param} URI templates. routePatternToTemplate (exported from Resources.ts) is the shared :param/*wildcard → {param} converter.

Tests: ../unitTests/resources/paramRoutes.test.js (unit) and ../integrationTests/apiTests/param-routes.test.mjs (end-to-end); enumeration coverage in ../unitTests/resources/openApi.test.js and ../unitTests/components/mcp/resources.test.js.

Typed, discoverable resources (code-first schema + request contract)

Design record: the full RFC and its type-level design proofs live in the design PR (#1503); this section is the retained summary.

The principle. Harper strips TypeScript at runtime (--conditions=typestrip), which erases types and rules out metadata-emitting decorators. So runtime metadata must be values, and TypeScript types are derived from those values — never the reverse. Everything here is erasable syntax; the values survive stripping, the types are inferred, nothing can drift. One shared IR — JsonSchemaFragment (jsonSchemaTypes.ts), produced by attributeToFragment — feeds validation, OpenAPI, and MCP, so those surfaces cannot silently disagree.

Code-first tables (defineTable.ts). defineTable(name, shape, opts) authors a table in TypeScript and eagerly registers it through the same table() factory GraphQL drives — the returned value is the live table class (Track.get/put/... work, new Track()/instanceof hold). Fields come from the types vocabulary (getter flags: string.indexed, id.primaryKey, date.createdTime); per-verb shapes are inferred projections discoverable as members ((typeof Track)['$insert' | '$upsert' | '$patch' | '$query' | '$record']). Relationships use lazy thunks (types.relation(() => Album, { from })) so forward references/cycles resolve at query time; relationOf/hasManyOf are the escape hatch for a mutual pair whose eager const-inference would otherwise collapse to any.

Per-method request contract (defineResource.ts). Two front-ends, same runtime metadata:

  • defineResource(contract, impl) — function form (an object of verb handlers).
  • Resource.withSchema(contract) — class form; extends it and implement the declared verbs. It pins static loadAsInstance = false so instance verbs receive the converged (target, data) arg order the types assume (the default dispatch order is (data, target) — see Resource.post/put/patch).

A contract is { path, record?, get/post/put/patch/delete: { query?, body?, response? } }. Handler types are derived from it: path params from a template-literal parse of path, query/body/response from the schema's inferred type. It is a subset, not a fork — a handler gets the SAME RequestTarget, structurally narrowed (target.id: string, target.get('expand') typed by the query schema), and the resource still registers/serves like a plain one. The narrowed types are justified by runtime enforcement: each declared verb validates/coerces query/body before dispatch and throws a structured 400 (ValidationError, per-field { path, code, message }[]) — the same bargain Table.validate makes for tables.

Vocabulary. The built-in t (t.string/number/integer/boolean/date/enum/array/object) and schemaOf<T>(source?) both reduce to JsonSchemaFragment. A defineTable projection slots straight into a contract body/response — schemaOf<(typeof Track)['$insert']>({ table: Track, projection: 'insert' }) derives the compile-time type from the projection and the runtime fragment from the table's attributes (via projectTableFragment → attributeToFragment). Nullability: non-nullable by default (a bare t.string rejects null); .optional allows absence, .nullable allows an explicit null; table-derived bodies mirror Table.validate's policy (null rejected only when nullable === false).

Downstream surfaces. openApi.ts emits a contract's query params, request body, and response for parameterised routes; components/mcp/tools/application.ts drives the tool input/output schema off the contract and binds arbitrary path params + query (applyContractInputs), which lifts the generated-verb binding restriction for contract resources. ValidationError (../utility/errors/hdbError.ts) extends ClientError (400) so existing HTTP handling is unchanged; the structured issues ride on .detail/.errors.

Tests: ../unitTests/resources/defineResource.test.js, ../unitTests/resources/defineTable-registration.test.js, ../unitTests/resources/openApi-contract.test.js, ../unitTests/components/mcp/tools/application-contract.test.js.

Conventions

  • Never remove transactional() from a static method on Resource — it owns transaction context lifetime.
  • New Resource subclasses should override instance methods (get, put, ...) for behavior; static methods are the protocol entry points and stay generic.
  • When adding a new early-return path inside a commit handler in _writeUpdate, follow the blob-cleanup protocol documented in ../DESIGN.md ("Blob orphan cleanup").
  • If you add a new top-level section to Table.ts, drop a // #section: <name> marker at its start and add a row to the section map above.
  • Tests for this layer live in ../unitTests/resources/.