[Models] Phase 1 — Foundation: scope.models interfaces, hdb_model_calls, TestBackend

[heskew]

Scope

Foundation for the model-access API in #510:

• TypeScript interfaces (ModelBackend, Models, options types, ModelCallResult, TokenUsage).
• Models facade exposed as scope.models on the Scope class — internally reads contextStorage.getStore() (the existing ALS at resources/Table.ts:3484) to pick up per-request context (tenant, signal, app).
• Backend registry (config-driven models.<embedding|generative>.<name> → backend instance).
• hdb_model_calls system table + buffered writer, declared imperatively per existing system-table convention.
• In-memory TestBackend for tests.
• opts.conversationId accepted on GenerateOpts (inert until Add ConversationResource for agent memory and conversation state #511 lands; no observable behavior in this phase).

No real backends ship in this phase — interface plus test fixture only. Real backends land in Phases 2 (ollama), 3 (openai), and 8 (anthropic, bedrock).

Why first

Every other phase depends on the interface, the facade, the backend registry, and the analytics writer. Sub-issues that build on this: Phase 2, 3, 4, 5, 8 (#510 sub-issues) and #612 (agent-loop orchestration). External issues that consume the interface: #511 (opts.conversationId activation).

API surface

// resources/models/types.ts

export interface Models {
  embed(input: string | string[], opts?: EmbedOpts): Promise<Float32Array[]>;
  generate(input: GenerateInput, opts?: GenerateOpts): Promise<GenerateResult>;
  generateStream(input: GenerateInput, opts?: GenerateOpts): AsyncIterable<GenerateChunk>;
}

export interface ModelBackend {
  readonly name: string;
  capabilities(): ModelCapabilities;
  embed?(input: string | string[], opts: BackendOpts<EmbedOpts>): Promise<ModelCallResult<Float32Array[]>>;
  generate?(input: GenerateInput, opts: BackendOpts<GenerateOpts>): Promise<ModelCallResult<GenerateResult>>;
  generateStream?(input: GenerateInput, opts: BackendOpts<GenerateOpts>): AsyncIterable<GenerateChunk>;
}

export interface ModelCapabilities {
  embed: boolean;
  generate: boolean;
  stream: boolean;
  tools: boolean;
  adapters: boolean;
}

export type EmbedOpts = {
  model?: string;
  inputType?: 'document' | 'query';   // for models that distinguish (nomic-embed-text etc.); ignored by models that don't
  signal?: AbortSignal;
};

export type GenerateOpts = {
  model?: string;
  adapter?: string;
  temperature?: number;
  maxTokens?: number;
  responseFormat?: 'text' | 'json' | { schema: object };
  tools?: ToolDef[];
  toolMode?: 'return' | 'auto';
  conversationId?: string;            // accepted; inert until #511 lands
  signal?: AbortSignal;
};

export type GenerateInput =
  | string
  | Message[]
  | { messages: Message[]; tools?: ToolDef[]; system?: string };

export type BackendOpts<TOpts> = TOpts & { accounting: AccountingContext };

export interface AccountingContext {
  tenantId?: string;
  app?: string;
}

// Backend return type. 'pending' is reserved for future long-running operations
// (Bedrock batch, future fabric LRO); no Phase 1 backend emits it. The public
// Models facade unwraps 'completed' and matches #510's Promise<Float32Array[]>
// / Promise<GenerateResult> signatures.
export type ModelCallResult<T> =
  | { status: 'completed'; output: T; usage?: TokenUsage }
  | { status: 'pending'; operationId: string; resumeAfter?: number };

export interface TokenUsage {
  promptTokens?: number;
  completionTokens?: number;
  embeddingTokens?: number;
  gpuMs?: number;
  latencyMs?: number;
}

// Message, ToolDef, ToolCall, GenerateResult, GenerateChunk: concrete types as in #510's API surface section

Implementation pattern: ALS-backed scope.models

The developer writes scope.models.embed(text) from anywhere in their component — handleApplication(scope) init code, Resource methods (via closure), nested helpers. ALS provides per-request context when called inside a Resource scope; falls back to no-tenant accounting when called outside one.

// components/Scope.ts (modified)
export class Scope extends EventEmitter {
  readonly models: Models;
  constructor(/* existing args */) {
    super();
    // existing init
    this.models = new Models(globalBackendRegistry, globalAnalyticsWriter);
  }
}

// resources/models/Models.ts
import { contextStorage } from '../transaction.ts';   // existing ALS

export class Models {
  async embed(input: string | string[], opts: EmbedOpts = {}): Promise<Float32Array[]> {
    const backend = this.#registry.resolveEmbedding(opts.model);
    const ctx = contextStorage.getStore();             // current request, if any
    const accounting: AccountingContext = {
      tenantId: extractTenantId(ctx?.user),            // free-form string for v1
      app: (ctx as any)?.handlerPath,
    };
    const signal = opts.signal ?? ctx?.signal;
    const backendOpts: BackendOpts<EmbedOpts> = { ...opts, signal, accounting };
    const result = await backend.embed!(input, backendOpts);
    this.#recordCall({ /* ... */ });
    if (result.status === 'completed') return result.output;
    throw new Error(`Backend ${backend.name} returned pending; long-running ops not yet supported`);
  }
  // generate(), generateStream() follow same pattern
}

Precedent: resources/Table.ts:3484 already uses contextStorage.getStore() inside the struct prototype getter for computed-field resolvers. Same store, same pattern.

Behavior across call sites:

Where called	ALS resolves to	Accounting result
Inside a Resource method (REST / WebSocket / GraphQL / Operations API)	The current request context	Full per-tenant accounting + signal propagation
Inside handleApplication(scope) init code	No store	No-tenant accounting (tenant undefined; call still recorded)
Inside a job, cron, internal handler	Whatever the caller set up (often none)	Same as above

hdb_model_calls table

Imperative table({...}) declaration, mirroring existing system tables (hdb_certificate, hdb_analytics, DurableSubscriptionsSession). Type annotations follow the style at server/DurableSubscriptionsSession.ts:42-46.

// resources/models/analyticsTable.ts
table({
  table: 'hdb_model_calls',
  database: 'system',
  audit: true,
  trackDeletes: false,
  attributes: [
    { name: 'id', isPrimaryKey: true },
    { name: 'tenant', type: 'string', indexed: true },
    { name: 'app', type: 'string', indexed: true },
    { name: 'model', type: 'string', indexed: true },
    { name: 'backend', type: 'string', indexed: true },
    { name: 'method', type: 'string', indexed: true },           // 'embed' | 'generate' | 'generateStream'
    { name: 'adapter', type: 'string', indexed: true },
    { name: 'conversation_id', type: 'string', indexed: true },
    { name: 'prompt_tokens', type: 'number' },
    { name: 'completion_tokens', type: 'number' },
    { name: 'embedding_tokens', type: 'number' },
    { name: 'gpu_ms', type: 'number' },
    { name: 'latency_ms', type: 'number', indexed: true },
    { name: 'success', type: 'boolean', indexed: true },
    { name: 'error_code', type: 'string' },
  ],
});

Field-by-field source:

• Twelve fields named verbatim from Add unified model-access API (scope.models) #510's accounting section: tenant, app, model, backend, prompt_tokens, completion_tokens, embedding_tokens, gpu_ms, latency_ms, success, conversation_id, adapter.
• Three additions for index / triage utility: id (primary key), method, error_code.
• Timestamp is auto-tracked via Harper's record metadata ($updatedtime / $createdtime); no explicit field needed.

Writer pattern: in-memory buffer with periodic flush (model: resources/analytics/write.ts:39-106), but writes per-call rows rather than aggregating. Buffer capped at ~10s or ~1000 records, whichever comes first; configurable.

Retention: explicit periodic cleanup() mirroring analytics/write.ts:721-722 (where RAW_EXPIRATION = 1 hour, AGGREGATE_EXPIRATION = 1 year). Default for hdb_model_calls: 90 days, configurable via a new env (ANALYTICS_MODELCALL_RETENTION_DAYS). Tuned for billing windows.

TestBackend

Deterministic in-memory implementation for tests — no external dependencies, no model files to download.

// resources/models/TestBackend.ts
export class TestBackend implements ModelBackend {
  readonly name = 'test';
  capabilities(): ModelCapabilities {
    return { embed: true, generate: true, stream: true, tools: false, adapters: false };
  }
  async embed(input, opts): Promise<ModelCallResult<Float32Array[]>> {
    const texts = Array.isArray(input) ? input : [input];
    const vectors = texts.map(t => deterministicVector(t, 16));   // hash → seed → fixed 16-dim
    return { status: 'completed', output: vectors, usage: { embeddingTokens: texts.reduce((s, t) => s + t.length, 0) } };
  }
  // generate() echoes input with a "[TestBackend echoed]: " prefix
  // generateStream() yields the echo word-by-word
}

Files

Path	Change
resources/models/types.ts	new — all TypeScript interfaces
resources/models/Models.ts	new — facade class with ALS lookup
resources/models/backendRegistry.ts	new — registration + config-driven resolution
resources/models/analyticsTable.ts	new — hdb_model_calls declaration + buffered writer
resources/models/TestBackend.ts	new — in-memory test backend
components/Scope.ts:36	modified — add models: Models property, constructed in Scope's constructor

Acceptance criteria

• ModelBackend, Models, EmbedOpts, GenerateOpts, ModelCapabilities, ModelCallResult, AccountingContext, TokenUsage defined in resources/models/types.ts.
• Models facade exposed as scope.models on every Scope instance.
• scope.models.embed() from inside a Resource method picks up the request's user, signal, and handlerPath via ALS.
• scope.models.embed() from outside a Resource scope (init code, internal jobs) proceeds with tenantId: undefined and no error.
• hdb_model_calls system table created in the system database with the schema above.
• Per-call records written to hdb_model_calls with tenant, app, model, backend, method, token counts, latency, success.
• TestBackend available for tests; produces deterministic vectors and text.
• opts.conversationId accepted on GenerateOpts; no behavior change in this phase (activation lands with Add ConversationResource for agent memory and conversation state #511).
• Backend registry resolves models.embedding.default.backend → registered backend instance.
• Capability negotiation enforced: calling a method the backend doesn't support throws a structured error.
• Unit tests cover: ALS resolution, no-ALS fallback, capability-filtered dispatch, analytics record shape, AbortSignal propagation, backend registry resolution, ModelCallResult.pending rejection path.
• CI green (unit + integration + 3 Node versions).

Out of scope

• Real backends (Phases 2, 3, 8).
• @embed schema directive (Phase 5).
• /v1/embeddings + /v1/chat/completions (Phase 4).
• toolMode: 'auto' orchestration (split to Add agent-loop orchestration / toolMode: 'auto' to scope.models #612).
• conversationId activation (wired but inert until Add ConversationResource for agent memory and conversation state #511 lands).
• Tenant identity beyond a free-form string (see comment thread on Add unified model-access API (scope.models) #510).
• LRO operation polling / getOperation / cancelOperation methods on the public facade (interface reserves ModelCallResult.pending for forward-compat; polling lands when a real LRO-capable backend ships).

Stacks on

Nothing. This is the foundation.

Hard prerequisites

• Expose request.signal (AbortSignal) for Resource methods #513 (request.signal exposed on Resource methods) — Models facade reads ctx.signal from the ALS-bound context.

Branch & PR conventions

• Branch: feat/models-foundation
• PR base: main
• Closes this issue via Closes #<self>; references Add unified model-access API (scope.models) #510 via Tracking: #510.
• Merge with a merge commit (per CONTRIBUTING.md).

Smoke test

// In a Resource method, with TestBackend registered as default embedding backend:
class TestResource extends Resource {
  async post(_target, _data, _request) {
    const vectors = await scope.models.embed('hello world');
    return { vectorLength: vectors[0].length };
  }
}

// POST to the Resource:
//   curl -X POST http://localhost:9926/TestResource/ -d '{}'
// Expected: { vectorLength: 16 }
// Verify: SELECT * FROM system.hdb_model_calls WHERE backend = 'test'
//   shows one row with tenant/app populated from the request, method='embed', latency_ms set, success=true.

// And from app-init (outside any Resource scope, no ALS context):
//   scope.models.embed('init')
// Expected: vector returned, hdb_model_calls row has tenant undefined.

Tracking

Part of #510. Sub-issue 1 of 6.

---

🤖 Generated with Claude Code