feat(models): @embed directive + write-time hook + auto-HNSW (Phase 5 of #510)

[heskew]

Summary

Phase 5 of #510. An @embed schema directive that, on write, embeds a source field into a vector attribute and HNSW-indexes it — no app code.

type Document @table @export {
  id: ID @primaryKey
  content: String
  embedding: [Float] @embed(source: "content", model: "default")
}

Three lines replace a custom embedding pipeline.

What ships

• Parser (resources/graphql.ts): an @embed(source, model) branch. Both args are required and must be string literals — a malformed directive fails schema install with a ClientError(400). When the attribute has no explicit @indexed, HNSW indexing is auto-attached. The model is recorded as the attribute version so a model change triggers a reindex.
• Override API (resources/Table.ts): Table.setEmbedAttribute(name, embedder) lets a component author replace the default embedder (e.g. multi-field concatenation or a different inputType).
• Write-time hook (resources/models/embedHook.ts, resources/Table.ts): on a write whose payload includes the source field, the embedder runs and writes the vector onto the record before the write commits (sync-by-default). It also wires the cache-fill (getFromSource) path so caching tables embed on source fetch.
• Storage shape: vectors are stored as a plain number[]. Harper's record encoder does not round-trip typed arrays, so any embedder output (incl. Float32Array) is flattened at the boundary. Declaring the attribute [Float] reuses the existing array type and HNSW path — no new primitive type.
• Replication-receiver skip: the embedder is skipped when the write already carries the vector — cluster-replication receive (isNotification), REST x-replicate-from: none, or audit-log replay — so receivers store the originator's vector verbatim and the cluster stays in one embedding space.
• Error sanitization: backend URLs / model IDs / API-key tails stay in server logs; the caller sees only Failed to compute embedding for attribute "<name>".

Design notes

• Sync-by-default per Kris on [Models] Phase 5 — @embed directive + write-time hook + auto-HNSW + model tracking #632 ("overwhelmingly the mode that should and would be used") — fits the existing pre-commit write path with no new infrastructure.
• No Vector primitive. A vector column is just [Float]; HNSW already indexes a plain number[]. (Reviewed and dropped in favor of reusing [Float].)
• Model-change handling is partial. A model change reindexes the existing stored vectors — it does not re-embed source fields through the new model. New writes use the new model; existing rows keep their vectors until rewritten. Full re-embed backfill is tracked in [Models] Derived caching table for @embed (low-latency writes + model-change backfill) #750.

Known limitation

Proactive source-subscribe pushes into a caching table set isNotification, so they currently skip embedding (the predicate can't yet distinguish a cluster-replication receive from an external source push). The lazy getFromSource path and direct writes embed normally. Closing the subscribe path is folded into the derived-caching work (#750).

Tests

• Unit (unitTests/resources/models/embedHook.test.js): default embedder, the replication-skip signals, source-presence semantics (embed on present, clear on null, skip on omit), typed-array → number[] normalization, CRDT-op skip, and error sanitization.
• Integration (integrationTests/server/embed-directive.test.ts): end-to-end against an in-test fake Ollama — schema install + auto-HNSW, POST/PUT/PATCH embed behavior, replication-receiver skip, and the caching-table path.

Out of scope (follow-ups)

• Derived caching table ([Models] Derived caching table for @embed (low-latency writes + model-change backfill) #750): low-latency writes, re-embed-on-model-change backfill, and the source-subscribe embed gap above.
• Cross-model migration tooling and multimodal embeddings.

Closes #632. Part of #510.

🤖 Generated with Claude Code

[claude]

Reviewed; no blockers found.

[kriszyp]

It seems like we are entirely missing support for embeddings with getFromSource writes. And using embeddings with caching tables actually seems like one of the most compelling uses cases.

My Gemini review caught a number of things:

1. Unhandled Promise in TableResource.update leads to Transaction Commits before Embed resolves — CRITICAL (Confirmed)

File: Table.ts
Domain: concurrency
What: The update(updates, fullUpdate) instance method of TableResource calls this._writeUpdate(...) which returns a Promise when @embed is active. However, update() completely ignores this promise and returns this
synchronously.
Why it matters: Because update() returns synchronously, the calling application will immediately proceed to run await instance.save() . However, the asynchronous embedder operation is still pending, meaning the write has not been
registered in the transaction's write list yet. The transaction will either commit without saving the updated record (causing silent data loss) or crash with a "Transaction already closed" error when the embedder finally resolves and
tries to invoke addWrite() on a committed transaction.
Code:

// lines 1243-1247                                                                                                                                                                                                                     
return when(loading, () => {                                                                                                                                                                                                           
    this.#changes = updates;                                                                                                                                                                                                              
    this._writeUpdate(id, this.#changes, false);                                                                                                                                                                                          
    return this;                                                                                                                                                                                                                          
});                                                                                                                                                                                                                                    
...                                                                                                                                                                                                                                    
// line 1251-1252                                                                                                                                                                                                                      
this._writeUpdate(id, this.#changes, fullUpdate);                                                                                                                                                                                      
return this;                                                                                                                                                                                                                           

Suggested fix:
Wrap both invocations of _writeUpdate inside the update method with Harper's when() helper so that the promise is properly propagated back to the caller when an asynchronous hook is active:

return when(loading, () => {                                                                                                                                                                                                           
    this.#changes = updates;                                                                                                                                                                                                              
    return when(this._writeUpdate(id, this.#changes, false), () => this);                                                                                                                                                                 
});                                                                                                                                                                                                                                    
...                                                                                                                                                                                                                                    
return when(this._writeUpdate(id, this.#changes, fullUpdate), () => this);                                                                                                                                                             

──────

2. coerceType for Vector returns Float32Array which is mangled by the Record Encoder — HIGH (Confirmed)

File: Table.ts
Domain: data-integrity
What: coerceType converts incoming Vector attribute values into a Float32Array . However, as documented in createDefaultEmbedder , Harper's record encoder ( updateAndFreeze ) does not cleanly serialize typed arrays, instead
mangling them into {0: val1, 1: val2, ...} objects in the database.
Why it matters: While the default embedder circumvents this by casting its result back to Array , any user-supplied vectors (e.g., replication receiver paths, programmatic writes, or manual API uploads) will pass through
coerceType and be coerced to a Float32Array . These will get stored in the database in a mangled object format, corrupting the vector column and breaking subsequent HNSW indexing and search operations.
Code:

case 'Vector':                                                                                                                                                                                                                         
    if (value === null || value === 'null') return null;                                                                                                                                                                                  
    if (value instanceof Float32Array) return value;                                                                                                                                                                                      
    if (Array.isArray(value)) return Float32Array.from(value);                                                                                                                                                                            
    if (ArrayBuffer.isView(value)) return Float32Array.from(value as any);                                                                                                                                                                
    throw new SyntaxError();                                                                                                                                                                                                              

Suggested fix:
Coerce vectors to standard JavaScript arrays ( number[] ) instead of Float32Array s, ensuring they bypass record-encoder mangling while remaining compatible with HNSW:

case 'Vector':                                                                                                                                                                                                                         
    if (value === null || value === 'null') return null;                                                                                                                                                                                  
    if (value instanceof Float32Array) return Array.from(value);                                                                                                                                                                          
    if (Array.isArray(value)) return value.map(Number);                                                                                                                                                                                   
    if (ArrayBuffer.isView(value)) return Array.from(value as any);                                                                                                                                                                       
    throw new SyntaxError();                                                                                                                                                                                                              

──────

3. Model Change via Schema Reload causes Dimension Mismatches or Mixed-Model Vectors in HNSW — MEDIUM (Confirmed)

File: graphql.ts and embedHook.ts (comments)
Domain: data-integrity / config
What: Modifying the @embed(model: "...") argument in a schema file updates the attribute version, triggering a schema reload and an HNSW reindex. However, existing rows are not automatically re-embedded through the new model.
Why it matters: If a user changes their embedding model (e.g., swapping a 384-dimension model for a 1536-dimension model), existing rows retain their old-dimension vectors. Loading these mixed-dimension vectors into the same HNSW
index
will either trigger a crash during index construction or severely pollute the embedding space, returning garbage query results.
Suggested fix:

1. Document this operational hazard explicitly.
2. Consider adding validation to block schema updates that change model definitions on populated tables unless the vector attribute is cleared, or plan a background job pathway to invalidate and backfill the vector column when a
   model version change is detected.
   ──────

4. Malformed @embed Directives Fail Silently during Schema Deployment — LOW (Confirmed)

File: graphql.ts
Domain: api / config
What: When a malformed @embed directive is encountered (e.g., omitting source or model , or supplying non-string literals), the schema compiler logs via console.error and skips registering the @embed property metadata, but
continues compiling.
Why it matters: Because no exception is thrown, the overall schema installation succeeds with a 200 OK . The field is silently registered as a regular, un-embedded Vector attribute. Users receive no indication that their @embed
directive was discarded unless they inspect the server console logs.
Code:

if (!embedDefinition.source || !embedDefinition.model) {
    console.error(
        `@embed on "${property.name}" requires both "source" and "model" arguments, at`,
        directive.loc
    );
}

Suggested fix:
Throw a descriptive ClientError to halt compilation and fail the schema deployment loudly, returning a helpful error message to the client:

if (!embedDefinition.source || !embedDefinition.model) {
    throw new ClientError(
        `@embed on "${property.name}" requires both "source" and "model" arguments`,
        400
    );
}

──────

5. Dependency on globalThis.logger in embedHook.ts — LOW (Likely)

File: embedHook.ts
Domain: config
What: Failure logging in embedHook.ts relies on resolving the logger dynamically via (globalThis as any).logger rather than using standard module imports.
Why it matters: Relying on globalThis for logger references deviates from ES module standards, bypasses static type checks, and can lead to unhandled runtime exceptions if the logger isn't populated on globalThis in specific
worker threads.
Code:

const logger = (globalThis as any).logger;
logger?.error?.(`Embedder for attribute "${attr.name}" failed:`, err);

Suggested fix:
Import the standard Harper logger statically:

import { logger } from '#src/utility/logging/logger.ts';

[heskew]

Thanks Kris — all three addressed in 5e5ce295:

• Vector type dropped. You were right it's redundant — HNSW already indexes plain [Float] columns, so the declaration is now embedding: [Float] @embed(...). Removed the primitive and its coerceType case. Also sidesteps the Float64/Float16 question — everything's number[] now; F16-in-HNSW is a separate follow-up.
• Out of Resource.ts. Removed the @embed special-casing from the base class, along with the duplicate-id sequencing it existed to serve — that payload-order guarantee was speculative, so it and its test are gone. The async embed hook is still awaited before commit via when() on the put/patch/update paths.
• Comments. Full pass — no more file:line refs, commit hashes, or process narrative. embedHook.ts went 286→136 lines. Point taken on tokens better spent reading the code.

−377 lines net. Codex review clean; unit + integration suites green.

---

🤖 Posted by Claude on Nathan's behalf

[kriszyp]

This is going to be awesome, good job.

[kriszyp]

Heads-up on a friendly seam where this meets the HNSW int8 work in #894.

#894 makes the index store vectors as int8 for fast graph traversal (the per-node-visit vector decode was 54–71% of search/insert CPU; quantizing the index gave ~4.9× search and a ~3× smaller index). This PR's Vector type stores the record vector as Float32Array — which turns out to be the ideal substrate for the planned over-fetch + rerank step (recompute exact distances from the record after int8 navigation, to recover exact $distance and recall): a Float32Array record makes that rerank decode cheap.

One small coordination point, no conflict (#894 is index-only): after this lands, HierarchicalNavigableSmallWorld.index() may receive a Float32Array rather than a number[]. quantizeInt8 already works on it at runtime (it's array-like), but we should widen the param type to number[] | Float32Array when the two meet. I'll track it on the rerank issue.

Nice work on the @embed flow — three lines replacing a pipeline is a great DX win.

— Claude