feat(core): expose references to the public content query API - #2509
feat(core): expose references to the public content query API#2509MA2153 wants to merge 5 commits into
Conversation
A reference field's selections are edges in _emdash_content_references, not a column on the collection's table. The registry skips column DDL for storage-less field types, the schema handlers own the backing relation's lifecycle (created with the field, destroyed with it, target collection immutable), and the previously unregistered relation and reference-edge routes are wired into injectCoreRoutes. A storage-less field never appears in `data` in either direction: it is excluded from the generated Zod shape (so a required reference field is satisfiable at all), rejected with a VALIDATION_ERROR when a caller sends one, and filtered out of reads so a column left behind by an older version cannot round-trip back into a save. That replaces the reference-target existence pass in validateContentData, which validated a column-backed value that no longer exists. Seeds apply a reference field's $ref: value as an edge, so seed files keep working unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reference selections ride in the content create/update body under a `references` key and are written in the same transaction as the entry, so a child that fails to resolve aborts the whole save rather than leaving an entry with taxonomies, bylines and SEO already committed. The editor GET opts into hydrating the first page of each reference field's children. Two paths that used to lose edges now carry them: duplicating an entry copies its outgoing references onto the copy, and purging a row clears the group's edges only once no sibling — trashed ones included, they are still restorable — is left to own them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A reference field's `multiple: false` and `required` settings were UI-only: the content body and the edge endpoint would both happily write several children to a single-reference field, or clear a required one. Both checks now live in `setReferenceChildren`, the one function every edge write passes through, so the content body, the standalone endpoint and the seed engine can't drift from each other. The create path additionally rejects a payload that omits a required reference field altogether, which no edge write would otherwise visit. Updates keep partial semantics: a field the payload doesn't mention is left alone. A relation with no reference field behind it stays unconstrained — the relations API can create one directly, and it carries no field config to enforce. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`getEmDashEntry` and `getEmDashCollection` now attach `entry.references`, keyed by field slug, with each referenced entry carrying its own `data` so a template can render a related entry without a follow-up query. Published targets only — an edge pointing at a draft resolves to nothing, exactly like one whose target was deleted, so a public read cannot leak an unpublished slug. Hydration is unconditional, so it has to be free when unused: the content query carries an uncorrelated existence probe on the edge table (the same shape as `_emdash_bylines_exist`), and an empty table stops hydration before it looks at the schema. Query counts are unchanged on every route in the snapshot; only the query text moves. A page of entries costs a constant number of queries — one batched edge read for the whole page, then one resolve per target collection. Generated types no longer describe a reference field as a string on `data`, and the `reference()` factory documents where the value actually lives. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 6758e0d The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![emdashbot[bot]](https://avatars.githubusercontent.com/in/3255304?s=60&v=4){kind=small}
There was a problem hiding this comment.
The PR solves a clear gap: reference selections were reachable in the admin but invisible to public templates. The approach — a folded existence probe in the content query, one batched edge read per page, and per-target-collection resolution — is consistent with how EmDash already hydrates bylines and taxonomy terms, and the dedicated query-count test is the right shape (real SQLite, real loader, actual SQL logged).
I found two cache-correctness issues that don't show up in the uncached query-count tests but will bite in production:
-
getReferenceFieldsrelies onCacheNamespace.SCHEMA, but schema mutations never invalidate it. The PR description says "Schema mutations already bustCacheNamespace.SCHEMA", butinvalidateSchemaObjectCache()is only used for URL-pattern invalidation inquery.ts;createField/updateField/deleteFieldetc. do not call it. After adding a reference field, public reads may keep serving the cached empty field list until the SCHEMA entry expires. -
Cross-collection references are cached without invalidation.
entrySnapshot/reviveEntrystore hydratedreferencesinside the collection-scoped L2 cache. Publishing or editing a referenced child entry invalidates the child's content namespace, not the parent's, so parent pages can serve stalereferencesuntil the parent's own cache entry expires. Bylines and taxonomies avoid this by includingCacheNamespace.BYLINES/TAXONOMIESincontentNamespaces; references have no equivalent hook.
Both are needs_fixing correctness issues. Fix option for #1 is to stop using the distributed SCHEMA cache for reference-field metadata and rely on the per-request cache (the edge-table probe already prevents the lookup when no references exist). Fix option for #2 is to stop caching references in the snapshot and rehydrate after revive.
The rest of the change is solid: query counts are held flat for uncached logged-out routes, the locale-fallback logic is correct, and the type changes accurately reflect that reference values live on entry.references, not in entry.data. I am not requesting changes because these are correctness/staleness regressions rather than security, data loss, or build breaks; a maintainer should decide whether to address them before merge or in a fast follow-up.
Findings
-
[needs fixing]
packages/core/src/schema/reference-fields.ts:114getReferenceFieldscaches underCacheNamespace.SCHEMA, and the comment above it claims schema mutations already invalidate that namespace. That is not true:invalidateSchemaObjectCache()is defined but is only called frominvalidateUrlPatternCache()inquery.ts; no field/collection create/update/delete path calls it. After an admin adds a reference field, public reads may continue to see the old empty field list until the SCHEMA cache entry expires.The cheapest fix is to stop distributed-caching this metadata. The request cache still dedupes multiple calls per render, and the folded
_emdash_references_existprobe means the lookup is skipped entirely when the edge table is empty, so a site without references pays nothing.export function getReferenceFields(collection: string): Promise<ReferenceFieldConfig[]> { return requestCached(`reference-fields:${collection}`, async () => referenceFields(await getDb(), collection), ); } -
[needs fixing]
packages/core/src/query.ts:691entrySnapshotcopiesreferencesinto the cached snapshot. The content cache key is scoped to the parent collection (contentNamespaces(type)), so publishing or editing a referenced child in a different collection only invalidates the child collection's cache — the parent page keeps serving the oldreferencesarray.Bylines and taxonomies avoid exactly this by including
CacheNamespace.BYLINESandCacheNamespace.TAXONOMIESincontentNamespaces. There is no equivalent cross-collection hook for references.Drop
referencesfrom the snapshot so cache hits are rehydrated from the current edges, just like cache misses:function entrySnapshot<D>(entry: ContentEntry<D>): Record<string, unknown> { const data = entryData(entry); const rawCursor = Reflect.get(data, CURSOR_RAW_VALUES); // Drop the `edit` function and `references`; both are derived per-render. const { edit: _edit, references: _references, ...rest } = entry as ContentEntry<D> & { edit?: unknown; }; return { ...rest, data: { ...data, [CURSOR_RAW_FIELD]: rawCursor ?? {} }, }; }After this change, callers that revive cached snapshots need to run
hydrateEntryReferences(and initializeentry.references = {}first if the type is strict) so a cache hit does not leavereferencesundefined.
Scope checkThis PR changes 4,022 lines across 42 files. Large PRs are harder to review and more likely to be closed without review. If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs. See CONTRIBUTING.md for contribution guidelines. |
github-actions
Bot
added
area/core
size/XL
review/awaiting-author
@emdash-cms/admin
@emdash-cms/auth
@emdash-cms/auth-atproto
@emdash-cms/blocks
@emdash-cms/cloudflare
@emdash-cms/contentful-to-portable-text
emdash
create-emdash
@emdash-cms/gutenberg-to-portable-text
@emdash-cms/plugin-cli
@emdash-cms/plugin-types
@emdash-cms/registry-client
@emdash-cms/registry-lexicons
@emdash-cms/registry-verification
@emdash-cms/sandbox-workerd
@emdash-cms/x402
@emdash-cms/plugin-ai-moderation
@emdash-cms/plugin-atproto
@emdash-cms/plugin-audit-log
@emdash-cms/plugin-color
@emdash-cms/plugin-embeds
@emdash-cms/plugin-field-kit
@emdash-cms/plugin-forms
@emdash-cms/plugin-webhook-notifier
commit: |
github-actions
Bot
added
review/needs-rereview
Overlapping PRsThis PR modifies files that are also changed by other open PRs:
This may cause merge conflicts or duplicated work. A maintainer will coordinate. |
1 participant
What does this PR do?
Closes the first of the two P1 findings on #1928 (r3757946584): a site could select a reference in the admin and had no way to render it.
getEmDashEntryandgetEmDashCollectionnow attachentry.references.Keyed by field slug, not by relation group. The relation group is an internal id a site author never sees; the field slug is what they named. (The admin's editor GET keys by relation group — it addresses fields by relation, and that stays as it is.)
Each reference carries the target's
data. The batched resolve already has the full row, so returning a summary and making templates re-query for a title would cost more and do less.Published targets only. An edge pointing at a draft resolves to nothing and is skipped, exactly like an edge whose target was deleted — a public read must never leak an unpublished entry's slug through a reference.
Locale. References are keyed by translation group, so a target that exists only in another language is still a real reference. Resolution prefers the reading entry's locale and falls back, reporting which variant answered in
localerather than presenting a wrong-locale entry as if it matched.Making unconditional hydration free
Hydration is always on, so the cost when a site doesn't use references has to be zero round trips — not "one cheap one". A first attempt did a schema lookup per render to discover whether the collection had reference fields, and
pnpm query-countscaught it immediately:The fix is the existing folded-probe pattern: the content query carries
(SELECT 1 FROM _emdash_content_references LIMIT 1), an uncorrelated subquery evaluated once per statement, the same shape as_emdash_bylines_exist. An empty edge table means no field anywhere can have a selection, so hydration stops before it touches the schema.Final snapshot: every route's query count is unchanged, including that one.
scripts/query-counts.snapshot.sqlite.jsonhas no diff at all; onlyquery-counts.queries.sqlite.jsonmoves, because every content query now carries the probe.For a site that does use references, a page costs a constant number of queries — one batched edge read for the whole page (
getChildrenForParents, chunked atSQL_BATCH_SIZE), then one resolve per target collection, so two fields pointing at the same collection share it.Types
The generated collection interface no longer lists reference fields — they hold no value in
data, sorelevant_posts?: stringwas describing something that never existed.entry.referencesis typed onContentEntry.fieldTypeToTypeScriptreturnsneverfor a reference now, so anyone reaching for that type gets an error rather than a plausible-looking string. Thereference()factory documents where the value actually lives.Part of #386.
Stack
required/multipleenforcementType of change
Checklist
pnpm typecheckpassespnpm lintpassespnpm testpasses (or targeted tests for my change)pnpm formathas been runmessages.pochanges are included.emdash: minor)AI-generated code disclosure
Screenshots / test output
tests/unit/query-references-hydration.test.tsdrives the real loader against a real SQLite database with a query counter on Kysely'sloghook — no repository mocks, so the query-count assertions are against actual SQL:data, in edge order, on both the collection and the single-entry path;_emdash_fieldslookup;