Plugin-Generated Thumbnails & Metadata for Non-Image Media

[MA2153]

Summary

Let plugins generate a thumbnail image and structured metadata for non-image
media (PDFs, office documents, CAD files, archives, …). The thumbnail is a real
raster image stored in the media bucket, so it flows through the existing
Astro/EmDash image-optimization pipeline unchanged and is available both in the
admin UI and to downstream frontend consumers — with a blur-up (LQIP) placeholder
just like uploaded images.

This is not image optimization. Optimization already exists for raster media.
This adds a generation step that turns an otherwise un-previewable file into an
image the existing pipeline can then optimize.

Motivation

Today, non-image media has no preview anywhere:

• Admin (packages/admin/src/components/MediaLibrary.tsx) falls back to an emoji
  icon via getFileIcon() for anything that isn't image/*.
• Frontend rendering of a non-image MediaValue through
  EmDashMedia.astro → local-runtime.getEmbed() falls through to "treat as image",
  producing a broken <img> pointing at the raw file bytes.

A PDF, a slide deck, or a Word document should show a recognizable preview in the
library grid, in content fields, and on the published site. Generating those
previews requires format-specific rendering (often heavy native libraries or an
external service), which is exactly the kind of thing plugins exist to provide.

Goals

• A plugin can register one or more media enrichers that produce a thumbnail
  image and/or metadata for media it claims by MIME type.
• Generation is deferred and out-of-band — uploads stay fast, and heavy or
  Workers-incompatible rendering can run via an external service.
• The generated thumbnail is a real stored image that the existing image
  pipeline optimizes with zero special-casing, including a generated LQIP
  placeholder.
• Previews and metadata are available in the admin UI and to frontend
  consumers.
• Purely additive and backwards compatible (EmDash is pre-1.0 but published
  and in active use; migrations are forward-only).

Non-goals (v1)

• Fully asynchronous external rendering with webhook callbacks (submit job →
  callback minutes later). The data model leaves room for it; v1 does not ship it.
• Multiple enrichers contributing/merging metadata for one media item.
• Multi-page / multi-asset preview sets (one thumbnail image per media item in v1).
• An admin-configurable enricher selection UI beyond plugin-provided settings.

Current architecture (relevant facts)

These findings shaped the design:

1. The image pipeline keys off a storage key alone. image-endpoint.ts and
   media/url.ts resolve /_emdash/api/media/file/{key} and read bytes straight
   from the storage adapter. There is no requirement that an optimizable image
   correspond to a media table row. → A derived thumbnail only needs to be a
   stored object, not a library entry.

2. The hot render path does not join media. Frontend rendering uses the
   cached MediaValue.meta (storageKey, blurhash, dominantColor) via
   getEmbed() — no DB hit per render. → Putting enrichment on a sidecar table
   does not add a join to the hot path.

3. MediaValue is cached at selection time. Because enrichment is async, a
   thumbnail generated after a file was referenced in content will not be in the
   stored MediaValue. → First-class frontend support requires render-time
   resolution of enrichment by media id, regardless of where it's stored.

4. LQIP precedent exists. Uploaded images get blurhash + dominant_color
   (migration 024_media_placeholders.ts), generated at upload via
   media/placeholder.ts generatePlaceholder(). The same helper can run over a
   generated thumbnail's bytes.

5. A core-owned recurring sweep precedent exists. runScheduledTasks()
   (emdash-runtime.ts) already aggregates cronExecutor.tick() +
   recoverStaleLocks() + publishDueContent() — batched, idempotent, driven by
   the Node timer and the Cloudflare scheduled() handler. Enrichment slots in
   beside it with no new scheduler infrastructure.

6. Storage keys are validated flat for transforms. isSafeTransformKey() /
   matchInternalMediaKey() in image-endpoint.ts use
   SAFE_STORAGE_KEY = /^[A-Za-z0-9._-]+$/, which rejects slashes. The raw file
   route media/file/[...key].ts does not validate (its [...key] rest param
   accepts slashes). → A namespaced thumbnails/… key serves fine but would not be
   optimized unless the transform validator is widened (see below).

Design

1. Data model — _emdash_media_enrichment sidecar table

A new system table, one row per media item in v1 (UNIQUE(media_id)). It also
doubles as the work queue: status = 'pending' rows are pending work.

column	type	notes
id	text PK	ULID
media_id	text	FK → media.id; indexed; UNIQUE for v1
enriched_by	text?	enricher id (provenance); nullable to leave room for async callback model
status	text	pending | ready | failed
thumbnail_storage_key	text?	thumbnails/{ulid}.{ext} — a real stored image
thumbnail_width	integer?	actual rendered width (result, not input)
thumbnail_height	integer?	actual rendered height
thumbnail_blurhash	text?	LQIP, generated by core from the thumbnail bytes
thumbnail_dominant_color	text?	LQIP fallback color
metadata	text (JSON)?	plugin-extracted metadata (open vocabulary, see below)
error	text?	last failure message
attempts	integer	retry/backoff bookkeeping, default 0
locked_at	text?	sweep lock (mirrors _emdash_cron_tasks)
created_at	text	defaultTo (datetime('now'))
updated_at	text

Indexes: idx_media_enrichment_media_id (unique), and a "find work" index on
(status, locked_at) to drive the reconciliation sweep.

Rationale for a sidecar table over columns on media (alternatives below):
enrichment is a record with its own lifecycle (pending/ready/failed), its own
provenance (which enricher), open-ended metadata, and applies only to a
subset of media — unlike the fixed scalar blurhash/dominant_color columns,
which are intrinsic to every image. A new table is purely additive and forward-only.

2. Plugin interface — MediaEnricher

Declared on the plugin definition, the same way media providers are declared, so
core can build a registry and select an enricher without executing plugin code:

interface MediaEnricher {
  /** Unique id; stored as `enriched_by` for provenance. */
  id: string;
  /** MIME types this enricher claims — exact (`application/pdf`) or prefix (`application/`). */
  mimeTypes: string[];
  /** Higher wins when multiple enrichers match. One enricher per media in v1. */
  priority?: number;
  /**
   * Produce a thumbnail and/or metadata for the given media item.
   * Runs in the deferred sweep, not the upload request path.
   * The plugin may read its own (admin-configurable) settings from `ctx` to
   * decide target size/format; output dimensions are whatever it returns.
   */
  enrich(media: EnrichTarget, ctx: PluginContext): Promise<{
    thumbnail?: { bytes: ArrayBuffer; mimeType: string; width?: number; height?: number };
    metadata?: Record<string, unknown>;
  }>;
}

// On the plugin definition, alongside `mediaProviders`:
interface PluginDefinition {
  // ...
  mediaEnrichers?: MediaEnricher[];
}

• Core-orchestrated, not plugin-orchestrated. The plugin declares mimeTypes
  and implements enrich(). Core owns selection, scheduling, storage write, LQIP
  generation, status, and retries. The plugin needs no new write capability —
  it returns bytes + metadata and core persists them.
• Selection is deterministic. Among enrichers whose mimeTypes match the
  uploaded file, the highest priority wins; ties broken by registration order.
• Workers compatibility. Since sharp / PDF renderers can't run in-isolate, an
  enricher running on Cloudflare uses ctx.http (the network:fetch capability)
  to call an external rendering service inside enrich().
• Configurable output. Anything a plugin author wants the site admin to control
  (output dimensions, format, quality, which sub-formats are handled) is read from
  the plugin's own settings inside enrich(). Output dimensions are results
  stored in the table, never declared in the contract.

Enrichers are intentionally a separate concept from media providers. Providers
are sources (where media comes from, often with their own previews); enrichers
are transforms of already-stored local media. Folding enrichment into provider
getThumbnailUrl() would conflate the two and make no sense for external-provider
media.

3. Execution model — sidecar-as-queue, two triggers

On upload (routes/api/media.ts, after handleMediaCreate):
if the enricher registry contains an enricher whose mimeTypes match the new
file, core inserts a pending enrichment row stamped with the winning
enriched_by, then fires after() to process it immediately. The after() work
is mostly awaiting an external service, so CPU stays low even on Workers. Inserting
the row synchronously (before after()) is the durable at-least-once anchor.

Reconciliation sweep — a new processPendingEnrichments(db) added to
runScheduledTasks() beside publishDueContent():
batched and idempotent, it claims pending (and retryable failed) rows whose
locked_at is null/stale, processes them, and bounds retries via attempts. This
drains anything after() missed (Worker killed mid-flight, redeploy, backfill) and
is the same stale-lock pattern the cron executor already uses. Driven by the Node
timer and the Cloudflare scheduled() handler — no new scheduler.

Processing a row:

1. Load the enricher by enriched_by; load the media row.
2. await enricher.enrich(media, ctx).
3. If thumbnail returned: write bytes to storage under
   thumbnails/{ulid}.{ext} (ext from returned mime); run generatePlaceholder()
   over the bytes for blurhash + dominant_color; record thumbnail_* columns.
4. Persist metadata; set status = 'ready', clear locked_at.
5. On error: set status = 'failed', store error, increment attempts
   (retryable until a cap, then terminal).

4. Thumbnail storage & the thumbnails/ prefix

Generated bytes are written to the same Storage adapter as uploaded media
(emdash.storage → local dir / R2 / S3), by core, under a namespaced key:

thumbnails/{ulid}.{ext}      e.g. thumbnails/01J9Z…X7.webp

• The thumbnails/ prefix keeps derived images from being lumped with originals in
  the bucket. The object is referenced only by the sidecar's
  thumbnail_storage_key — it is not a media table row, so it never appears
  in the library grid.
• Required change for optimization: the transform-key validator must permit
  exactly one reserved prefix segment. isSafeTransformKey() and
  matchInternalMediaKey() in image-endpoint.ts change from
  /^[A-Za-z0-9._-]+$/ to /^(?:thumbnails\/)?[A-Za-z0-9._-]+$/ — still no ..,
  no nested slashes, no traversal. The raw file route already serves slashed keys
  via its [...key] rest param. This is a small, security-sensitive diff and the
  proposal calls it out explicitly for review.
• Lifecycle/cleanup: deleting the parent media row deletes the sidecar row and
  its stored thumbnail object, wired into the existing delete paths
  (handleMediaDelete and the local provider delete(), which already remove the
  original from storage).

5. Frontend consumption

Because MediaValue is cached at selection time (fact #3), the thumbnail is
resolved at render time by media id, cached per request:

• Add a FileEmbed variant to the EmbedResult union (additive, backwards
  compatible):

  interface FileEmbed {
    type: "file";
    src: string;            // original file URL (download / link target)
    mimeType: string;
    filename?: string;
    thumbnailSrc?: string;  // /_emdash/api/media/file/thumbnails/{ulid}.{ext} — optimizable
    thumbnailWidth?: number;
    thumbnailHeight?: number;
    blurhash?: string;      // LQIP for the thumbnail
    dominantColor?: string;
    metadata?: Record<string, unknown>;
  }

• local-runtime.getEmbed() becomes async for non-image MIME types: it looks up
  enrichment by media.id (wrapped in requestCached, so one query per media per
  render) and returns a FileEmbed. getEmbed already permits returning a Promise.

• EmDashMedia.astro gains a file branch: renders the thumbnail <img>
  (run through the image pipeline, with the blurhash placeholder behind it, exactly
  like images) linking to the original file. Falls back to the existing
  icon/download treatment when no thumbnail exists yet (pending/failed/none).

The thumbnail URL is an ordinary media-file URL, so responsive srcset/format
negotiation works without any new code.

6. Admin UI

• MediaLibrary.tsx / MediaDetailPanel.tsx: show the thumbnail when
  status = 'ready', a subtle spinner while pending, and the existing
  getFileIcon() emoji when there is no enrichment or it failed.
• The local provider list() already returns a meta bag; extend it with
  enrichment fields via a single LEFT JOIN to _emdash_media_enrichment.
• The detail panel surfaces extracted metadata. When the recommended vocabulary
  keys are present (below), they can render with friendly labels; unknown keys
  render generically.
• A "Regenerate preview" action re-enqueues (sets status = 'pending',
  clears lock/attempts).
• All new strings localized via Lingui; layout uses RTL-safe logical classes.

7. Backfill & lifecycle

• New uploads enrich automatically.
• Existing media: an on-demand admin action ("Generate preview"), plus an optional
  CLI batch command that inserts pending rows for media whose MIME matches a
  registered enricher and lets the sweep drain them in batches.

8. Metadata vocabulary

metadata is an open Record<string, unknown> — plugin-defined and never enforced.
Core documents a small recommended vocabulary that the admin renders specially
when present, for cross-plugin interop:

• pageCount (number)
• title (string)
• author (string)
• durationSec (number)
• dimensions ({ width: number; height: number })

Backwards compatibility & migration

• New _emdash_media_enrichment table — purely additive; no change to media
  semantics. Forward-only migration registered in runner.ts per project
  conventions (create the file, add the static import, add to getMigrations()).
• New FileEmbed union member and optional mediaEnrichers field — additive; no
  existing plugin or content breaks.
• The transform-key validator widening (thumbnails/ prefix) is a strict
  superset of the current pattern — previously valid keys remain valid.
• A changeset accompanies the published-package changes, leading with the
  observable effect ("Adds plugin-generated thumbnails and metadata for non-image
  media …").

Alternatives considered

• Thumbnail as a separate media-library entry (parent_id). Rejected: the
  image pipeline needs only a storage key (fact Bump ask-bonk/ask-bonk from c39e982defd0114385df54e72012a3fc4333c4d4 to 5704e1685d6efc3951efb9166a8bd17a9ca28509 #1), so this buys nothing for
  optimization while cluttering the grid and forcing parent_id filtering
  everywhere.
• Columns on the media table (like blurhash). Rejected as the primary model:
  enrichment has its own lifecycle, provenance, open-ended metadata, and applies to
  a subset of media — a poor fit for fixed columns on the hot table. (Kept as the
  simpler fallback if reviewers weight "smallest change" highest.)
• Plugin-orchestrated (plugin uses media:afterUpload + its own cron + a new
  media write-back capability). Rejected: pushes boilerplate and inconsistent
  lifecycle into every plugin; core-orchestration gives uniform status/retry/admin
  states.
• Fold into media provider getThumbnailUrl(). Rejected: conflates sources
  with transforms of stored media.

Open questions (for the Discussion)

1. External async / webhook rendering. v1 supports only enrich() that
   resolves with the result (awaiting a reasonably fast external call inside the
   sweep). The status + nullable enriched_by design leaves room for a later
   callback route that transitions a row to ready. Confirm this stays out of v1.
2. Transform-key validator widening. The thumbnails/ prefix requires
   loosening isSafeTransformKey()/matchInternalMediaKey() to a single reserved
   prefix. Is the proposed regex acceptable, or is a different namespacing scheme
   (e.g. a flat thumb_ filename prefix with no slash, avoiding any validator
   change) preferred?
3. Metadata vocabulary. Is the recommended-but-unenforced key set the right
   call, versus a fully open bag or a stricter typed schema?

Phased implementation sketch

1. Migration + repository: _emdash_media_enrichment table, indexes, a
   MediaEnrichmentRepository (claim/lock, mark ready/failed, find by media id).
2. Registry + types: MediaEnricher interface, mediaEnrichers on the plugin
   definition, registry built at plugin load.
3. Orchestration: selection at upload, pending insert + after(), and
   processPendingEnrichments() wired into runScheduledTasks(); storage write
   under thumbnails/…, LQIP via generatePlaceholder().
4. Validator widening for the thumbnails/ prefix, with tests proving no
   traversal regression.
5. Frontend: FileEmbed, async getEmbed() resolution with requestCached,
   EmDashMedia.astro file branch with LQIP.
6. Admin: list join, pending/ready/failed states, metadata display,
   "Regenerate preview" action (all localized, RTL-safe).
7. Backfill: admin action + CLI batch command.
8. Reference plugin (e.g. a PDF-first-page enricher) demonstrating the contract,
   plus unit/integration tests across dialects.

---

Claude helped me reason through the design and draft the proposal. I gave it a full-read before submitting.