Support incremental global-data state and batched input changes using existing fingerprint invalidation

[bcomnes]

Motivation

Follow-up from the Oro website migration: oro-computer/website#1

Application plan and design discussion: oro-computer/website#1 (comment)

Oro now uses page-owned Markdown outputs from #323/#322, so an ordinary docs body edit no longer rewrites a collection of raw files. But global data still reads and renders all 585 source documents to build search text on every page-build pass. Existing global-data fingerprinting correctly keeps most downstream outputs unchanged; the expensive producer work has already happened.

Support application-owned incremental indexes through explicit producer state and batched input-change context. The application decides what to cache, how to group/order it, and how to derive public data. Existing fingerprints and dataDeps must remain the only downstream data invalidation system.

Existing behavior inspected in 12.0.0-beta.6

• index.js queues individual watcher handlers; builds are serialized, not exposed as coalesced producer-input batches.
• lib/build-pages/index.js initializes all source pages, calls global data, then compares top-level fingerprints and expands output filters for subscribers.
• lib/build-pages/resolve-vars.js calls the global-data function with only { pages }.
• Each page-build pass creates a new worker, so a module-local Map in global.data does not persist across builds.
• lib/build-pages/watch-dependencies.js fingerprints public top-level data values with stable JSON/SHA-256 and selects consumers through declared subscriptions. Its state is not an application indexing cache.

Preserve one invalidation pipeline

Existing watch/dependency planning
  → potentially affected source inputs
  → application updates cached index
  → ordinary returned global data
  → existing fingerprint comparison
  → existing dataDeps consumer selection

Input changes answer which values may need recomputation. Fingerprints answer which published values actually changed. A hash cannot supply the previously rendered text needed to avoid recomputing it.

Do not add a parallel dependency graph, app-owned output hashes, changedKeys declarations, or manual consumer invalidation. Initially continue fingerprinting all returned top-level values. An application can cheaply reconstruct an affected collection’s navigation and let its unchanged fingerprint suppress downstream work, instead of duplicating field-by-field invalidation rules.

Illustrative API—not a finalized contract

Extend the existing function context without changing the meaning of its return value:

export default async function globalData({
  pages,
  previousState,
  changes,
  setState,
}) {
  const next = await updateIndex(previousState, pages, changes)
  setState(next.state)
  return next.data
}

• Existing functions using only { pages } and static object exports remain valid.
• pages remains the complete current source-backed collection. Generated pages remain downstream.
• previousState is private, structured-cloneable application state from the last successful build; it is not public data and is not fingerprinted as a global-data key.
• setState(next) stages candidate state; it does not immediately commit it.
• The returned object is ordinary global data and follows the unchanged fingerprint/subscription path.

A reset-or-delta shape could be:

type Changes =
  | { kind: "reset"; reason: string; events: WatchEvent[] }
  | {
      kind: "delta"
      upserted: PageData[]
      removed: string[]
      events: WatchEvent[]
    }

Exact names and generics are open. Use normalized source identities, not URLs or titles. Upsert means potentially affected, not guaranteed different. Renames are remove plus add. Raw events may accompany the batch for diagnostics/app-specific decisions, but semantic invalidation is the correctness contract.

Reuse and extend existing dependency tracking

Derive affected source pages from the existing planner and dependency maps, not a second graph:

Change	Safe producer invalidation
Source body/frontmatter or page vars/helper	Upsert affected pages; H1 edits can change inferred title
Layout vars, renderer, parent chain, or imports	Upsert affected source-page consumers
Markdown settings or imports	Invalidate Markdown inputs, or reset initially
Global vars or imports	Reset initially
Global-data module or transitive producer imports	Reset, since algorithm/state shape may have changed
Adds/deletes/draft transitions/discovery changes	Rediscover and reconcile actual eligible source membership
Unknown or unreliable dependency information	Reset

A full output rebuild is not inherently a producer-state reset; structural rediscovery may yield a useful delta. Conversely, a small helper edit can invalidate many inputs.

Extend dependency coverage for global-vars/settings imports: beta.6 does not track these roots as comprehensively as page/layout/global-data imports. Do not imply that arbitrary filesystem reads, external data, environment changes, or dynamic imports are automatically tracked. Document the supported boundary and require explicit invalidation/restart for unsupported inputs.

Batching and state lifecycle

1. Accumulate events and detach one batch when work begins.
2. Union affected inputs against the last successful baseline. Reconcile discovery membership rather than retaining only the last event for each path; editor atomic saves must work.
3. Keep events arriving during a build pending for a subsequent batch. Cover events during startup too.
4. Pass an isolated candidate state into the build worker.
5. Commit candidate state alongside the successful dependency/fingerprint/input baseline only after the relevant page build and cleanup succeed.
6. On failure retain the previous baseline; retry accumulated changes or conservatively reset. A later rendering failure must not leave application state ahead of subscriber state.

Filesystem events are not a snapshot: promise convergence when changes settle, not exact historical replay or transactional filesystem outputs.

For v1, keep structured-cloneable state in the owning watch session and round-trip it through the existing fresh build workers. Persist projected records, Maps, arrays, and strings—not PageData, functions, or parser instances. New sessions/clean builds start empty; producer changes reset; worker failure discards candidates. Report uncloneable state clearly. Stop/restart should release session state.

Persistent workers could reduce transfer costs but introduce module reload and rollback complexity; defer them.

Oro adoption

Oro will keep a source-keyed processed-document index (metadata, Markdown, search text, URLs) with collection membership/order referencing source IDs. It will replace/remove affected entries, rebuild affected collection projections, and return the existing navigation/search/LLM/redirect/route keys. Page-owned raw outputs remain unchanged.

No giant public index subscription, no app-specific collection DSL in DOMStack, and no requirement to declare exact changed public keys. The application controls indexing; DOMStack controls consumer invalidation through existing fingerprints.

Focused acceptance criteria

• Existing global-data functions/object exports behave unchanged.
• State survives successive per-build workers and is isolated between watch sessions.
• One ordinary body edit performs one application index/search render; unchanged navigation fingerprints do not invalidate sibling docs.
• H1/explicit-title precedence, companion/inherited vars, and transitive renderer/settings/producer imports invalidate safely.
• Adds, deletes, renames, draft transitions, and changed collection/redirect membership converge to the clean result.
• Buffered events and edits arriving during builds are not lost; atomic saves work.
• Producer and later-render failures followed by recovery match a clean build; removed global-data keys retain existing semantics.
• Invalid/uncloneable state reports actionable errors.
• Incremental public data and output match a fresh build after each scenario.

Measure Markdown reads/renders separately from source initialization, state transfer, projection, fingerprinting, and output writes. This proposal initially leaves all-page initialization, cloning, serialization, and fingerprint traversal in place. The target is repeated expensive producer computation, not a claim that the entire build becomes O(changed pages).

Related: #289 (output dependency tracking), #322/#323 (page-owned additional outputs).