Add incremental global-data state and docs navigation indexing

docs/data/README.md

@@ -18,7 +18,7 @@ For output definitions, see [Generation](../generation/); for ordinary configura
 
 The `global.data.ts` file is an optional file that can live anywhere in your `src` tree.
 The first one found wins and duplicates warn.
-It runs **once per build**, after [source-backed pages](../pages/#page-files) are initialized and before generated-page factories run.
+Its callback runs **once each time DOMStack builds pages**, after [source-backed pages](../pages/#page-files) are initialized and before generated-page factories run.
 
 > [!NOTE]
 > `global.data.js` works too.
@@ -27,6 +27,7 @@ See [Supported file types](../typescript/#supported-file-types) for all availabl
 For data that aggregates across multiple pages — like blog indexes, sitemaps, recent-post lists, or RSS feed content — use `global.data.ts`.
 It is the only public build hook that receives the source-backed `PageData[]` collection.
 It returns an object of named, top-level values that downstream consumers can explicitly subscribe to.
+The default export can be an object or a synchronous or asynchronous function that returns one.
 
 ```typescript
 // src/global.data.ts
@@ -142,9 +143,8 @@ When one layout calls another layout function directly, the composing layout mus
 - Gives pages, layouts, templates, and page factories only their declared top-level keys through `data`.
 - Keeps global data separate from ordinary `vars`, so derived values cannot silently collide with page or layout configuration.
 - Runs inside the worker process (same as all other dynamic imports) to avoid ESM caching issues.
-- Skipped entirely if no `global.data.*` file exists — zero overhead.
-- In watch mode, DOMStack fingerprints each top-level returned value and rebuilds only consumers subscribed to changed keys.
-- Editing `global.data.*` or one of its statically imported helpers recomputes data; a shared helper also rebuilds its direct page, layout, template, and factory consumers.
+- During targeted watch rebuilds, DOMStack compares each top-level returned value with the previous build and also rebuilds consumers subscribed to values that changed.
+- Editing `global.data.*` or its watched imports recomputes the shared data.
 - Values composed of JSON-safe primitives, arrays, and plain objects get stable fingerprints; opaque values such as functions, class instances, maps, sets, or cycles conservatively invalidate their subscribers on every page build.
 - A declaration naming a missing key fails the build, and access to an existing but undeclared key throws a focused error.
 
@@ -180,6 +180,7 @@ export default globalData
 Use `AsyncGlobalDataFunction<DerivedData>` instead when the implementation needs to await rendering, network requests, or other asynchronous work.
 For typed source input, use `GlobalDataFunction<Result, SourceVars, SourceContent>` or its async counterpart.
 Helpers can accept `GlobalDataFunctionParams<SourceVars, SourceContent>['pages']` without recovering types from the full global-data result.
+For callbacks that cache results between watch builds, see [Incremental global data](#incremental-global-data).
 
 ## Global data caveats
 
@@ -276,6 +277,7 @@ The current `page` is a `PageInfo` object with the following properties:
 - `generated`: Metadata about the `*.pages.ts` file that created a generated page, or `undefined` for a source-backed page.
 
 Each `PageData` entry supplied to `global.data.ts` exposes this object as `page.pageInfo`.
+When caching entries by source page, use `page.sourceId` as the key and `page.pageInfo.url` as the output URL.
 Combine `page.pageInfo.url` with a `siteUrl` from `global.vars.ts` to build an absolute URL: `` `${vars.siteUrl}${page.pageInfo.url}` ``.
 The [RSS and JSON feed recipe](../cookbook/feeds/) uses this pattern for feed item URLs.
 
@@ -347,3 +349,95 @@ export default globalData
 
 Rendering performed inside `global.data.ts` cannot use the derived values that the same file is still computing.
 After `global.data.ts` returns, consumers receive only the values named by their `dataDeps` declarations.
+
+## Incremental global data
+
+> [!NOTE]
+> Most sites do not need incremental global data.
+> Start with a regular `global.data.ts` callback and consider incremental indexing when a large number of pages makes watch rebuilds slow.
+
+An incremental index caches results for each source page so you can update affected entries instead of processing every page on every watch rebuild.
+Keep the index in saved state and return the shared values your pages need, such as a navigation list.
+Saved state is private to the callback; it is not exposed through `data`.
+
+The callback receives four fields:
+
+- `pages`: All current source pages, excluding generated pages.
+- `previousState`: A copy of your saved state, or `undefined` when starting fresh.
+  You can modify it, but must call `setState` to save your updates.
+- `changes`: On `kind: 'reset'`, rebuild the index from `pages`.
+  On `kind: 'delta'`, update entries for the new or affected pages in `changes.upserted` and delete the IDs in `changes.removed`.
+- `setState(next)`: Take a snapshot of your index to reuse on the next build.
+  DOMStack keeps that snapshot only if the current build succeeds.
+
+Use `page.sourceId` as the index key: a read-only source-relative path such as `docs/data/README.md`, using `/` separators on every platform.
+`changes.removed` contains these same IDs.
+
+### Simple documentation index
+
+This example caches titles and URLs for Markdown pages under `/docs/` and returns a sorted navigation list.
+
+```typescript
+// src/global.data.ts
+import type { GlobalDataFunctionParams } from '@domstack/static/types.js'
+
+type SourceVars = { title?: string }
+type Entry = { title: string, url: string }
+type Index = Map<string, Entry>
+
+export default function ({
+  pages, previousState, changes, setState,
+}: GlobalDataFunctionParams<SourceVars, string, Index>) {
+  let index: Index
+  let inputs: typeof pages
+  switch (changes.kind) {
+    case 'reset':
+      index = new Map()
+      inputs = pages
+      break
+    case 'delta':
+      index = previousState ?? new Map()
+      inputs = previousState === undefined ? pages : changes.upserted
+      for (const sourceId of changes.removed) index.delete(sourceId)
+      break
+    default:
+      throw new Error('Unhandled global-data changes', { cause: changes satisfies never })
+  }
+
+  for (const page of inputs) {
+    const { url, type } = page.pageInfo
+    if (type !== 'md' || !url.startsWith('/docs/')) {
+      index.delete(page.sourceId)
+      continue
+    }
+    index.set(page.sourceId, { title: page.vars.title ?? url, url })
+  }
+
+  setState(index)
+  return {
+    docsNavigation: [...index.values()].sort((a, b) => a.url.localeCompare(b.url)),
+  }
+}
+```
+
+A layout subscribes with `export const vars = { dataDeps: ['docsNavigation'] }` and reads `data.docsNavigation`.
+If you edit a Markdown page without changing its title or URL, the navigation stays the same, so other pages do not rebuild just because they subscribe to it.
+For an example that also caches heading links, see the [documentation site's index](https://github.com/bcomnes/domstack/blob/master/site/globals/global.data.ts).
+
+The third type argument in `GlobalDataFunctionParams<SourceVars, SourceContent, State>` describes the saved state—`Index` in this example.
+For `GlobalDataFunction` and `AsyncGlobalDataFunction`, `State` is the fourth type argument, after the result, source-vars, and source-content types.
+It defaults to `unknown`.
+Helpers that process `changes` can use the exported `GlobalDataChanges` type.
+
+### Usage notes
+
+- State lasts for one watch session, not across process restarts.
+  Always handle `changes.kind === 'reset'`, including after failed builds.
+- Markdown and HTML source edits can update individual entries.
+  Changes to watched modules, including JavaScript/TypeScript pages, helpers, browser entries, and settings, reset the saved index and rebuild the site.
+- Store structured-cloneable values such as plain records, arrays, or maps—not `PageData` instances, functions, or native objects with shared mutable storage.
+- A page can appear in `changes.upserted` even when its content is unchanged.
+  Replace its cached entry, or delete it if the page no longer belongs in your index.
+- Restart watch mode after changing imported JSON, files read with `fs.readFile()`, environment variables, or other inputs outside page dependency tracking.
+- Static re-exports (`export … from`) are not followed by dependency tracking ([#328](https://github.com/bcomnes/domstack/issues/328)).
+  Use an explicit import followed by a local export, or restart watch mode after those dependencies change.

docs/implementation/README.md

@@ -229,7 +229,7 @@ Require a full page retry`"]
 
 Bundle replanning returns only a page plan or a skip; it does not restart esbuild again.
 After a page-build failure, the next page-producing plan retries the complete page phase rather than trusting incremental filters.
-Manifest-settings changes and service-worker entry additions or removals retain their intentional page-phase skips.
+Manifest-settings changes and service-worker entry additions or removals skip page rendering unless an incremental global-data index needs to be reset.
 The one-shot manifest pipeline is not part of watch execution.
 
 > [!NOTE]
@@ -245,11 +245,14 @@ DOMStack uses these rebuild scopes:
 - **Full page/template rebuild**: DOMStack renders every source-backed and generated page and every template without restarting esbuild.
 - **Full rebuild**: DOMStack rediscovers the source tree, restarts esbuild, renders all pages and templates, and refreshes its dependency maps.
 
-Like templates, generated-pages modules rebuild when their own source or imported dependencies change.
+Like templates, generated-pages modules rebuild when their own source or watched imported dependencies change.
 When a targeted build recomputes global data, DOMStack compares top-level values with the previous successful build and adds only subscribers of changed keys to the rebuild set.
 
 ### What triggers what
 
+The tables below apply when `global.data.ts` does not save state with `setState`.
+With an [incremental index](../data/#incremental-global-data), Markdown and HTML edits can update individual entries, while changes to watched modules reset the index and trigger a **full rebuild**.
+
 | Change | Rebuild scope |
 |---|---|
 | Existing `page.ts`, `page.html`, `page.md`, or adjacent `page.vars.ts` | That page, plus subscribers of any changed global-data keys |
@@ -280,7 +283,8 @@ When a full page/template rebuild or targeted generated-pages rebuild no longer
 ### Dependency tracking
 
 DOMStack uses [`@11ty/dependency-tree-typescript`](https://github.com/11ty/dependency-tree-typescript) to statically analyze ESM imports.
-It maintains maps for:
+Imported JSON changes do not trigger page rebuilds; restart watch mode after editing those files.
+DOMStack maintains dependency maps for:
 
 - Layout dependencies, source-backed pages using each layout, and generated-page owner layout membership
 - TypeScript pages and adjacent page-variable dependencies
@@ -294,8 +298,8 @@ Dependency analysis is best-effort.
 When DOMStack cannot safely determine a targeted scope, it falls back to a broader rebuild or skips an unrelated changed module.
 
 esbuild tracks browser-entry dependencies independently.
-Changing a module imported only by `client.ts` rebundles that entry without rendering page HTML.
-When a module has both browser and server-side consumers, the planner unions the server-side consumers rather than skipping the page phase.
+A module imported only by `client.ts` normally triggers rebundling without rendering page HTML.
+If `global.data.ts` has saved an incremental index, the module edit also resets that index and triggers a full rebuild.
 
 ### Stable entry filenames
 
@@ -322,7 +326,7 @@ Page HTML points to stable entry files during watch mode. esbuild can update an
 ### Manifest behavior
 
 Watch mode builds and rebundles the site service worker, but it does not finalize, return, or write the [DOMStack manifest](../../docs/workers/#domstack-manifest).
-Changes to `domstack-manifest.settings.ts` therefore do not trigger a watch rebuild.
+Editing `domstack-manifest.settings.ts` only triggers a rebuild when an incremental global-data index needs to be reset; it does not generate a manifest.
 
 Use `domstack --serve` when testing manifest-driven cache behavior.
 It runs a one-shot build and serves the result without watch-mode filenames or live-reload HTML injection.