Refactor DOMStack data boundaries and migrate tooling to TypeScript

[bcomnes]

Summary

Follow up on the DOMStack migration with native page-owned exports, narrower data boundaries, checked layout contracts, TypeScript-only tooling, and targeted client improvements. Preserve the static architecture, branding, article Markdown, existing HTML/raw routes, heading anchors, and upstream ownership policies. Retire unused public navigation JSON endpoints intentionally.

DOMStack data, outputs, and types

• Pin @domstack/static to 12.0.0-beta.6 and use native pageOutputs from the shared docs/spec layouts. Each page owns its raw Markdown file; six collection-wide source templates are removed. No framework patch or patch-package dependency.
• Preserve raw URLs and frontmatter-stripped Markdown bytes, including nested paths and .txt exports. Watch mode skips unchanged raw writes and cleans up renamed/deleted outputs.
• Keep one docs layout for all six collections and the dedicated spec layout. Body-only edits leave sibling HTML untouched; navigation metadata changes intentionally rebuild all ordinary docs through one lightweight navigation dependency.
• Prepare typed navigation and separate collection search/LLM projections in src/globals/global.data.ts; move global vars alongside it. Global search preparation still processes all docs: the page-output improvement fixes write amplification, not that computation.
• Co-locate search and LLM templates with their output directories. Keep sitemap, site LLM, CNAME, and hidden nojekyll templates at the root.
• Remove unused navigation index.json exports; retain search JSON used by the browser. Content audits derive expected raw outputs from committed Markdown metadata rather than removed JSON catalogs.
• Register all seven layouts through the native type-only LayoutRegistry, referencing actual exports. Validate all 20 HTML vars companions with ValidatePageVars through CheckedPageVars and actual global vars. No runtime registry or added subscriptions; Markdown metadata keeps runtime validation.
• Use package import aliases for shared lib callers; keep article transforms page-local and retain existing Markdown plugin policy.
• Derive edit links from actual source paths, honor site configuration, escape sitemap XML, and avoid fixed specification-year links.

TypeScript-only tooling

• Replace Python ingestion, content audits, subprocess bridging, and ingestion tests with TypeScript.
• Call the staged importer directly, preserving curated regions, ownership/exclusion precedence, pruning, missing-wiki behavior, and fenced examples.
• Convert remaining JavaScript clients, syntax grammars, and content helpers to checked TypeScript.
• Use a Node browser-test server for the already-built artifact.
• Update importer layout assignments, historical migration utility, and documentation. Historical migration still reads legacy checkout indexes where needed.

Clients, assets, and workflows

• Fix failed search-index caching when a pending query is cleared.
• Move Ask AI outside the scrolling navigation boundary and support keyboard focus/dismissal.
• Scope homepage/learn styles and route-specific clients; select the most-specific current subnavigation link.
• Remove Python setup and duplicate production-push audit runs; deploy the checked artifact and retain browser failure artifacts.
• Supply npm start, clean build, and development scripts.

Latest title and redirect changes

• Use destination-page redirectFrom metadata for the two existing path aliases, with validated collection data and a typed generated-pages factory. Redirects retain the full root layout, canonical metadata, query/fragment handling, and no-JavaScript fallback.
• Derive documentation titles from H1, normalize inline Markdown to plain text for metadata consumers, and remove all 585 redundant frontmatter titles. Article bodies/raw outputs remain unchanged; explicit title overrides remain supported and preserved by ingestion.
• Full local validation passed after these changes, including 42 unit/integration tests and 18 browser tests using Brave.
• Proposed incremental global-data indexing is a follow-up, not implemented here: Oro plan, DOMStack #325.

Review notes

All documentation article bodies remain unchanged. Frontmatter now omits 585 redundant titles and records the two existing path aliases through redirectFrom. No collection-specific layout wrappers or per-document TypeScript companions are introduced.

Ingestion remains manual and separate from rendering and CI. No upstream refresh or production deployment was performed. Tests focus on changed behavior and port parity rather than broad coverage expansion.

Suggested review order: shared data and layouts; page outputs and artifact templates; layout registry and vars checks; ingestion/audit ports; clients and workflow/documentation changes.

Validation

The complete npm run check gate passed locally after the final registry integration:

• TypeScript checking, including compile-time layout contract checks, and unit/regression tests.
• Clean build, internal link/anchor audit, all content audits, and static-output tests.
• 18 desktop/mobile browser tests.
• 10 tooling tests, including the standalone full-site watcher.
• Identical repeated build output with no source changes.

Focused verification:

• All 585 raw outputs exactly match their source Markdown bodies using the existing frontmatter-strip semantics.
• Actual watcher body edit changes exactly one raw export and leaves sibling HTML unchanged.
• Frontmatter-only title edit refreshes navigation without changing any raw-export mtimes.
• Raw-path rename and source deletion remove old owned files, including nested .txt output.
• Both docs and spec layout output fixtures preserve whitespace, CRLF, Unicode, and literal braces.
• All six obsolete public navigation JSON files are absent; search and LLM output remains.
• Layout type checks cover missing required vars, invalid names/overrides, malformed chapters, parent chains, and subscription/data separation.
• All 585 article bodies and raw exports were verified byte-identical to the preceding committed version after title deduplication; whitespace checks pass.

Earlier validation also covered Python/TypeScript ingestion parity and computed styles on nine routes at three viewport widths.

Browser caveat: local browser validation uses installed Brave via ORO_BROWSER_EXECUTABLE; CI remains configured to install Playwright Chromium. Local validation passing does not imply the newly triggered CI run has finished.

Deployment

Review and merge through the normal Pages workflow. No upstream ingestion is required to build this branch.

[bcomnes]

Latest implementation and proposed next step: incremental global-data preparation

Pushed 3fcbc14 with all previously uncommitted changes:

• Destination-owned redirectFrom metadata and generated redirects using the full root/redirect layout chain, retaining query/fragment handling and no-JavaScript fallback.
• H1-derived docs titles, shared plain-text conversion, and removal of 585 redundant frontmatter titles. Explicit overrides remain supported and ingestion no longer regenerates redundant titles.
• Focused redirect/title/watch/import regression coverage and documentation.

The complete local npm run check passed before this commit (browser tests used Brave). Final verification confirmed all 585 article bodies and raw exports are byte-identical to the preceding committed version. Frontmatter intentionally differs now; the older statement that all Markdown files match master exactly is superseded.

Remaining performance problem

Page-owned Markdown outputs have removed the collection-wide raw-write amplification. However, src/globals/global.data.ts still reads and renders all 585 documents on each page-build pass to prepare search text, then rebuilds collection projections. Downstream fingerprinting correctly avoids most output rebuilds, but the producer has already done the expensive work.

Proposed division of responsibility

Keep existing fingerprints and dataDeps as the single authority for downstream invalidation.

Existing watch planner / source dependency maps
  → conservative affected-input batch
  → Oro updates its cached document index
  → ordinary global-data result
  → existing top-level fingerprint comparison
  → existing dataDeps subscriber selection

This proposal is not implemented in the current PR.

Oro portion

1. Refactor data preparation into an index and pure projection functions, initially retaining a full-build path.
2. Keep one source-file-keyed entry per document containing validated metadata, URL/edit URL, Markdown, and rendered search text. Keep lightweight source-page records for routes and redirect ownership.
3. Represent collection membership/ordering through source identities referencing that index rather than independently maintained full document copies.
4. On a reset, index all source pages. On a delta, remove deleted entries and replace only affected entries; reconcile both old and new memberships when a page changes collection or identity.
5. Rebuild projections for affected collections from cached entries. Initially, recompute all of an affected collection’s projections rather than creating field-by-field dirty rules.
6. Return the existing public data shapes and subscriptions. Do not expose one giant index key that every layout must subscribe to.

For a paragraph edit, Oro may reconstruct Runtime navigation as well as search/LLM projections. The existing fingerprints will establish that navigation is unchanged and only search/exports invalidate their subscribers. The source event independently selects the edited page and its raw output.

DOMStack portion

Provide backward-compatible, explicit application state and batched input-change context to global data. The context must derive from the existing watch/dependency machinery rather than a second dependency graph. Distinguish reset from conservative page upserts/removals; raw filesystem events can accompany the semantic batch but cannot be the sole correctness contract.

State must survive DOMStack’s fresh per-build workers. A minimal first version can retain structured-cloneable state in the watch session, pass it into the worker, and commit a candidate only after the relevant build succeeds. Producer-code changes, incompatible state, or uncertain dependencies reset it. Existing return values remain public global data; private producer state is separate.

Explicit non-goals

• No application-supplied hashes, changed-key declarations, subscriber graph, or manual selection of output pages.
• No reference-equality shortcut across workers.
• No persistent worker/module-cache rewrite in v1.
• No separate per-field invalidation language.

Fingerprint all published top-level values as today. If fingerprinting becomes a measured bottleneck, optimize the existing tracker rather than adding a competing application system. A fingerprint establishes equality but cannot replace cached rendered search text.

Validation and limits

Instrument source initialization, Markdown reads/search renders, projection work, state transfer, fingerprinting, and output work separately. Prove one ordinary body edit performs one index/search render, then compare incremental data/output with clean builds after H1 changes, overrides, adds/deletes/moves, membership/alias changes, shared vars/settings/helper edits, batched events, and failed-build recovery.

This does not initially eliminate all-page source initialization, state cloning, serialization, or fingerprint traversal. The first target is repeated full-corpus Markdown rendering. Ingestion stays manual and separate from builds.

Framework work is tracked in bcomnes/domstack#325, which links back to this application plan. Neither the incremental state API nor the Oro index optimization is implemented yet.

[bcomnes]

Superseded by the requested two-PR stack:

1. Replace Python ingestion and audit tooling with TypeScript #2 — Replace Python ingestion and audit tooling with TypeScript (base: master). No src changes; existing DOMStack version and title behavior retained.
2. Improve DOMStack outputs, authoring, and incremental builds #3 — Improve DOMStack outputs, authoring, and incremental builds (base: refactor/typescript-tooling). Includes all remaining website work and the beta.7 data index.

Merge #2 first, then retarget/rebase #3 onto master as needed. Both replacement PRs are drafts and passed their full local validation suites independently against their parents (Brave used for local browser tests).

The original branch is preserved and pushed at 515761e, including the formerly uncommitted index work. The final stack tree and that checkpoint have the same Git tree ID: 3f6e1ea523fca7c4a917a5dd16b4d2d112cc2076. No combined work was lost.

Closing this combined PR only to direct review to the two slices; retaining this branch and discussion as history. The incremental-data plan above is now implemented in #3 using DOMStack beta.7.