feat: front-matter doc index generator

- Introduce `tools/gen_doc_index.py` to derive class document tables and topic listings from document front matter, with strict validation for metadata, stub status, topic vocabulary, and optional nav/tree consistency checks for CI (`--check`, `--nav`).
- Update documentation standards and tooling docs to require front matter-driven indexing and two-level Diataxis indexes.
- Add pytest coverage (plus test import setup) for generation, validation failures, and navigation drift cases.

Author: tazlin

docs/meta/documentation.md

@@ -5,7 +5,8 @@ documents written for contributors and operators, not to docstrings, which are c
 
 In brief:
 
-- Organize `docs/` by [Diátaxis](https://diataxis.fr/) class, with `docs/README.md` as the index.
+- Organize `docs/` by [Diátaxis](https://diataxis.fr/) class, each class carrying its own index.
+- Declare a document's title, summary, status, and topics in front matter, and generate the indexes from it.
 - Open every document with a summary that survives truncation.
 - Order headings from overview to detail so a reader can stop at any depth.
 - Give reference documents a code map from concept to file and symbol.
@@ -21,11 +22,43 @@ set. Change proposals are welcome where a rule is overly restrictive, missing, o
 ## Organization
 
 Documents are filed by Diátaxis class: `explanation/`, `reference/`, `how-to/`, and `tutorial/`, using only the classes
-a repository actually needs. `docs/README.md` indexes the tree and says what each document holds.
+a repository actually needs. A class directory is created when its first document is written, never in advance.
+
+Indexing is two-level. Each class carries a `README.md` that says what the class is for, lists its documents, and holds
+any conventions specific to that class. `docs/README.md` routes between the classes and does not list individual
+documents, so the landing page stays a constant size as the tree grows.
 
 Diátaxis settles where a document is filed and what mode it is written in. The rules below govern quality, and they
 apply within every class.
 
+## Front matter and generated indexes
+
+Every curated document declares its own metadata, and every index that lists documents is generated from those
+declarations. An index maintained by hand disagrees with the tree it indexes, and the disagreement is invisible until a
+reader is already lost.
+
+```yaml
+---
+title: Job matching and pop # equals the document's level-one heading
+summary: One line, at most 200 characters, shown wherever this document is listed.
+status: authored # authored | stub
+topics: [requests, workers] # each defined in the topics page
+order: 30 # optional; sequences the document within its class
+---
+```
+
+- `summary` is what an index shows. It states what the document holds, not that the document exists.
+- `status` is `authored` or `stub`. A stub is published rather than withheld: it scopes what the document will cover,
+  names the code that holds the behavior until then, and is marked as planned in its class index. A stub says what is
+  planned; it never says when.
+- `topics` are subject tags shared across classes. A topics page defines each tag and collects the documents carrying
+  it, which is how a reader holding a subject reaches the explanation, the reference, and the procedure together. A tag
+  must be defined before it is used, and a defined tag no document uses is removed. Without that rule the tag set
+  splinters into near-duplicates and stops being a navigation surface.
+
+Where a repository publishes a site, navigation order and display titles stay curated; navigation membership does not.
+Membership is checked against the tree so a document cannot be added without appearing, or removed while still listed.
+
 ## Opening summary
 
 Every document opens with a summary a reader can stop after and still know what the document covers, who it is for, and
@@ -93,7 +126,9 @@ A documentation change lands in the same pull request as the behavior change it
 the code it documents is treated as a defect in that pull request.
 
 A repository publishing its docs runs the gates in CI: the site builds in strict mode, every generated index verifies
-clean against its records (`--check`), license compliance passes, and internal links resolve. A generated file that can
+clean against its inputs (`--check`), front matter is valid and agrees with the document body, site navigation matches
+the tree, cited code paths exist, prose style rules that can be checked mechanically are checked, license compliance
+passes, and internal links resolve. Tooling that gates the documentation is itself tested. A generated file that can
 drift from its inputs without failing CI will.
 
 ## Architecture Decision Records
@@ -124,8 +159,9 @@ and links its replacement. A decision changed in part stays `accepted` and links
 two ways as well.
 
 The directory stays flat and numbers are global: never reused, never renumbered, never sharded into subdirectories.
-Topical organization belongs to the index. While a repository holds few records, `docs/README.md` lists them; past
-that, `docs/decisions/README.md` indexes them, grouped by system and showing status.
+Topical organization belongs to the index. The records are indexed by a generated `docs/decisions/index.md` carrying
+each record's status and date, and that page is how the corpus enters navigation: records are never listed
+individually, so navigation does not grow with the record count.
 
 An accepted record is immutable in substance. Permissible edits are typo fixes and repairing file or symbol references
 that the codebase has moved out from under the record. Context, Decision Outcome, and Consequences are never rewritten

tools/README.md

@@ -20,6 +20,49 @@ python docs/haidra-assets/tools/gen_adr_index.py docs/decisions --check
 The index renders on GitHub (which draws `mermaid` fences natively) and on any static site generator whose
 Markdown pipeline has a Mermaid fence configured, such as Zensical via `pymdownx.superfences`.
 
+## gen_doc_index.py
+
+Builds the parts of a documentation tree that can be derived from the documents themselves, so an index cannot
+disagree with what it indexes. Each curated document carries YAML front matter:
+
+```yaml
+---
+title: Job matching and pop            # must equal the document's level-one heading
+summary: One line, at most 200 characters, shown in every index that lists this document.
+status: authored                       # authored | stub
+topics: [requests, workers]            # each topic must be defined in topics.md
+order: 30                              # optional, orders the document within its class (default 100)
+---
+```
+
+From that, the tool rewrites two kinds of generated block, leaving the hand-written prose around them untouched:
+
+- `documents` in each class index (`docs/<class>/README.md`): a table of authored documents, then a table of
+  planned stubs.
+- `topics` in `docs/topics.md`: every topic with the documents that declare it, across classes.
+
+A block is delimited by `<!-- BEGIN GENERATED: name (gen_doc_index.py) -->` and `<!-- END GENERATED: name -->`.
+A documentation class is any directory under the docs root holding a `README.md`, minus `--exclude` names
+(`haidra-assets` by default). Decision records are excluded by construction: they carry an `index.md`, not a
+`README.md`, and are indexed by `gen_adr_index.py`.
+
+```sh
+# Rewrite the generated blocks
+python docs/haidra-assets/tools/gen_doc_index.py docs
+
+# CI: fail on a stale block, invalid front matter, or navigation drift
+python docs/haidra-assets/tools/gen_doc_index.py docs --check --nav zensical.toml
+```
+
+The tool fails rather than generating a misleading index when a document's front matter is missing or invalid,
+its title disagrees with its heading, its `status` disagrees with the presence of a stub banner, or it uses a
+topic that `topics.md` does not define (and, in reverse, when `topics.md` defines a topic no document uses).
+
+`--nav` cross-checks a Zensical configuration against the tree. Navigation order and display titles stay
+curated; membership does not. It fails when a curated document is missing from `nav`, listed twice, or absent
+from disk, and when a generated record such as `decisions/0001-*.md` is listed individually instead of being
+reached through its index.
+
 ## check_code_refs.py
 
 Fails when documentation cites a repository path that does not exist. It extracts backticked tokens that start