docs: update READMEs and docs for multi-granularity converter

Author: chris-c-thomas

.changeset/docs-multi-granularity.md

@@ -0,0 +1,10 @@
+---
+"@lexbuild/ecfr": patch
+"@lexbuild/core": patch
+"@lexbuild/usc": patch
+"@lexbuild/fr": patch
+"@lexbuild/cli": patch
+"@lexbuild/mcp": patch
+---
+
+Refresh documentation for the single-pass multi-granularity converter. README.md files across the monorepo (root, `@lexbuild/cli`, `@lexbuild/usc`, `@lexbuild/ecfr`, `@lexbuild/core`, `apps/astro`) now document the `--granularities <list>` and `--output-<granularity>` flags on `convert-usc`/`convert-ecfr`, the new `granularities` option on `convertTitle`/`convertEcfrTitle`, the builder's `ReadonlySet<LevelType>` emit mode, and the updated `update-usc.sh`/`update-ecfr.sh` single-invocation pattern. Public docs on the Astro site and internal docs under `.claude/internal/docs/` were updated in parallel. No code changes in this release — the bump exists to ship the refreshed package README copy to npm.

CLAUDE.md

@@ -303,6 +303,7 @@ Note: identifiers use `/us/cfr/` (content type) not `/us/ecfr/` (data source). B
 - **VPS PM2 logs live at `/home/ubuntu/pm2/logs/lexbuild/`**, not `~/.pm2/logs/`. The latter is legacy — only `pm2-logrotate-out.log` still writes there. Check the new path when debugging PM2-managed services.
 - **VPS has 6 GiB swap** at `/swapfile` (persisted in `/etc/fstab`). Added as defense against Meilisearch OOM during bulk upserts on a 7.6 GiB RAM Lightsail box. Don't remove.
 - **Stuck Meilisearch tasks crash-loop across restarts**: document-addition tasks that OOM Meilisearch are persisted in LMDB and re-attempted after every PM2 restart (observed ~60s crash cycle, 160+ restarts in 2.5 hours). Cancel via `curl -XPOST -H "Authorization: Bearer $MEILI_MASTER_KEY" "http://127.0.0.1:7700/tasks/cancel?uids=<list>"` — the cancellation typically executes during a healthy window even if the stuck task itself can't complete.
+- **`_meta.json` / `README.md` carry wall-clock timestamps**: Converter outputs include a `generated_at` field. Byte-parity tests comparing outputs across runs must skip these files (assert existence, not content).
 
 ## When Adding New Source Types
 

README.md

@@ -143,6 +143,8 @@ Update scripts handle change detection, download, convert, and deploy in one com
 ./scripts/update-usc.sh --skip-deploy
 ```
 
+`update-usc.sh` and `update-ecfr.sh` convert every granularity in one parse using the `--granularities` flag (see below), so the convert step no longer scales with the number of output granularities.
+
 ---
 
 ## Commands
@@ -173,6 +175,13 @@ lexbuild convert-usc --all                                   # All downloaded ti
 lexbuild convert-usc --titles 1 -g chapter                   # Chapter-level output
 lexbuild convert-usc --titles 26 --dry-run                   # Preview without writing
 lexbuild convert-usc ./downloads/usc/xml/usc01.xml           # Direct file path
+
+# Emit section + chapter + title in a single parse
+lexbuild convert-usc --all \
+  --granularities section,title,chapter \
+  --output ./output \
+  --output-title ./output-title \
+  --output-chapter ./output-chapter
 ```
 
 | Option | Default | Description |
@@ -182,6 +191,10 @@ lexbuild convert-usc ./downloads/usc/xml/usc01.xml           # Direct file path
 | `-i, --input-dir <dir>` | `./downloads/usc/xml` | Input XML directory |
 | `-o, --output <dir>` | `./output` | Output directory |
 | `-g, --granularity` | `section` | `section`, `chapter`, or `title` |
+| `--granularities <list>` | — | Comma-separated granularities. Mutually exclusive with `-g`. |
+| `--output-section <dir>` | — | Output dir for section when using `--granularities` (defaults to `--output`) |
+| `--output-chapter <dir>` | — | Output dir for chapter when using `--granularities` |
+| `--output-title <dir>` | — | Output dir for title when using `--granularities` |
 | `--link-style` | `plaintext` | `plaintext`, `canonical`, or `relative` |
 | `--no-include-source-credits` | — | Exclude source credits |
 | `--no-include-notes` | — | Exclude all notes |
@@ -235,6 +248,14 @@ lexbuild convert-ecfr --all                                  # All downloaded ti
 lexbuild convert-ecfr --titles 17 -g part                    # Part-level output
 lexbuild convert-ecfr --all --dry-run                        # Preview without writing
 lexbuild convert-ecfr ./downloads/ecfr/xml/ECFR-title17.xml  # Direct file path
+
+# Emit all four granularities in a single parse
+lexbuild convert-ecfr --all \
+  --granularities section,title,chapter,part \
+  --output ./output \
+  --output-title ./output-title \
+  --output-chapter ./output-chapter \
+  --output-part ./output-part
 ```
 
 | Option | Default | Description |
@@ -244,6 +265,11 @@ lexbuild convert-ecfr ./downloads/ecfr/xml/ECFR-title17.xml  # Direct file path
 | `-i, --input-dir <dir>` | `./downloads/ecfr/xml` | Input XML directory |
 | `-o, --output <dir>` | `./output` | Output directory |
 | `-g, --granularity` | `section` | `section`, `part`, `chapter`, or `title` |
+| `--granularities <list>` | — | Comma-separated granularities. Mutually exclusive with `-g`. |
+| `--output-section <dir>` | — | Output dir for section when using `--granularities` (defaults to `--output`) |
+| `--output-part <dir>` | — | Output dir for part when using `--granularities` |
+| `--output-chapter <dir>` | — | Output dir for chapter when using `--granularities` |
+| `--output-title <dir>` | — | Output dir for title when using `--granularities` |
 | `--link-style` | `plaintext` | `plaintext`, `canonical`, or `relative` |
 | `--no-include-source-credits` | — | Exclude source credits |
 | `--no-include-notes` | — | Exclude all notes |

apps/astro/README.md

@@ -22,16 +22,36 @@ The web application for [LexBuild](https://github.com/chris-c-thomas/LexBuild) 
 
 - **Node.js** >= 22
 - **pnpm** >= 10
-- **Converted content** — run the CLI to generate Markdown files before starting the app:
+- **Converted content** — run the CLI to generate Markdown files before starting the app. The app browses every granularity (section, chapter, title, and part for eCFR), so emit them all in a single parse using `--granularities`:
 
 ```bash
 # From the monorepo root
 pnpm turbo build
-node packages/cli/dist/index.js download-usc --all && node packages/cli/dist/index.js convert-usc --all
-node packages/cli/dist/index.js download-ecfr --all && node packages/cli/dist/index.js convert-ecfr --all
-node packages/cli/dist/index.js download-fr --recent 30 && node packages/cli/dist/index.js convert-fr --all
+
+# USC: download once, then emit section + chapter + title from one parse
+node packages/cli/dist/index.js download-usc --all
+node packages/cli/dist/index.js convert-usc --all \
+  --granularities section,title,chapter \
+  --output ./output \
+  --output-title ./output-title \
+  --output-chapter ./output-chapter
+
+# eCFR: download once, then emit all four granularities from one parse
+node packages/cli/dist/index.js download-ecfr --all
+node packages/cli/dist/index.js convert-ecfr --all \
+  --granularities section,title,chapter,part \
+  --output ./output \
+  --output-title ./output-title \
+  --output-chapter ./output-chapter \
+  --output-part ./output-part
+
+# Federal Register (no granularities — documents are atomic)
+node packages/cli/dist/index.js download-fr --recent 30
+node packages/cli/dist/index.js convert-fr --all
 ```
 
+For routine updates, use `./scripts/update.sh` — the wrapper scripts already use `--granularities` internally.
+
 ## Local Development
 
 ### 1. Link Content

apps/astro/src/content/docs/architecture/conversion-pipeline.md

@@ -90,6 +90,24 @@ const builder = new ASTBuilder({
 
 Levels above the emit level (for example, `title` and `chapter` when emitting at `section`) are tracked as lightweight `AncestorInfo` objects containing just the level type, identifier, number, and heading. Their child subtrees are never accumulated in memory.
 
+### Multi-Level Emit
+
+`emitAt` also accepts a `ReadonlySet<LevelType>`. Deeper levels fire first (sections before their containing title), and emitted nodes remain attached to their parents so a higher-level emission sees the full subtree. Attach-to-parent is gated by "any enclosing stack frame is itself an emit target" — this live stack check is what keeps the logic correct for USLM's permissive level nesting (for example, an appendix inside a part), where hierarchy index ordering would be misleading.
+
+```typescript
+const byLevel = new Map<LevelType, CollectedSection[]>();
+
+const builder = new ASTBuilder({
+  emitAt: new Set(["section", "chapter", "title"]),
+  onEmit: (node, context) => {
+    const bucket = byLevel.get(node.levelType);
+    if (bucket) bucket.push({ node, context });
+  },
+});
+```
+
+The converter uses this to collect per-level buckets in a single pass and write every requested granularity from the matching bucket. This is how `--granularities` on the CLI produces multiple output trees without re-parsing the XML.
+
 ## The Collect-Then-Write Pattern
 
 Both USC and eCFR converters collect all emitted nodes synchronously during parsing, then write files after parsing completes. The collect phase pushes `{ node, context }` pairs into an array; the write phase iterates this array to render and write each file.

apps/astro/src/content/docs/cli/commands.md

@@ -52,6 +52,10 @@ These flags are available on all convert commands:
 |---|---|---|
 | `-o, --output <dir>` | `./output` | Output directory for converted Markdown |
 | `-g, --granularity <level>` | `section` | Output granularity (varies by source) |
+| `--granularities <list>` | -- | Comma-separated granularities for multi-pass output. Mutually exclusive with `-g`. |
+| `--output-chapter <dir>` | -- | Output directory for chapter granularity (when using `--granularities`) |
+| `--output-title <dir>` | -- | Output directory for title granularity (when using `--granularities`) |
+| `--output-part <dir>` | -- | Output directory for part granularity, eCFR only (when using `--granularities`) |
 | `--link-style <style>` | `plaintext` | Cross-reference link style: `plaintext`, `relative`, or `canonical` |
 | `--dry-run` | off | Parse and report statistics without writing files |
 | `-v, --verbose` | off | Print detailed output including file paths |
@@ -63,6 +67,29 @@ These flags are available on all convert commands:
 > [!NOTE]
 > Setting any selective note flag (`--include-editorial-notes`, `--include-statutory-notes`, `--include-amendments`) automatically disables the broad `--include-notes` flag.
 
+### Multi-Granularity Single-Pass Mode
+
+Use `--granularities` to emit several granularity levels from a single parse of the source XML. Each listed granularity needs a matching output directory — section uses `--output` (or `--output-section`); chapter, title, and part (eCFR only) each take their own `--output-<granularity>` flag.
+
+```bash
+# USC: three granularities in one parse
+lexbuild convert-usc --all \
+  --granularities section,title,chapter \
+  --output ./output \
+  --output-title ./output-title \
+  --output-chapter ./output-chapter
+
+# eCFR: four granularities in one parse
+lexbuild convert-ecfr --all \
+  --granularities section,title,chapter,part \
+  --output ./output \
+  --output-title ./output-title \
+  --output-chapter ./output-chapter \
+  --output-part ./output-part
+```
+
+`--granularities` is mutually exclusive with `-g/--granularity`. The builder parses the XML once and emits at every requested level, so multi-granularity runs are roughly ~40–50% faster than N separate single-granularity invocations.
+
 ## Full Pipeline Examples
 
 ### U.S. Code
@@ -110,7 +137,7 @@ For routine updates, wrapper scripts handle the full pipeline (detect changes, d
 ./scripts/update-usc.sh                  # USC, checks release point
 ```
 
-Each script auto-detects what changed and only processes updates. See [Incremental Updates](/docs/guides/bulk-download#incremental-updates) for details.
+Each script auto-detects what changed and only processes updates. `update-usc.sh` and `update-ecfr.sh` convert all granularities in one call using `--granularities` (see above), so the convert step parses the XML once per title rather than once per granularity. See [Incremental Updates](/docs/guides/bulk-download#incremental-updates) for details.
 
 ## Getting Help
 

apps/astro/src/content/docs/cli/sources/ecfr.md

@@ -99,6 +99,21 @@ lexbuild convert-ecfr --all -g part
 lexbuild convert-ecfr --titles 17 -g title
 ```
 
+### Multi-Granularity (Single Pass)
+
+Emit all four granularities from one parse:
+
+```bash
+lexbuild convert-ecfr --all \
+  --granularities section,title,chapter,part \
+  --output ./output \
+  --output-title ./output-title \
+  --output-chapter ./output-chapter \
+  --output-part ./output-part
+```
+
+`--granularities` is mutually exclusive with `-g`. Because the XML is parsed once and fanned out to each requested directory, this is roughly ~40–50% faster than running `convert-ecfr` four times with different `-g` values.
+
 ### Notes Filtering
 
 Notes filtering works the same as for the U.S. Code:

apps/astro/src/content/docs/cli/sources/us-code.md

@@ -87,6 +87,20 @@ lexbuild convert-usc --all -g chapter
 lexbuild convert-usc --titles 26 -g title
 ```
 
+### Multi-Granularity (Single Pass)
+
+Emit section, chapter, and title outputs from one parse:
+
+```bash
+lexbuild convert-usc --all \
+  --granularities section,title,chapter \
+  --output ./output \
+  --output-title ./output-title \
+  --output-chapter ./output-chapter
+```
+
+`--granularities` is mutually exclusive with `-g`. Because the XML is parsed once and fanned out to each requested directory, this is roughly ~40–50% faster than running `convert-usc` three times with different `-g` values.
+
 ### Notes Filtering
 
 All notes (editorial, statutory, and amendment history) are included by default. Disable them entirely or filter selectively:

apps/astro/src/content/docs/guides/bulk-download.md

@@ -197,6 +197,29 @@ lexbuild convert-usc --all -g title -o ./output/title
 
 Produces 54 files for USC, 50 for eCFR. Files can be large (1-100 MB). Title-level files include extra frontmatter fields: `chapter_count`, `section_count`, and `total_token_estimate`.
 
+### All Granularities in One Pass
+
+If you need more than one granularity, `--granularities` emits them from a single parse of the source XML (~40–50% faster than running `convert-*` N times):
+
+```bash
+# USC: section + chapter + title from one parse
+lexbuild convert-usc --all \
+  --granularities section,title,chapter \
+  --output ./output \
+  --output-title ./output-title \
+  --output-chapter ./output-chapter
+
+# eCFR: all four granularities from one parse
+lexbuild convert-ecfr --all \
+  --granularities section,title,chapter,part \
+  --output ./output \
+  --output-title ./output-title \
+  --output-chapter ./output-chapter \
+  --output-part ./output-part
+```
+
+`--granularities` is mutually exclusive with `-g/--granularity`.
+
 > [!NOTE]
 > The `-o` flag appends source subdirectories automatically. `convert-usc -o /some/path` writes to `/some/path/usc/`, not `/some/path/` directly.
 

apps/astro/src/content/docs/guides/rag-pipeline.md

@@ -41,14 +41,18 @@ You can convert the same source at different granularity levels to support diffe
 | `chapter` / `part` | 10-100k tokens | Topic-level context, summarization with large context models |
 | `title` | 100k-2M tokens | Whole-title analysis, full corpus summarization |
 
-Generate multiple granularities to different output directories:
+Generate multiple granularities to different output directories in a single parse:
 
 ```bash
-lexbuild convert-usc --all -g section -o ./output/section
-lexbuild convert-usc --all -g chapter -o ./output/chapter
-lexbuild convert-usc --all -g title -o ./output/title
+lexbuild convert-usc --all \
+  --granularities section,title,chapter \
+  --output ./output/section \
+  --output-title ./output/title \
+  --output-chapter ./output/chapter
 ```
 
+`--granularities` emits every requested level from one read of the XML. For a single granularity, use the older `-g <level> -o <dir>` form.
+
 ## Parsing Frontmatter
 
 Use [gray-matter](https://github.com/jonschlinkert/gray-matter) (JavaScript) or any YAML frontmatter parser to separate metadata from body text.