Merge branch 'main' into dependabot/npm_and_yarn/npm_and_yarn-da47bb892b

Author: chris-c-thomas

CLAUDE.md

@@ -130,6 +130,8 @@ pnpm turbo build:api --filter=@lexbuild/api        # Production build
 ./scripts/deploy.sh --search-docker-seed          # Seed Docker volume from VPS (recover after volume loss)
 
 # Incremental content updates (from monorepo root)
+# Search indexing runs locally in Docker, not on the VPS — each update script's
+# final step delegates to `deploy.sh --search-docker --source <name>`.
 ./scripts/update.sh                                # All sources incrementally
 ./scripts/update.sh --source ecfr                  # One source
 ./scripts/update.sh --skip-deploy                  # Local only
@@ -298,6 +300,9 @@ Note: identifiers use `/us/cfr/` (content type) not `/us/ecfr/` (data source). B
 - **Docker search index checkpoints**: The incremental indexing script writes checkpoint files (`.search-indexed-at-{source}`) into the content directory. For Docker runs, these are persisted in `downloads/.search-checkpoints/` and restored into the temp content dir on each run. If this directory is deleted, the next Docker index run will scan all files from scratch.
 - **Docker volume profiles**: `MEILI_PROFILE=dev|full` selects volume (`meili-data-dev` or `meili-data-full`). Dev mode runs without master key (`MEILI_ENV=development`). Full mode requires `MEILI_MASTER_KEY` for VPS-compatible data.
 - **Cloudflare "Managed robots.txt"**: When enabled, Cloudflare overwrites the site's `robots.txt` to block AI crawlers. For LexBuild (public domain legal content), this should be **OFF**. The custom `robots.txt` at `apps/astro/public/robots.txt` blocks AI crawlers from `/_astro/` (hashed static assets), `/nav/` (internal JSON), and `/api/` while allowing legal content.
+- **VPS PM2 logs live at `/home/ubuntu/pm2/logs/lexbuild/`**, not `~/.pm2/logs/`. The latter is legacy — only `pm2-logrotate-out.log` still writes there. Check the new path when debugging PM2-managed services.
+- **VPS has 6 GiB swap** at `/swapfile` (persisted in `/etc/fstab`). Added as defense against Meilisearch OOM during bulk upserts on a 7.6 GiB RAM Lightsail box. Don't remove.
+- **Stuck Meilisearch tasks crash-loop across restarts**: document-addition tasks that OOM Meilisearch are persisted in LMDB and re-attempted after every PM2 restart (observed ~60s crash cycle, 160+ restarts in 2.5 hours). Cancel via `curl -XPOST -H "Authorization: Bearer $MEILI_MASTER_KEY" "http://127.0.0.1:7700/tasks/cancel?uids=<list>"` — the cancellation typically executes during a healthy window even if the stuck task itself can't complete.
 
 ## When Adding New Source Types
 

apps/astro/CLAUDE.md

@@ -158,7 +158,7 @@ npx tsx scripts/index-search-incremental.ts --set-checkpoint  # Set checkpoint w
 
 Script notes:
 - **generate-highlights.ts**: Forks child processes in 2k-file chunks (default, tunable via `--chunk-size N`) to avoid Shiki OOM. Each child is heap-capped at 2GB (`--max-old-space-size`). Uses `matter(raw, { cache: false })` to prevent gray-matter from caching every file in memory. Supports `--limit N` for testing. Changing themes requires updating both this script and `src/lib/shiki.ts`, then deleting existing `.highlighted.html` files.
-- **index-search.ts** and **index-search-incremental.ts**: Must be kept in sync — sources indexed, `SearchDocument` shape, and `configureIndex` settings must match. Both index USC, eCFR, and FR. Full reindex deletes and rebuilds; incremental upserts only changed files (mtime-based per-source checkpoints in `.search-indexed-at-{usc,ecfr,fr}`). Checkpoints are always written after indexing, even with `--source` — each source tracks independently. 500 docs/batch, 300s waitForTask timeout. Document IDs sanitized (dots/colons → underscores).
+- **index-search.ts** and **index-search-incremental.ts**: Must be kept in sync — sources indexed, `SearchDocument` shape, and `configureIndex` settings must match. Both index USC, eCFR, and FR. Full reindex deletes and rebuilds; incremental upserts only changed files (mtime-based per-source checkpoints in `.search-indexed-at-{usc,ecfr,fr}`). Checkpoints are always written after indexing, even with `--source` — each source tracks independently. Default 500 docs/batch (override with `--batch-size N` or `MEILI_BATCH_SIZE` env var), 300s waitForTask timeout. `--verbose-batches` logs first/last doc ID per flush — pair with `--batch-size 1` to bisect poison docs. Document IDs sanitized (dots/colons → underscores).
 - **generate-nav.ts**: Includes reserved title placeholders (USC 53, eCFR 35). Chapter grouping for eCFR derived from filesystem directories, not `_meta.json`.
 - **All pipeline scripts support `--source usc|ecfr|fr`**: `generate-nav.ts`, `generate-sitemap.ts`, `generate-highlights.ts`, `index-search.ts` (full), and `index-search-incremental.ts` all accept `--source` to process a single source. Sitemap `--source` doesn't rewrite the sitemap index (run without `--source` to rebuild the full index). Highlights `--source` filters by content path prefix.
 

apps/astro/scripts/index-search-incremental.ts

@@ -17,15 +17,22 @@
  *   npx tsx scripts/index-search-incremental.ts [content-dir] [--prune] [--source <name>]
  *
  * Options:
- *   content-dir      Path to content directory (default: ./content)
- *   --prune          Remove documents from the index for sections that no longer
- *                    exist on disk (compares Meilisearch IDs against filesystem)
- *   --source <name>  Only index a specific source: usc, ecfr, or fr
- *   --set-checkpoint Write the checkpoint timestamp and exit (no indexing)
+ *   content-dir           Path to content directory (default: ./content)
+ *   --prune               Remove documents from the index for sections that no longer
+ *                         exist on disk (compares Meilisearch IDs against filesystem)
+ *   --source <name>       Only index a specific source: usc, ecfr, or fr
+ *   --set-checkpoint      Write the checkpoint timestamp and exit (no indexing)
+ *   --batch-size <n>      Override docs-per-batch (default 500). Smaller batches use
+ *                         less Meilisearch memory per flush — useful to isolate
+ *                         pathological docs or avoid OOM on memory-tight hosts.
+ *   --verbose-batches     Print the first/last doc ID of each flushed batch. Combined
+ *                         with --batch-size 1 this produces a per-doc log that makes
+ *                         a poison document obvious (last printed ID before a crash).
  *
  * Environment:
  *   MEILI_URL          Meilisearch endpoint (default: http://127.0.0.1:7700)
  *   MEILI_MASTER_KEY   Master key for admin operations (default: none for dev)
+ *   MEILI_BATCH_SIZE   Fallback for --batch-size when the flag is not set.
  */
 
 import { readdir, readFile, writeFile, stat } from "node:fs/promises";
@@ -38,7 +45,7 @@ import matter from "gray-matter";
 const MEILI_URL = process.env.MEILI_URL ?? "http://127.0.0.1:7700";
 const MEILI_MASTER_KEY = process.env.MEILI_MASTER_KEY ?? "";
 const INDEX_NAME = "lexbuild";
-const BATCH_SIZE = 500;
+const DEFAULT_BATCH_SIZE = 500;
 const BODY_TRUNCATE_CHARS = 5000;
 const CHECKPOINT_PREFIX = ".search-indexed-at";
 const SOURCES = ["usc", "ecfr", "fr"] as const;
@@ -184,6 +191,7 @@ class BatchIndexer {
     private readonly client: Meilisearch,
     private readonly indexName: string,
     private readonly batchSize: number,
+    private readonly verbose: boolean = false,
   ) {}
 
   async add(doc: SearchDocument): Promise<void> {
@@ -201,6 +209,14 @@ class BatchIndexer {
     const toSend = this.batch;
     this.batch = [];
 
+    if (this.verbose) {
+      const firstId = toSend[0]?.id ?? "";
+      const lastId = toSend[toSend.length - 1]?.id ?? "";
+      const label = toSend.length === 1 ? firstId : `${firstId} … ${lastId}`;
+      // Force-flush stdout so the last logged batch is durable on crash.
+      process.stdout.write(`  → flushing ${toSend.length} docs: ${label}\n`);
+    }
+
     const index = this.client.index(this.indexName);
     const task = await index.addDocuments(toSend);
     await this.client.tasks.waitForTask(task.taskUid, { timeout: 300_000 });
@@ -533,13 +549,36 @@ async function main(): Promise<void> {
   let prune = false;
   let sourceFilter: "usc" | "ecfr" | "fr" | null = null;
   let setCheckpoint = false;
+  let verboseBatches = false;
+
+  // Resolve batch size from env first; a --batch-size flag overrides.
+  let batchSize = DEFAULT_BATCH_SIZE;
+  const envBatch = process.env.MEILI_BATCH_SIZE;
+  if (envBatch) {
+    const parsed = Number.parseInt(envBatch, 10);
+    if (!Number.isFinite(parsed) || parsed < 1) {
+      console.error(`Invalid MEILI_BATCH_SIZE: ${envBatch}. Must be a positive integer.`);
+      process.exit(1);
+    }
+    batchSize = parsed;
+  }
 
   for (let i = 0; i < args.length; i++) {
     const arg = args[i]!;
     if (arg === "--set-checkpoint") {
       setCheckpoint = true;
     } else if (arg === "--prune") {
       prune = true;
+    } else if (arg === "--verbose-batches") {
+      verboseBatches = true;
+    } else if (arg === "--batch-size" && args[i + 1]) {
+      const parsed = Number.parseInt(args[i + 1]!, 10);
+      if (!Number.isFinite(parsed) || parsed < 1) {
+        console.error(`Invalid --batch-size: ${args[i + 1]}. Must be a positive integer.`);
+        process.exit(1);
+      }
+      batchSize = parsed;
+      i++;
     } else if (arg === "--source" && args[i + 1]) {
       const val = args[i + 1]!;
       if (val !== "usc" && val !== "ecfr" && val !== "fr") {
@@ -570,6 +609,7 @@ async function main(): Promise<void> {
   console.log(`Meilisearch URL: ${MEILI_URL}`);
   console.log(`Index name: ${INDEX_NAME}`);
   console.log(`Mode: incremental upsert${prune ? " + prune" : ""}${sourceFilter ? ` (${sourceFilter} only)` : ""}`);
+  console.log(`Batch size: ${batchSize}${verboseBatches ? " (verbose)" : ""}`);
   console.log(`  Preserves the existing index. Adds new documents and updates existing ones.`);
 
   const client = new Meilisearch({
@@ -617,7 +657,7 @@ async function main(): Promise<void> {
     }
   }
 
-  const indexer = new BatchIndexer(client, INDEX_NAME, BATCH_SIZE);
+  const indexer = new BatchIndexer(client, INDEX_NAME, batchSize, verboseBatches);
 
   // Track all expected IDs for pruning
   const expectedIds = new Set<string>();

fixtures/expected/duplicate-first.md

@@ -13,7 +13,7 @@ positive_law: true
 currency: "unknown"
 last_updated: "2025-12-03"
 format_version: "1.1.0"
-generator: "lexbuild@1.24.0"
+generator: "lexbuild@1.24.1"
 source_credit: "(Pub. L. 108–458, Nov. 17, 2006.)"
 ---
 

fixtures/expected/duplicate-other.md

@@ -13,7 +13,7 @@ positive_law: true
 currency: "unknown"
 last_updated: "2025-12-03"
 format_version: "1.1.0"
-generator: "lexbuild@1.24.0"
+generator: "lexbuild@1.24.1"
 source_credit: "(Added Pub. L. 107–296.)"
 ---
 

fixtures/expected/duplicate-second.md

@@ -13,7 +13,7 @@ positive_law: true
 currency: "unknown"
 last_updated: "2025-12-03"
 format_version: "1.1.0"
-generator: "lexbuild@1.24.0"
+generator: "lexbuild@1.24.1"
 source_credit: "(Pub. L. 110–181, Jan. 28, 2008.)"
 ---
 

fixtures/expected/layout.md

@@ -13,7 +13,7 @@ positive_law: true
 currency: "unknown"
 last_updated: "2025-12-03"
 format_version: "1.1.0"
-generator: "lexbuild@1.24.0"
+generator: "lexbuild@1.24.1"
 source_credit: "(Test source.)"
 ---
 

fixtures/expected/notes-all.md

@@ -13,7 +13,7 @@ positive_law: true
 currency: "unknown"
 last_updated: "2025-12-03"
 format_version: "1.1.0"
-generator: "lexbuild@1.24.0"
+generator: "lexbuild@1.24.1"
 source_credit: "(Added Pub. L. 104–199, § 3(a), Sept. 21, 1996.)"
 ---
 

fixtures/expected/notes-amendments-only.md

@@ -13,7 +13,7 @@ positive_law: true
 currency: "unknown"
 last_updated: "2025-12-03"
 format_version: "1.1.0"
-generator: "lexbuild@1.24.0"
+generator: "lexbuild@1.24.1"
 source_credit: "(Added Pub. L. 104–199, § 3(a), Sept. 21, 1996.)"
 ---
 

fixtures/expected/notes-none.md

@@ -13,7 +13,7 @@ positive_law: true
 currency: "unknown"
 last_updated: "2025-12-03"
 format_version: "1.1.0"
-generator: "lexbuild@1.24.0"
+generator: "lexbuild@1.24.1"
 source_credit: "(Added Pub. L. 104–199, § 3(a), Sept. 21, 1996.)"
 ---