feat(crf): plan and build a CRF corpus from a manifest

CHANGELOG.md

@@ -7,6 +7,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **`leech.crf.corpus`: cut a CRF training corpus from a manifest.**
+  `plan_corpus` decides which reads and in which split, touching no POD5;
+  `build_corpus` extracts their signal, streaming it to a memory-mappable
+  `<out>_X.npy` beside a `<out>_meta.npz`. The signal is never held in RAM as a
+  whole — an 80-plex corpus is tens of gigabytes, and the memmap is what makes
+  the size a disk question instead of an allocation that fails.
+
+  The two stages are separate because everything subtle is in the plan, and a
+  corpus planned wrongly still trains and still reports a number. Four rules,
+  each pinned by a test: a cap only caps if every class can reach it
+  (`per_group="auto"` is the rarest class's *trainable* depth, with the test
+  fraction reserved first); the split is carved before capping and ranked per
+  class globally across batches, since per-`(batch, class)` ranking multiplies
+  the cap by the batch count whenever classes are crossed with batch; batches
+  are interleaved rather than concatenated, or the whole held-out set comes from
+  whichever batch sorts first and the headline number measures batch; and
+  sharding happens after planning, so every shard keeps its share of one global
+  split. Extracting nothing, or less than half the plan, is a hard error with a
+  different message for each — the causes differ, and a 0-row corpus otherwise
+  exits cleanly and reaches a GPU job.
+
+  Validated against the production ldx manifest: 1,139,602 rows plan to 391,174
+  reads at `chunk=3000`, the same count escapepod-models' extractor reports for
+  that input, with the training pool balanced exactly across all 16 groups and
+  held-out reads drawn from both flowcells.
+
+  `load_corpus` / `load_corpus_meta` read both this layout and the legacy
+  single-`.npz` one, so corpora written before the split layout keep loading.
+
 ## [0.8.0] - 2026-08-25
 
 Minor rather than patch: one new capability, no change to anything that

CLAUDE.md

@@ -472,6 +472,20 @@ Two rules the manifest exists to enforce, both silent when broken:
   trains, converges, and quietly discriminates on fewer bases than designed.
   Pass a *measured* `samples_per_base` — leech has dwell times — not a constant.
 
+**Corpus planning is separate from extraction, and that is where the rules
+are.** `crf.plan_corpus` decides which reads and which split without touching a
+POD5; `crf.build_corpus` streams their signal to a memmap. Four rules, all
+silent when broken: a cap only caps if every class can reach it (`per_group="auto"`
+= the rarest class's *trainable* depth, test fraction reserved first); the split
+is carved before capping and ranked per class **globally across batches**, since
+per-`(batch, class)` ranking multiplies the cap by the batch count whenever
+classes are crossed with batch; batches are **interleaved, not concatenated**,
+or the whole test set comes from whichever batch sorts first and the headline
+number measures batch; and sharding happens **after** planning, so each shard
+keeps its share of one global split. Validated against the production ldx
+manifest: 1.14M rows plan to 391,174 reads, the same count that repo's extractor
+reports, with train balanced exactly across all 16 groups.
+
 The analytic forward-backward in `_analytic.py` is the loss path; the plain
 scans in `loss.py` are the readable reference the tests check it against, and
 the Triton kernels check against those. Keep all three — the fallback chain is

docs/api/crf.md

@@ -92,6 +92,38 @@ and belongs to whatever project defines those. Everything below is signal ML.
 There is deliberately **no `keep` boolean**: quality travels as numbers so the
 gate stays sweepable without re-cutting the corpus.
 
+## Building a corpus
+
+`plan_corpus` decides *which reads and in which split*, touching no POD5;
+`build_corpus` then extracts their signal, streaming it to a memory-mappable
+`<out>_X.npy` beside a `<out>_meta.npz`. The split is deliberately separate: it
+is where every subtle rule lives, and it is testable without a gigabyte of
+fixture.
+
+```python
+from leech.crf import load_manifest, plan_corpus, build_corpus, load_corpus
+
+plan = plan_corpus(load_manifest("manifest.parquet"), chunk=3000, per_group="auto")
+print(len(plan), plan.cap, plan.counts_by_split())
+build_corpus(plan, "corpus")
+signal, targets, groups, read_ids, split = load_corpus("corpus")   # signal is mmap'd
+```
+
+Four rules it enforces, each of which fails silently when broken:
+
+- **A cap only caps if every class can reach it.** `per_group="auto"` uses the
+  rarest class's *trainable* depth (the test fraction is reserved first), which
+  is the only value that actually balances. A larger explicit cap warns and
+  de-balances the corpus it was meant to balance.
+- **The split is carved before capping, ranked per class and globally across
+  batches.** Ranking per `(batch, class)` multiplies the cap by the number of
+  batches whenever classes are crossed with batch.
+- **Batches are interleaved, not concatenated.** Otherwise the whole test set
+  comes from whichever batch sorts first and the headline number measures batch.
+- **Shard after planning.** The plan is deterministic in `(manifest, seed)`, so
+  each shard keeps its share of one global split; filtering first would give
+  each shard its own test set drawn from one batch.
+
 ## Encoder
 
 ::: leech.crf.encoder.CrfEncoder
@@ -151,6 +183,28 @@ gate stays sweepable without re-cutting the corpus.
     options:
       show_root_heading: true
 
+## Corpus
+
+::: leech.crf.corpus.plan_corpus
+    options:
+      show_root_heading: true
+
+::: leech.crf.corpus.CorpusPlan
+    options:
+      show_root_heading: true
+
+::: leech.crf.corpus.build_corpus
+    options:
+      show_root_heading: true
+
+::: leech.crf.corpus.load_corpus
+    options:
+      show_root_heading: true
+
+::: leech.crf.corpus.load_corpus_meta
+    options:
+      show_root_heading: true
+
 ## Configuration
 
 ::: leech.crf.config.load_config

src/leech/crf/__init__.py

@@ -63,6 +63,11 @@
     "encoder_config_from_toml": "leech.crf.encoder",
     "load_crf_state_dict": "leech.crf.encoder",
     "CtcCrfLoss": "leech.crf.loss",
+    "CorpusPlan": "leech.crf.corpus",
+    "build_corpus": "leech.crf.corpus",
+    "load_corpus": "leech.crf.corpus",
+    "load_corpus_meta": "leech.crf.corpus",
+    "plan_corpus": "leech.crf.corpus",
     "CrfManifest": "leech.crf.manifest",
     "OPTIONAL_COLUMNS": "leech.crf.manifest",
     "REQUIRED_COLUMNS": "leech.crf.manifest",
@@ -78,18 +83,23 @@
     "DEFAULT_CONFIG",
     "OPTIONAL_COLUMNS",
     "REQUIRED_COLUMNS",
+    "CorpusPlan",
     "CrfEncoder",
     "CtcCrfLoss",
     "EncoderConfig",
     "CrfManifest",
     "best_path",
+    "build_corpus",
     "check_geometry",
     "decode_batch",
     "emitted_target",
     "encoder_config_from_toml",
     "load_config",
+    "load_corpus",
+    "load_corpus_meta",
     "load_crf_state_dict",
     "load_manifest",
+    "plan_corpus",
 ]