feat(crf): plan and build a CRF corpus from a manifest

[jayhesselberth]

plan_corpus decides which reads and in which split, touching no POD5;
build_corpus extracts their signal into a memory-mappable <out>_X.npy beside
a <out>_meta.npz. This is the piece that lets escapepod-models' extractor
become assay-free code that lives here — with the target resolved into the
manifest (#216), nothing in the path needs a panel.

Why the two stages are separate

Everything subtle is in the plan, so it is testable without a gigabyte of
fixture. And it needs testing: a corpus planned wrongly does not fail, it trains
and reports a number.

rule	what breaks silently without it
per_group="auto" = rarest class's trainable depth (test fraction reserved first)	a larger cap de-balances the corpus it was meant to balance — one class gives everything it has while others are held back
split carved before capping, ranked per class globally across batches	per-(batch, class) ranking multiplies the cap by the batch count whenever classes are crossed with batch
batches interleaved, not concatenated	the whole held-out set comes from whichever batch sorts first, and the headline number measures batch rather than signal
shard after planning	each shard otherwise gets its own test set drawn from one batch

Reads are sorted by read_id before shuffling, so the draw does not depend on
manifest row order.

Validated on production data, not just fixtures

The real ldx manifest — 1,139,602 rows — plans in 0.3 s to:

planned 391,174 reads  (cap=17,706, dropped for short anchor: 60,749)
splits: {'train': 283296, 'test': 107878}
train per group: min 17,706 max 17,706 -> balanced: True
test reads per batch: {'LDX1-8': 48981, 'LDX9-16': 58897}

391,174 is the same count escapepod-models' extractor reports for that
manifest at chunk=3000 (it appears in that script's own comments). That is
the parity evidence; the unit tests pin the individual rules.

Failure modes are hard errors

Extracting nothing and extracting less than half the plan get different
messages, because the causes differ — zero means the POD5 path is wrong, a
shortfall means the manifest and the POD5s disagree about which reads exist.
Neither is a small corpus. Without the check a 0-row array is written, exits
cleanly, and reaches a GPU job, which is the expensive way to find out.

_trim clamps its block bound on both sides: the source keeps the planned
row count, so an unclamped final block refuses to broadcast and loses an
already-extracted corpus at the very last step. It only fires when reads were
dropped and the remainder straddles a block boundary — pinned by a test.

Import weight

polars and leech.io.pod5_reader (which pulls pysam and escapepod) are imported
inside the functions that need them, so leech.crf still costs only torch and
numpy on import, and planning code that never extracts pays for neither. Guarded
by a test that resolves the lazy attributes and asserts all three are absent.

Testing

• 25 corpus tests, one per rule and failure mode
• Full suite 1408 passed, 44 skipped
• ruff format --check, ruff check, ty check, zensical build clean

Next

Trainer (CrfTrainer + CLI) over this corpus, then the ONNX export and metrics.
The trainer's acceptance test is a paired 3-seed retrain of barcode_crf_ldx16
against the shipped run — the corpus and all its inputs are intact in scratch.

🤖 Generated with Claude Code