refactor(rust): take the signal-level k-mer path from escapepod-signal (#222)

* refactor(rust): take the signal-level k-mer encoding from escapepod-signal

leech held the only copy of this rule, inside a `crate-type = ["cdylib"]`
Python extension module — so a native runtime for a leech `signal_kmer` model
could not link it and had to transcribe it, which is a second definition that
diverges silently. That is not hypothetical in this stack: `extract_levels` was
written twice with different centring conventions and moved 25 of 100 features,
and escapepod-classify reproduced a superseded feature definition for two
months, its golden missing it because all 19 fixture reads took the other
branch.

It is also the natural pair to `escapepod_signal::mapping`, which *produces*
the base-to-signal map the encoding consumes. The producing half was already
upstream and the consuming half was not. escapepod-rs#272 fixed that; this is
leech taking the call (escapepod-signal v0.16.0).

Two primitives delegate, both byte-identical to what was here:

- `encoding.rs::encode_signal_kmer_inner` -> `seq_encoding::encode_signal_kmer`
- `features.rs::sequence_to_int` -> `seq_encoding::sequence_to_int`, the same
  A/C/G/T-U table with the same `-1` for anything else (upstream's
  `UNKNOWN_BASE` is `-1`)

A third overlap is deliberately NOT delegated. The k-mer context slice in
`signal_mapping.rs` covers the same range with the same padding as upstream's
`sequence_ints_with_context`, but leech needs it as BASES — it serializes
`sequence_with_kmer_context` as a string — where upstream returns ints. Same
rule, different type; swapping it would change a serialized chunk field.

This is a delegation, so identity is the only acceptable outcome, and the
evidence is that nothing had to change to accommodate it: 198 parity tests pass
untouched, including `test_backend_parity.py`, which compares every array in the
npz between the Rust and Python backends — and the Python reference is not
touched by this commit, so it is an independent implementation.

Also here, because the pin moves anyway:

- `escapepod>=0.16.0`, and the full suite re-run against that Python package as
  well as against the Rust crate (they are independent: leech_core links the
  crate statically, `leech.io` imports the wheel). 1467 passed on both.
- `uv.lock` gains onnx/onnxruntime/onnxscript/protobuf. #220 added the `onnx`
  extra to pyproject and never locked it — CI installs from pyproject so it
  worked, but `uv sync --extra onnx` would not have.
- The ONNX contract stops telling consumers to reimplement the encoder and
  names the crate that owns it.

* refactor(rust): take the k-mer context windowing from escapepod-signal too

The third and last overlap, and the one that needed an upstream change first
(escapepod-rs#274, released in 0.16.1). It could not delegate before: leech
needs the context window as BASES, because the corpus serializes
`sequence_with_kmer_context` as a string, where upstream only offered ints.
escapepod-signal now exposes both forms over one windowing rule, with
`sequence_to_int(bases) == ints` pinned by a test there.

So all three halves of the signal-level k-mer path live upstream now:
`mapping` produces the base-to-signal map, `sequence_bases_with_context` cuts
the window it covers plus the k-mer context, and `encode_signal_kmer` scatters
the one-hot context along the signal axis. leech held the middle and the last
of those, inside a cdylib no Rust consumer could link.

This is the one of the three most worth getting out of a second copy. It is
where `before` and `after` are not interchangeable — swap them and every k-mer
is read from a window displaced by `before - after` bases, silently, and
`encode_signal_kmer` cannot detect it because it only sees the total width.

Identical by construction and by test: the ranges agree
(`(seq_start - before)..(seq_end + after)` is upstream's
`core_start - before` plus `n_bases + before + after` with
`n_bases = seq_end - seq_start`), and both pad rather than shift, leech with
`b'N'` and upstream with `UNKNOWN_BASE_CHAR`, which is `b'N'`. 198 parity tests
pass untouched, and `sequence_with_kmer_context` is one of the fields
`test_backend_parity.py` compares array-by-array between the Rust and Python
backends — the Python side is not touched here, so it is an independent check.

Only `rust/Cargo.toml`'s git tag moves to v0.16.1; `Cargo.lock` pins it at
b4c9afae. The `escapepod` Python pin stays `>=0.16.0`, because the new function
is in the Rust crate and not in the Python bindings, and the two are
independent — leech_core links the crate statically, `leech.io` imports the
wheel.

Full suite 1467 passed, 44 skipped.

Author: jayhesselberth

CHANGELOG.md

@@ -7,6 +7,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- **The signal-level k-mer encoding comes from escapepod-signal** rather than
+  being held here (escapepod-rs#271 / #272; requires escapepod 0.16.0).
+  `rust/src/encoding.rs` and `sequence_to_int` are now calls into
+  `escapepod_signal::seq_encoding`.
+
+  leech held the only copy of this rule, inside a `crate-type = ["cdylib"]`
+  Python extension module — so a native runtime for a leech `signal_kmer` model
+  could not link it and had to transcribe it, which is a second definition that
+  diverges silently. It is also the natural pair to `escapepod_signal::mapping`,
+  which *produces* the base-to-signal map the encoding consumes: the producing
+  half was already upstream and the consuming half was not.
+
+  This is a delegation, so the only acceptable outcome is identity: 198 parity
+  tests pass unchanged, including `test_backend_parity.py`, which compares every
+  array in the npz between the Rust and Python backends against a Python
+  reference this change does not touch.
+
+  The k-mer *context slice* delegates too, via `sequence_bases_with_context`
+  (escapepod-rs#274, escapepod 0.16.1). It could not at first: leech needs the
+  window as **bases**, since the corpus serializes `sequence_with_kmer_context`
+  as a string, where upstream only offered ints. Upstream now exposes both forms
+  over one windowing rule, with `sequence_to_int(bases) == ints` pinned by a
+  test there — so the three halves of the signal-level k-mer path (the map, the
+  window, the encoding) all live in `escapepod-signal` and none is duplicated
+  here.
+
+  That third one is the highest-stakes of the three: it is where `before` and
+  `after` are not interchangeable, and swapping them displaces every k-mer
+  silently because the encoder only sees the total width. It is also the most
+  directly checkable — `sequence_with_kmer_context` is one of the fields
+  `test_backend_parity.py` compares array-by-array between backends.
+
+  Only `rust/Cargo.toml`'s git tag moves to v0.16.1; the `escapepod` Python pin
+  stays `>=0.16.0`, because the new function is in the Rust crate and not in the
+  Python bindings.
+
 ### Added
 
 - **ONNX export, for the classifier arms and the CRF encoder** (#217).

CLAUDE.md

@@ -344,12 +344,25 @@ escapepod reads as one DP pass.
 Since escapepod v0.15.0 the settings themselves are escapepod's
 `RefineSettings::move_table_refinement` preset (escapepod-rs#257), so there is
 no longer a literal here to drift. **Do not hand-build `RefineSettings` in this
-crate again** — that is what the preset exists to prevent. Four primitives now
+crate again** — that is what the preset exists to prevent. Five primitives now
 come from upstream rather than being held locally: the preset, the POD5 reader
 cache (`escapepod_signal::cached_reader`), per-base statistics
 (`features::span_stats`, configured `SpanFill::Zero` / `SpanBounds::Clamp` /
-`MedianConvention::SortPartialCmp`), and the move-table and CIGAR coordinate
-mapping (`escapepod_signal::mapping`).
+`MedianConvention::SortPartialCmp`), the move-table and CIGAR coordinate mapping
+(`escapepod_signal::mapping`), and the whole signal-level k-mer path —
+the base alphabet, the context windowing and the encoding itself
+(`escapepod_signal::seq_encoding`, escapepod-rs#272 and #274).
+
+**All three halves of that path are now upstream, and that is the point.**
+`mapping` produces the base-to-signal map, `sequence_bases_with_context` cuts
+the window it covers plus the k-mer context, and `encode_signal_kmer` scatters
+the one-hot context along the signal axis. leech held the middle and the last
+for a while, inside a `cdylib` no Rust consumer could link. The windowing is the
+one to watch: `before` and `after` are **not** interchangeable there, swapping
+them displaces every k-mer, and the encoder cannot detect it because it only
+sees the total width. leech keeps the window as *bases* rather than ints because
+the corpus serializes `sequence_with_kmer_context` as a string —
+`sequence_ints_with_context` is the same window in the other alphabet.
 
 ### The corpus is written and merged in exactly one place each
 
@@ -529,20 +542,14 @@ the graph. The contract names the function and its parameters rather than
 leaving them to be re-derived — the same choice escapepod-rs's charging bundle
 makes by carrying its recipe in `metadata.json`.
 
-**But "one definition" is not yet true, and the fix is upstream.** `leech-core`
-ships the encoder, and it is tempting to say a consumer should just call it —
-except `leech_core` is `crate-type = ["cdylib"]`, a Python extension module, so
-escapepod-rs cannot link it. The primitive itself
-(`rust/src/encoding.rs::encode_signal_kmer_inner`) is pure, dependency-free and
-carries no model vocabulary: sequence ints, a base-to-signal map, a signal
-length and a k-mer context in, a `(4 * kmer_len, signal_len)` scatter out. That
-is an `escapepod-signal` primitive by every rule this stack already applies —
-and escapepod-signal owns `mapping`, which *produces* the map this consumes, so
-today the producer is upstream and the consumer is downstream. Moving it is
-rnabioco/escapepod-rs#271. Until it does,
-any Rust consumer has to transcribe it, which is a second definition, and the
-mitigation is a cross-language golden of the kind the CRF decode and the
-charging features already have.
+**And the rule has exactly one definition, upstream.** It briefly did not:
+`leech_core` held the only copy, inside a `crate-type = ["cdylib"]` Python
+extension module that Rust cannot link, so a native runtime had to transcribe
+it. `escapepod_signal::seq_encoding` owns it now (escapepod-rs#271 / #272), and
+`rust/src/encoding.rs` is a call into it — the natural pairing, since
+`escapepod_signal::mapping` *produces* the base-to-signal map this consumes.
+A Rust consumer links the crate; a Python one calls `_rs_encode_signal_kmer`,
+which is the same code. **Do not reimplement it on either side.**
 
 **Verification crosses the serialization boundary.** `verify_onnx` runs
 onnxruntime against torch and returns the max absolute difference, which is what

pyproject.toml

@@ -25,7 +25,7 @@ dependencies = [
   "click>=8.1.0",
   # POD5 I/O and signal normalization. Imported unconditionally by
   # leech.io.pod5_reader and leech.features, so it cannot be optional.
-  "escapepod>=0.15.0",
+  "escapepod>=0.16.0",
   "numpy>=2.0.0",
   "nvitop>=1.6.1",
   "polars>=1.43.2",

rust/Cargo.lock

@@ -28,7 +28,7 @@ pyo3 = { version = "0.28", features = ["extension-module", "abi3-py312"] }
 numpy = "0.28"
 # Not on crates.io; pulled straight from the public GitHub repo. Pin by tag so
 # the build is reproducible — bump the tag to take a new escapepod release.
-escapepod-signal = { git = "https://github.com/rnabioco/escapepod-rs", tag = "v0.15.0" }
+escapepod-signal = { git = "https://github.com/rnabioco/escapepod-rs", tag = "v0.16.1" }
 rayon = "1"
 
 [profile.release]

rust/Cargo.toml

@@ -2,43 +2,31 @@ use numpy::ndarray::Array2;
 use numpy::{IntoPyArray, PyArray2, PyReadonlyArray1};
 use pyo3::prelude::*;
 
-/// Internal signal-level kmer encoding. Returns flat row-major Vec of shape
-/// `(4 * kmer_len, signal_len)`.
+/// Signal-level k-mer encoding, from `escapepod-signal`.
+///
+/// The rule used to live here, and it was the only copy — inside a `cdylib`
+/// that Rust cannot link, so any native runtime for a leech `signal_kmer` model
+/// had to transcribe it (rnabioco/escapepod-rs#271). It is the natural pair to
+/// `escapepod_signal::mapping`, which *produces* the base-to-signal map this
+/// consumes, so upstream now owns both halves and this is a call.
+///
+/// Returns a flat row-major `Vec` of shape `(4 * kmer_len, signal_len)`.
 pub(crate) fn encode_signal_kmer_inner(
     seq_ints: &[i8],
     sig_map: &[i64],
     signal_len: usize,
     kmer_before: usize,
     kmer_after: usize,
 ) -> Vec<f32> {
-    let kmer_len = kmer_before + 1 + kmer_after;
-    let seq_len = sig_map.len().saturating_sub(1);
-    let enc_dim = 4 * kmer_len;
-    let mut enc = vec![0.0f32; enc_dim * signal_len];
-
-    for kmer_pos in 0..kmer_len {
-        let offset = 4 * kmer_pos;
-        for seq_pos in 0..seq_len {
-            let base_idx = seq_pos + kmer_pos;
-            if base_idx >= seq_ints.len() {
-                continue;
-            }
-            let base = seq_ints[base_idx];
-            if base < 0 {
-                continue;
-            }
-            let sig_start = (sig_map[seq_pos] as usize).min(signal_len);
-            let sig_end = (sig_map[seq_pos + 1] as usize).min(signal_len);
-            if sig_start < sig_end {
-                let row = offset + base as usize;
-                for s in sig_start..sig_end {
-                    enc[row * signal_len + s] = 1.0;
-                }
-            }
-        }
-    }
-
-    enc
+    escapepod_signal::seq_encoding::encode_signal_kmer(
+        seq_ints,
+        sig_map,
+        signal_len,
+        escapepod_signal::seq_encoding::KmerContext {
+            before: kmer_before,
+            after: kmer_after,
+        },
+    )
 }
 
 /// PyO3 wrapper for signal-level kmer encoding.

rust/src/encoding.rs

@@ -98,17 +98,12 @@ pub(super) fn compute_signal_residual(
     residual
 }
 
+/// Bases as `0..3`, anything else `-1`. From `escapepod-signal`, which owns
+/// the alphabet the signal-level encodings are written in; the table here was
+/// identical, and two copies of an alphabet is how a `U` starts meaning
+/// something different on one side.
 pub(super) fn sequence_to_int(sequence: &[u8]) -> Vec<i8> {
-    sequence
-        .iter()
-        .map(|&c| match c {
-            b'A' | b'a' => 0,
-            b'C' | b'c' => 1,
-            b'G' | b'g' => 2,
-            b'T' | b't' | b'U' | b'u' => 3,
-            _ => -1,
-        })
-        .collect()
+    escapepod_signal::seq_encoding::sequence_to_int(sequence)
 }
 
 pub(super) fn encode_base_onehot(sequence: &[u8]) -> Vec<f32> {

rust/src/inference_pipeline/features.rs

@@ -112,15 +112,25 @@ pub(super) fn chunk_signal_kmer_inputs(
     map[0] = 0;
     map[last] = chunk_len as i64;
 
+    // The windowing comes from escapepod-signal, which also owns the map this
+    // sits between and the encoding it feeds. Kept as BASES rather than ints
+    // because the corpus serializes `sequence_with_kmer_context` as a string;
+    // `sequence_ints_with_context` is the same window in the other alphabet,
+    // and `sequence_to_int` of this is exactly that (escapepod-rs#274).
+    //
+    // This is the step where `before` and `after` are NOT interchangeable:
+    // swapping them displaces every k-mer silently, and the encoder cannot
+    // detect it because it only sees the total width.
     let (kmer_before, kmer_after) = skmer_ctx;
-    let ctx_lo = seq_start as i64 - kmer_before as i64;
-    let ctx_hi = seq_end as i64 + kmer_after as i64;
-    let ctx: Vec<u8> = (ctx_lo..ctx_hi)
-        .map(|i| match usize::try_from(i) {
-            Ok(u) if u < seq_bytes.len() => seq_bytes[u],
-            _ => b'N',
-        })
-        .collect();
+    let ctx = escapepod_signal::seq_encoding::sequence_bases_with_context(
+        seq_bytes,
+        seq_start,
+        seq_end - seq_start,
+        escapepod_signal::seq_encoding::KmerContext {
+            before: kmer_before,
+            after: kmer_after,
+        },
+    );
 
     (map, ctx)
 }

rust/src/inference_pipeline/signal_mapping.rs

@@ -87,11 +87,13 @@ def describe_inputs(config: dict, example_inputs: tuple) -> list[InputSpec]:
             "leech.features.encode_signal_kmer(signal, seq_to_sig_map, sequence, "
             f"context={list(kmer_context)}) — a scatter of the one-hot k-mer context "
             "along the signal axis. It lives in the dataset, not the model, so it is "
-            "NOT in this graph. Python consumers should call leech-core's "
-            "`_rs_encode_signal_kmer`. A non-Python consumer currently has to "
-            "reimplement it — leech_core is a cdylib and cannot be linked from Rust — "
-            "so pin any reimplementation against a golden rather than trusting it; "
-            "two definitions of this rule diverge silently."
+            "NOT in this graph — a consumer must produce it before calling the "
+            "model. The rule is owned by escapepod-signal "
+            "(`escapepod_signal::seq_encoding::encode_signal_kmer`), which leech "
+            "calls rather than duplicates; a Rust consumer should link that crate, "
+            "and a Python one can call leech-core's `_rs_encode_signal_kmer`, which "
+            "is the same code. Do not reimplement it — two definitions of this rule "
+            "diverge silently."
         )
     else:
         produced_by = "one-hot encoding of the k-mer context, 4 channels"