From 3c585c2a3df3fffec218d2e1ef3c06e1baca67a5 Mon Sep 17 00:00:00 2001 From: Logan Nye <87274608+logannye@users.noreply.github.com> Date: Wed, 27 May 2026 08:50:36 -0700 Subject: [PATCH 1/2] docs(roadmap): open-problems thesis + re-derived B->E sequencing Adds docs/OPEN_PROBLEMS.md: the research thesis (memory as a declared, predictable, never-refusing, verifiable contract; the sqrt(t*loglog t) machinery as the continuous space/time knob), the open problem (time-vs-space; the intermediate-state-vs-input lens), sublinear-space index construction as the beachhead, and the re-derived B->E roadmap. Realigns target-architecture s1 (sharpened bet), s8 (theory layer KEPT + repurposed as load-bearing, superseding the feature-gate decision), and s9 (B genome-scale -> C memory-contract -> D space-bounded construction -> E reach+substrate). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/OPEN_PROBLEMS.md | 162 ++++++++++++++++++ ...2026-05-26-rosalind-target-architecture.md | 59 +++++-- 2 files changed, 202 insertions(+), 19 deletions(-) create mode 100644 docs/OPEN_PROBLEMS.md diff --git a/docs/OPEN_PROBLEMS.md b/docs/OPEN_PROBLEMS.md new file mode 100644 index 0000000..9b4dc06 --- /dev/null +++ b/docs/OPEN_PROBLEMS.md @@ -0,0 +1,162 @@ +# Rosalind — Open Problems & Research Thesis + +**Status:** Living research-direction document — 2026-05-27. Audience: builders and researchers +who fork, extend, or build on Rosalind. Companion to the engineering roadmap in +[`docs/superpowers/specs/2026-05-26-rosalind-target-architecture.md`](superpowers/specs/2026-05-26-rosalind-target-architecture.md). + +--- + +## 1. The thesis + +Rosalind does not try to out-align bwa-mem2/minimap2 or out-call DeepVariant — that race is a +non-goal. Its differentiated, defensible contribution is a property almost no production genomics +tool offers: + +> **Memory is a declared, predictable, never-refusing, verifiable contract** — across the entire +> genomics lifecycle (index construction, alignment, pileup, calling, query), with a **continuous +> space/time tradeoff** the builder selects by stating a RAM budget. + +You tell Rosalind the RAM you have; it tells you — *before you commit* — whether the job fits and +how long it will take; it then honors that ceiling, sliding along a space/time curve to do so; and +when the budget is below the comfortable in-RAM regime it **degrades gracefully** (more +recomputation, then spill to disk) rather than refusing. Every run emits a receipt recording the +*realized* peak working set, and CI enforces the bound. This is the capability that matters to the +people who actually need Rosalind: **edge, field, clinical, and large-/meta-/pan-genome** settings +where RAM is fixed, swap is death, and an unpredictable OOM at hour 20 of a build is unacceptable. + +## 2. The mechanism: a space/time *curve*, not an operating point + +The enabling machinery is the square-root-space evaluation framework Rosalind is built over +(Williams 2025, *Simulating Time with Square-Root Space*; Cook–Mertz 2024 tree evaluation). The +headline is usually written *O(√t)*; the construction here realizes a space bound of +**O(√(t · log log t))** (a refinement of Williams' *O(√(t log t))*; the "O(√t)" in the code and +older docs is the simplified label). + +The key reframing — and what makes this a *systems* contribution rather than a theory citation — +is that this is **not a single low-space operating point.** Block-respecting simulation with a +tunable block size `b` (and checkpoint density) yields a **continuous curve**: from `b = t` +(standard, fast, O(t) space) through `b = √t` (minimal space ≈ O(√(t · log log t)), more +recomputation) and onward toward external-memory spill. **A declared RAM budget simply selects the +point on that curve.** The theory layer's job is to *be the knob.* + +## 3. The open problem + +The Williams/Cook–Mertz result is a **space** upper bound that, in the general simulation, **buys +space by spending time** (recomputation). So the real research question is not "can we hit +√(t·log log t) space" — it is: + +> **For which genomics computations can we realize the full space/time curve with *tolerable time +> overhead*?** I.e., where is the trade a small polynomial / modest constant factor rather than a +> blow-up? + +### 3.1 The decisive lens: intermediate-state-bound vs. input-bound + +The simulation shrinks the **working state** of a time-*t* computation. It does **nothing** to +shrink the memory needed to *hold the inputs or outputs.* This single fact tells you where the +framework can and cannot help: + +| Computation | Dominant memory | Framework applies? | +|---|---|---| +| **FM-index / suffix-array construction** | the suffix array + recursion **workspace** (≈4–8·n), not the reference (≈n/4 packed) — **intermediate state** | **Yes — sweet spot** | +| Long-sequence/graph **DP** (alignment) | the DP matrix — **intermediate state** (though Hirschberg/banding already help the common cases) | Partially | +| **Joint / cohort calling** | holding *N* samples' data — **input** (O(N)) | No — beat by streaming samples, not √t | +| **Pangenome-graph** analysis | the graph/index itself — **input** | No — the graph dominates; √t doesn't shrink it | + +This is *why* construction is the beachhead, and the honest test any future target must pass: +**is the memory bottleneck the computation's working state, or its inputs?** Input-bound problems +(cohort calling, pangenomes) are real and important, but they need *orthogonal* techniques +(streaming, succinct/compressed inputs) — they are framed here as separate open problems, not as +things this framework will magically solve. + +### 3.2 The beachhead: sublinear-space full-text index construction + +Building the FM-index/BWT is Rosalind's own headline memory limitation ("the FM-index is built in +memory at the start of each run; memory proportional to the reference"). It is the ideal first +target because it is: + +1. **Intermediate-state-bound** — the SA-IS suffix array + recursion workspace dominate; the input + (reference) is small. Exactly the framework's sweet spot. +2. **One-time and offline** — so the time-for-space trade is *most tolerable*: build slowly, even on + the constrained device itself; distribute the artifact; mmap it forever. +3. **On the critical path** — it unblocks indexing references that do not fit in RAM today (T2T, + wheat ~16 Gbp, conifers, metagenomic catalogs, pangenome linearizations) on commodity hardware, + and completes Rosalind's bounded-memory story *end to end* (build *and* use within a declared + budget). +4. **A clean, composable primitive** — useful to a builder immediately, not a giant end-to-end + subsystem they must adopt wholesale. + +**Open question (D):** characterize and minimize the time overhead of budget-tunable, sublinear- +space full-text index construction; identify the regimes where it is practical. + +### 3.3 The broader landscape (framed, not yet committed) + +- **DP-heavy steps under budget** — long-read/graph alignment DP as a tunable space/time computation. +- **Input-bound frontiers** — space-bounded joint/cohort calling and pangenome-graph analysis; + these require streaming + succinct-structure techniques, *complementary* to (not solved by) the + √t machinery. Honest open problems. + +## 4. The contract design (what "declared memory" means concretely) + +A single declared budget governs the whole system, not just one stage: + +- **Declare → predict.** `rosalind plan --budget 4G …` reports *feasibility*, the *predicted peak + working set*, the *estimated time*, and the *curve* ("at 8 GB → ~3× faster") — so you commit with + your eyes open. No surprise OOM, no surprise multi-day run. +- **Honor or refuse — then degrade rather than refuse.** Every streaming stage exposes a provable + working-set bound and honors the ceiling; below the in-RAM threshold it slides further down the + curve (more recomputation → external-memory spill) and **still completes**, slower. A 2 GB field + device builds a human index; it does not get "won't fit." +- **Lifecycle-wide.** The same budget bounds **construction, mmap-query (tunable SA-sampling), + alignment, pileup, calling, and sort.** State your RAM once. +- **Verifiable.** The run receipt records the *realized* peak working set (content-addressed, + alongside tool version + input/param/output hashes); CI enforces the envelope on growing inputs; + `rosalind verify` re-checks a receipt without re-running. +- **Deterministic regardless of the point chosen.** The same inputs + budget produce byte-identical + artifacts; parallelism is thread-count-invariant. The space/time point never changes the *output*. + +## 5. Roadmap (B → E), re-derived around the thesis + +Prioritized for the builders who need the unique capability first; performance/breadth follow. The +practical pillars (B, C) stand alone — a great engine even if the research bet (D) runs long — and +D lands the space-complexity headline on top. + +- **B — Genome-scale kernel (finish).** Zero-copy persisted lean multi-contig index (build-once → + mmap), wired consumers, whole-genome pileup, pipe-native. Lay the `MemoryBudget` / space-accounting + hooks. The theory layer (`algebra`/`ledger`/`machine`/`tree`/`space`) is **kept and repurposed**, + not feature-gated away. +- **C — Memory as a verifiable contract.** Declared `MemoryBudget` threaded through every streaming + stage + mmap-query; `rosalind plan` (envelope + curve); honor-or-refuse + graceful degradation; + real RSS gates in CI; deterministic (thread-count-invariant) execution + receipts + `rosalind + verify`. → "declare your RAM → predictable, honored, verifiable, end-to-end" for + align/call/query/sort. +- **D — Sublinear-space index construction (beachhead research contribution).** The √t-family knob: + build the index under the declared budget across the full curve (in-RAM-fast → checkpointed + √(n·log log n) → external-memory spill), with the time overhead characterized; makes the theory + layer load-bearing; erases the O(reference) build-RAM caveat; completes the contract for the build + step. +- **E — Reach + substrate.** Fast deterministic-parallel aligner + germline indels + richer read QC + (broad usability); the Python/tensor **pileup-feature substrate** + pluggable models (the + ML-researcher/builder surface); rigorous eval + calibration validation + the open-problems + benchmark suite. + +(Deterministic-parallel *execution* is part of the contract in **C**; aligner-*algorithm* quality and +micro-optimization are **E**. The existing exact-match aligner is sufficient to exercise C and D on +real data — the point there is memory, not speed.) + +## 6. What this means for you + +- **Edge / field / clinical builders** — declare your device's RAM; get a checkable guarantee it + fits, a time estimate, byte-identical + auditable outputs, and a build that degrades rather than + dies. +- **Large / meta / pangenome researchers** — index references that do not fit in RAM today, on + commodity hardware. +- **ML-genomics builders** — (Phase E) a fast, deterministic, memory-bounded, Python/tensor- + accessible per-locus feature stream over a whole genome to compute on. +- **Algorithms / systems researchers** — a working testbed for space-bounded computation on a real, + memory-bound domain, and a concrete open problem (§3) with a clean first target. + +## 7. References + +- R. R. Williams. *Simulating Time with Square-Root Space.* 2025. +- J. Cook, I. Mertz. *Tree Evaluation in Sublinear Space* (catalytic / tree-evaluation lineage). 2024. +- The simplified "O(√t)" framing in the source headers refers to the O(√(t · log log t)) bound above. diff --git a/docs/superpowers/specs/2026-05-26-rosalind-target-architecture.md b/docs/superpowers/specs/2026-05-26-rosalind-target-architecture.md index 6b49768..c658b9b 100644 --- a/docs/superpowers/specs/2026-05-26-rosalind-target-architecture.md +++ b/docs/superpowers/specs/2026-05-26-rosalind-target-architecture.md @@ -36,6 +36,14 @@ hospital laptops, portable sequencers (MinION in the field), classrooms, and res want to *build on* a genomics engine rather than shell out to one. Variant calling is the first consumer of the kernel, not the whole product. +**Sharpened core bet (2026-05-27).** The differentiator is made precise: **memory is a declared, +predictable, never-refusing, verifiable contract across the whole lifecycle**, with the +square-root-space machinery (Williams 2025 / Cook–Mertz 2024, bound *O(√(t·log log t))*) as the +**continuous space/time knob** a declared RAM budget selects — beachheaded on **sublinear-space +index construction**, the one stage where the framework genuinely bites. The research thesis, the +open problem (time-vs-space), and the intermediate-state-vs-input lens are in +[`docs/OPEN_PROBLEMS.md`](../../OPEN_PROBLEMS.md). + Non-goals (on purpose, so the promises are airtight): out-accuracy-ing GATK/DeepVariant on benchmarks; cloud/cluster-scale orchestration; supporting every exotic format. @@ -162,33 +170,46 @@ we RSS-gate (promise #2), expose to Python (promise #5/embeddable), and let plug - **Rebuild approach: evolve toward the target** — rewrite the wrong layers; re-home/extend the correct tested kernels; each phase lands green; no throwaway of working code. -- **Theory layer: separate as a feature-gated `theory/` demo**; `ff`/`ark-poly` become optional - (`theory = ["dep:ff","dep:ark-poly"]`); decoupled from the genomics core. +- **Theory layer: KEEP and repurpose as load-bearing (revised 2026-05-27).** The √t / Cook–Mertz + machinery (`algebra`/`ledger`/`machine`/`tree`/`space`) becomes the *knob* that realizes the + budget-tunable space/time curve — beachheaded on sublinear-space index construction (Phase D; see + `docs/OPEN_PROBLEMS.md`). It is **not** feature-gated away, and `ff`/`ark-poly` stay as core deps. + (Supersedes the earlier "feature-gate the `theory/` demo" decision.) - **Guiding principle for all open design choices:** maximize *unique* value to edge builders — the kernel/substrate, memory-as-contract, calibrated honesty, verifiable reproducibility, embeddability (§1, §3). --- -## 9. Sequencing +## 9. Sequencing (re-derived 2026-05-27 — see `docs/OPEN_PROBLEMS.md`) + +Reprioritized around the core contribution — *memory as a declared, predictable, never-refusing, +verifiable contract*, beachheaded on sublinear-space index construction — and for the builders who +need that capability first (edge / field / large-/meta-/pan-genome), ahead of generic performance. +We commit to the optimal end-state rather than shippable intermediates. (The practical pillars B and +C stand alone; D lands the space-complexity headline on top, de-risking the research bet.) -1. **Phase A — the kernel + calling vertical + receipt.** `core/` (locus, sequence, record, - budget, `PileupColumn`), the public `pileup/` engine, `call/germline` (calibrated GL model) - + re-homed `call/somatic`, `io/vcf` (spec-valid), and a **minimal `provenance/` receipt** so - the first improved outputs are already verifiable. Detailed in +1. **Phase A — kernel + calling vertical + receipt.** ✅ Done. Detailed in `2026-05-26-phase-a-unified-pileup-genotype-design.md`. -2. **Phase B — make it real on a genome.** Index persistence (build-once → mmap), multi-contig, - streaming/gzip/multi-record ingestion, pipe-native composition. -3. **Phase D — fast & honest, memory-as-contract.** rayon (deterministic), `.csi` region - `fetch`, **real RSS gates + `rosalind plan`** (budget envelope) + `rosalind verify` (receipt - check), honest memory docs. -4. **Phase C — scoped calling.** Germline indels, richer read QC (mate-overlap dedup, - strand-bias filter, BQ/BAQ), long-read calling refinements. -5. **Phase P — Python substrate binding.** Expose the `PileupColumn` stream + index/align/call - to Python with type stubs, pytest, and a wheel matrix (manylinux/macOS/aarch64). Sequenced - right after the engine API stabilizes (after A/B). -6. **Phase E — interleaved.** Theory separation (§8), CI hardening (clippy `-D warnings`, - bcftools validation, wheels), README-honesty pass. +2. **Phase B — genome-scale kernel.** Zero-copy persisted lean multi-contig index (build-once → + mmap), wired consumers, whole-genome pileup, pipe-native; lay the `MemoryBudget` / + space-accounting hooks. (B1 streaming readers + B2 multi-contig FM-index ✅ done; B3 zero-copy + persistence + B4 wiring remain.) Detailed in `2026-05-27-phase-b-genome-scale-design.md`. +3. **Phase C — memory as a verifiable contract.** Declared `MemoryBudget` across every streaming + stage + mmap-query; `rosalind plan` (envelope + space/time curve); honor-or-refuse + graceful + degradation; **real RSS gates** in CI; deterministic (thread-count-invariant) execution + + receipts + `rosalind verify`. +4. **Phase D — sublinear-space index construction (beachhead).** The √t-family knob: build the index + under the declared budget across the full curve (in-RAM-fast → checkpointed √(n·log log n) → + external-memory spill), time overhead characterized; theory layer load-bearing; erases the + O(reference) build-RAM caveat. The headline space-complexity contribution. +5. **Phase E — reach + substrate.** Fast deterministic-parallel aligner + germline indels + richer + read QC; the Python/tensor **pileup-feature substrate** + pluggable models; rigorous eval + + calibration validation + the open-problems benchmark. (Deterministic-parallel *execution* is part + of C's contract; aligner-*algorithm* quality + micro-optimization are E.) + +The open research problem and the intermediate-state-vs-input lens that order these phases are in +[`docs/OPEN_PROBLEMS.md`](../../OPEN_PROBLEMS.md). --- From f2c4a060c5c4a9ca92402b2dc757640dd3e3c536 Mon Sep 17 00:00:00 2001 From: Logan Nye <87274608+logannye@users.noreply.github.com> Date: Wed, 27 May 2026 10:04:06 -0700 Subject: [PATCH 2/2] =?UTF-8?q?docs:=20simplify=20the=20space=20bound=20to?= =?UTF-8?q?=20~=E2=88=9At=20(drop=20the=20=E2=88=9A(t=C2=B7log=20log=20t)?= =?UTF-8?q?=20notation)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/OPEN_PROBLEMS.md | 13 ++++++------- .../2026-05-26-rosalind-target-architecture.md | 4 ++-- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/OPEN_PROBLEMS.md b/docs/OPEN_PROBLEMS.md index 9b4dc06..c2488b3 100644 --- a/docs/OPEN_PROBLEMS.md +++ b/docs/OPEN_PROBLEMS.md @@ -28,14 +28,13 @@ where RAM is fixed, swap is death, and an unpredictable OOM at hour 20 of a buil The enabling machinery is the square-root-space evaluation framework Rosalind is built over (Williams 2025, *Simulating Time with Square-Root Space*; Cook–Mertz 2024 tree evaluation). The -headline is usually written *O(√t)*; the construction here realizes a space bound of -**O(√(t · log log t))** (a refinement of Williams' *O(√(t log t))*; the "O(√t)" in the code and -older docs is the simplified label). +space bound is **~√t** — the square root of the running time, up to lower-order (sub-polynomial) +factors. (The "O(√t)" in the source headers denotes the same bound.) The key reframing — and what makes this a *systems* contribution rather than a theory citation — is that this is **not a single low-space operating point.** Block-respecting simulation with a tunable block size `b` (and checkpoint density) yields a **continuous curve**: from `b = t` -(standard, fast, O(t) space) through `b = √t` (minimal space ≈ O(√(t · log log t)), more +(standard, fast, O(t) space) through `b = √t` (minimal space ~√t, more recomputation) and onward toward external-memory spill. **A declared RAM budget simply selects the point on that curve.** The theory layer's job is to *be the knob.* @@ -43,7 +42,7 @@ point on that curve.** The theory layer's job is to *be the knob.* The Williams/Cook–Mertz result is a **space** upper bound that, in the general simulation, **buys space by spending time** (recomputation). So the real research question is not "can we hit -√(t·log log t) space" — it is: +~√t space" — it is: > **For which genomics computations can we realize the full space/time curve with *tolerable time > overhead*?** I.e., where is the trade a small polynomial / modest constant factor rather than a @@ -131,7 +130,7 @@ D lands the space-complexity headline on top. align/call/query/sort. - **D — Sublinear-space index construction (beachhead research contribution).** The √t-family knob: build the index under the declared budget across the full curve (in-RAM-fast → checkpointed - √(n·log log n) → external-memory spill), with the time overhead characterized; makes the theory + ~√n → external-memory spill), with the time overhead characterized; makes the theory layer load-bearing; erases the O(reference) build-RAM caveat; completes the contract for the build step. - **E — Reach + substrate.** Fast deterministic-parallel aligner + germline indels + richer read QC @@ -159,4 +158,4 @@ real data — the point there is memory, not speed.) - R. R. Williams. *Simulating Time with Square-Root Space.* 2025. - J. Cook, I. Mertz. *Tree Evaluation in Sublinear Space* (catalytic / tree-evaluation lineage). 2024. -- The simplified "O(√t)" framing in the source headers refers to the O(√(t · log log t)) bound above. +- The "O(√t)" label in the source headers denotes the same ~√t bound (square-root space, up to lower-order factors). diff --git a/docs/superpowers/specs/2026-05-26-rosalind-target-architecture.md b/docs/superpowers/specs/2026-05-26-rosalind-target-architecture.md index c658b9b..6a4acba 100644 --- a/docs/superpowers/specs/2026-05-26-rosalind-target-architecture.md +++ b/docs/superpowers/specs/2026-05-26-rosalind-target-architecture.md @@ -38,7 +38,7 @@ consumer of the kernel, not the whole product. **Sharpened core bet (2026-05-27).** The differentiator is made precise: **memory is a declared, predictable, never-refusing, verifiable contract across the whole lifecycle**, with the -square-root-space machinery (Williams 2025 / Cook–Mertz 2024, bound *O(√(t·log log t))*) as the +square-root-space machinery (Williams 2025 / Cook–Mertz 2024; bound **~√t**, up to lower-order factors) as the **continuous space/time knob** a declared RAM budget selects — beachheaded on **sublinear-space index construction**, the one stage where the framework genuinely bites. The research thesis, the open problem (time-vs-space), and the intermediate-state-vs-input lens are in @@ -200,7 +200,7 @@ C stand alone; D lands the space-complexity headline on top, de-risking the rese degradation; **real RSS gates** in CI; deterministic (thread-count-invariant) execution + receipts + `rosalind verify`. 4. **Phase D — sublinear-space index construction (beachhead).** The √t-family knob: build the index - under the declared budget across the full curve (in-RAM-fast → checkpointed √(n·log log n) → + under the declared budget across the full curve (in-RAM-fast → checkpointed ~√n → external-memory spill), time overhead characterized; theory layer load-bearing; erases the O(reference) build-RAM caveat. The headline space-complexity contribution. 5. **Phase E — reach + substrate.** Fast deterministic-parallel aligner + germline indels + richer