Skip to content

docs: open-problems thesis + re-derived B→E roadmap (space-bounded genomics as a memory contract) #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
logannye merged 2 commits into from
May 27, 2026
Merged

docs: open-problems thesis + re-derived B→E roadmap (space-bounded genomics as a memory contract) #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3c585c2
docs(roadmap): open-problems thesis + re-derived B->E sequencing
logannye May 27, 2026
f2c4a06
docs: simplify the space bound to ~√t (drop the √(t·log log t) notation)
logannye May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 161 additions & 0 deletions docs/OPEN_PROBLEMS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
# Rosalind — Open Problems & Research Thesis

**Status:** Living research-direction document — 2026-05-27. Audience: builders and researchers
who fork, extend, or build on Rosalind. Companion to the engineering roadmap in
[`docs/superpowers/specs/2026-05-26-rosalind-target-architecture.md`](superpowers/specs/2026-05-26-rosalind-target-architecture.md).

---

## 1. The thesis

Rosalind does not try to out-align bwa-mem2/minimap2 or out-call DeepVariant — that race is a
non-goal. Its differentiated, defensible contribution is a property almost no production genomics
tool offers:

> **Memory is a declared, predictable, never-refusing, verifiable contract** — across the entire
> genomics lifecycle (index construction, alignment, pileup, calling, query), with a **continuous
> space/time tradeoff** the builder selects by stating a RAM budget.

You tell Rosalind the RAM you have; it tells you — *before you commit* — whether the job fits and
how long it will take; it then honors that ceiling, sliding along a space/time curve to do so; and
when the budget is below the comfortable in-RAM regime it **degrades gracefully** (more
recomputation, then spill to disk) rather than refusing. Every run emits a receipt recording the
*realized* peak working set, and CI enforces the bound. This is the capability that matters to the
people who actually need Rosalind: **edge, field, clinical, and large-/meta-/pan-genome** settings
where RAM is fixed, swap is death, and an unpredictable OOM at hour 20 of a build is unacceptable.

## 2. The mechanism: a space/time *curve*, not an operating point

The enabling machinery is the square-root-space evaluation framework Rosalind is built over
(Williams 2025, *Simulating Time with Square-Root Space*; Cook–Mertz 2024 tree evaluation). The
space bound is **~√t** — the square root of the running time, up to lower-order (sub-polynomial)
factors. (The "O(√t)" in the source headers denotes the same bound.)

The key reframing — and what makes this a *systems* contribution rather than a theory citation —
is that this is **not a single low-space operating point.** Block-respecting simulation with a
tunable block size `b` (and checkpoint density) yields a **continuous curve**: from `b = t`
(standard, fast, O(t) space) through `b = √t` (minimal space ~√t, more
recomputation) and onward toward external-memory spill. **A declared RAM budget simply selects the
point on that curve.** The theory layer's job is to *be the knob.*

## 3. The open problem

The Williams/Cook–Mertz result is a **space** upper bound that, in the general simulation, **buys
space by spending time** (recomputation). So the real research question is not "can we hit
~√t space" — it is:

> **For which genomics computations can we realize the full space/time curve with *tolerable time
> overhead*?** I.e., where is the trade a small polynomial / modest constant factor rather than a
> blow-up?

### 3.1 The decisive lens: intermediate-state-bound vs. input-bound

The simulation shrinks the **working state** of a time-*t* computation. It does **nothing** to
shrink the memory needed to *hold the inputs or outputs.* This single fact tells you where the
framework can and cannot help:

| Computation | Dominant memory | Framework applies? |
|---|---|---|
| **FM-index / suffix-array construction** | the suffix array + recursion **workspace** (≈4–8·n), not the reference (≈n/4 packed) — **intermediate state** | **Yes — sweet spot** |
| Long-sequence/graph **DP** (alignment) | the DP matrix — **intermediate state** (though Hirschberg/banding already help the common cases) | Partially |
| **Joint / cohort calling** | holding *N* samples' data — **input** (O(N)) | No — beat by streaming samples, not √t |
| **Pangenome-graph** analysis | the graph/index itself — **input** | No — the graph dominates; √t doesn't shrink it |

This is *why* construction is the beachhead, and the honest test any future target must pass:
**is the memory bottleneck the computation's working state, or its inputs?** Input-bound problems
(cohort calling, pangenomes) are real and important, but they need *orthogonal* techniques
(streaming, succinct/compressed inputs) — they are framed here as separate open problems, not as
things this framework will magically solve.

### 3.2 The beachhead: sublinear-space full-text index construction

Building the FM-index/BWT is Rosalind's own headline memory limitation ("the FM-index is built in
memory at the start of each run; memory proportional to the reference"). It is the ideal first
target because it is:

1. **Intermediate-state-bound** — the SA-IS suffix array + recursion workspace dominate; the input
(reference) is small. Exactly the framework's sweet spot.
2. **One-time and offline** — so the time-for-space trade is *most tolerable*: build slowly, even on
the constrained device itself; distribute the artifact; mmap it forever.
3. **On the critical path** — it unblocks indexing references that do not fit in RAM today (T2T,
wheat ~16 Gbp, conifers, metagenomic catalogs, pangenome linearizations) on commodity hardware,
and completes Rosalind's bounded-memory story *end to end* (build *and* use within a declared
budget).
4. **A clean, composable primitive** — useful to a builder immediately, not a giant end-to-end
subsystem they must adopt wholesale.

**Open question (D):** characterize and minimize the time overhead of budget-tunable, sublinear-
space full-text index construction; identify the regimes where it is practical.

### 3.3 The broader landscape (framed, not yet committed)

- **DP-heavy steps under budget** — long-read/graph alignment DP as a tunable space/time computation.
- **Input-bound frontiers** — space-bounded joint/cohort calling and pangenome-graph analysis;
these require streaming + succinct-structure techniques, *complementary* to (not solved by) the
√t machinery. Honest open problems.

## 4. The contract design (what "declared memory" means concretely)

A single declared budget governs the whole system, not just one stage:

- **Declare → predict.** `rosalind plan --budget 4G …` reports *feasibility*, the *predicted peak
working set*, the *estimated time*, and the *curve* ("at 8 GB → ~3× faster") — so you commit with
your eyes open. No surprise OOM, no surprise multi-day run.
- **Honor or refuse — then degrade rather than refuse.** Every streaming stage exposes a provable
working-set bound and honors the ceiling; below the in-RAM threshold it slides further down the
curve (more recomputation → external-memory spill) and **still completes**, slower. A 2 GB field
device builds a human index; it does not get "won't fit."
- **Lifecycle-wide.** The same budget bounds **construction, mmap-query (tunable SA-sampling),
alignment, pileup, calling, and sort.** State your RAM once.
- **Verifiable.** The run receipt records the *realized* peak working set (content-addressed,
alongside tool version + input/param/output hashes); CI enforces the envelope on growing inputs;
`rosalind verify` re-checks a receipt without re-running.
- **Deterministic regardless of the point chosen.** The same inputs + budget produce byte-identical
artifacts; parallelism is thread-count-invariant. The space/time point never changes the *output*.

## 5. Roadmap (B → E), re-derived around the thesis

Prioritized for the builders who need the unique capability first; performance/breadth follow. The
practical pillars (B, C) stand alone — a great engine even if the research bet (D) runs long — and
D lands the space-complexity headline on top.

- **B — Genome-scale kernel (finish).** Zero-copy persisted lean multi-contig index (build-once →
mmap), wired consumers, whole-genome pileup, pipe-native. Lay the `MemoryBudget` / space-accounting
hooks. The theory layer (`algebra`/`ledger`/`machine`/`tree`/`space`) is **kept and repurposed**,
not feature-gated away.
- **C — Memory as a verifiable contract.** Declared `MemoryBudget` threaded through every streaming
stage + mmap-query; `rosalind plan` (envelope + curve); honor-or-refuse + graceful degradation;
real RSS gates in CI; deterministic (thread-count-invariant) execution + receipts + `rosalind
verify`. → "declare your RAM → predictable, honored, verifiable, end-to-end" for
align/call/query/sort.
- **D — Sublinear-space index construction (beachhead research contribution).** The √t-family knob:
build the index under the declared budget across the full curve (in-RAM-fast → checkpointed
~√n → external-memory spill), with the time overhead characterized; makes the theory
layer load-bearing; erases the O(reference) build-RAM caveat; completes the contract for the build
step.
- **E — Reach + substrate.** Fast deterministic-parallel aligner + germline indels + richer read QC
(broad usability); the Python/tensor **pileup-feature substrate** + pluggable models (the
ML-researcher/builder surface); rigorous eval + calibration validation + the open-problems
benchmark suite.

(Deterministic-parallel *execution* is part of the contract in **C**; aligner-*algorithm* quality and
micro-optimization are **E**. The existing exact-match aligner is sufficient to exercise C and D on
real data — the point there is memory, not speed.)

## 6. What this means for you

- **Edge / field / clinical builders** — declare your device's RAM; get a checkable guarantee it
fits, a time estimate, byte-identical + auditable outputs, and a build that degrades rather than
dies.
- **Large / meta / pangenome researchers** — index references that do not fit in RAM today, on
commodity hardware.
- **ML-genomics builders** — (Phase E) a fast, deterministic, memory-bounded, Python/tensor-
accessible per-locus feature stream over a whole genome to compute on.
- **Algorithms / systems researchers** — a working testbed for space-bounded computation on a real,
memory-bound domain, and a concrete open problem (§3) with a clean first target.

## 7. References

- R. R. Williams. *Simulating Time with Square-Root Space.* 2025.
- J. Cook, I. Mertz. *Tree Evaluation in Sublinear Space* (catalytic / tree-evaluation lineage). 2024.
- The "O(√t)" label in the source headers denotes the same ~√t bound (square-root space, up to lower-order factors).
59 changes: 40 additions & 19 deletions docs/superpowers/specs/2026-05-26-rosalind-target-architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,14 @@ hospital laptops, portable sequencers (MinION in the field), classrooms, and res
want to *build on* a genomics engine rather than shell out to one. Variant calling is the first
consumer of the kernel, not the whole product.

**Sharpened core bet (2026-05-27).** The differentiator is made precise: **memory is a declared,
predictable, never-refusing, verifiable contract across the whole lifecycle**, with the
square-root-space machinery (Williams 2025 / Cook–Mertz 2024; bound **~√t**, up to lower-order factors) as the
**continuous space/time knob** a declared RAM budget selects — beachheaded on **sublinear-space
index construction**, the one stage where the framework genuinely bites. The research thesis, the
open problem (time-vs-space), and the intermediate-state-vs-input lens are in
[`docs/OPEN_PROBLEMS.md`](../../OPEN_PROBLEMS.md).

Non-goals (on purpose, so the promises are airtight): out-accuracy-ing GATK/DeepVariant on
benchmarks; cloud/cluster-scale orchestration; supporting every exotic format.

Expand Down Expand Up @@ -162,33 +170,46 @@ we RSS-gate (promise #2), expose to Python (promise #5/embeddable), and let plug

- **Rebuild approach: evolve toward the target** — rewrite the wrong layers; re-home/extend the
correct tested kernels; each phase lands green; no throwaway of working code.
- **Theory layer: separate as a feature-gated `theory/` demo**; `ff`/`ark-poly` become optional
(`theory = ["dep:ff","dep:ark-poly"]`); decoupled from the genomics core.
- **Theory layer: KEEP and repurpose as load-bearing (revised 2026-05-27).** The √t / Cook–Mertz
machinery (`algebra`/`ledger`/`machine`/`tree`/`space`) becomes the *knob* that realizes the
budget-tunable space/time curve — beachheaded on sublinear-space index construction (Phase D; see
`docs/OPEN_PROBLEMS.md`). It is **not** feature-gated away, and `ff`/`ark-poly` stay as core deps.
(Supersedes the earlier "feature-gate the `theory/` demo" decision.)
- **Guiding principle for all open design choices:** maximize *unique* value to edge builders —
the kernel/substrate, memory-as-contract, calibrated honesty, verifiable reproducibility,
embeddability (§1, §3).

---

## 9. Sequencing
## 9. Sequencing (re-derived 2026-05-27 — see `docs/OPEN_PROBLEMS.md`)

Reprioritized around the core contribution — *memory as a declared, predictable, never-refusing,
verifiable contract*, beachheaded on sublinear-space index construction — and for the builders who
need that capability first (edge / field / large-/meta-/pan-genome), ahead of generic performance.
We commit to the optimal end-state rather than shippable intermediates. (The practical pillars B and
C stand alone; D lands the space-complexity headline on top, de-risking the research bet.)

1. **Phase A — the kernel + calling vertical + receipt.** `core/` (locus, sequence, record,
budget, `PileupColumn`), the public `pileup/` engine, `call/germline` (calibrated GL model)
+ re-homed `call/somatic`, `io/vcf` (spec-valid), and a **minimal `provenance/` receipt** so
the first improved outputs are already verifiable. Detailed in
1. **Phase A — kernel + calling vertical + receipt.** ✅ Done. Detailed in
`2026-05-26-phase-a-unified-pileup-genotype-design.md`.
2. **Phase B — make it real on a genome.** Index persistence (build-once → mmap), multi-contig,
streaming/gzip/multi-record ingestion, pipe-native composition.
3. **Phase D — fast & honest, memory-as-contract.** rayon (deterministic), `.csi` region
`fetch`, **real RSS gates + `rosalind plan`** (budget envelope) + `rosalind verify` (receipt
check), honest memory docs.
4. **Phase C — scoped calling.** Germline indels, richer read QC (mate-overlap dedup,
strand-bias filter, BQ/BAQ), long-read calling refinements.
5. **Phase P — Python substrate binding.** Expose the `PileupColumn` stream + index/align/call
to Python with type stubs, pytest, and a wheel matrix (manylinux/macOS/aarch64). Sequenced
right after the engine API stabilizes (after A/B).
6. **Phase E — interleaved.** Theory separation (§8), CI hardening (clippy `-D warnings`,
bcftools validation, wheels), README-honesty pass.
2. **Phase B — genome-scale kernel.** Zero-copy persisted lean multi-contig index (build-once →
mmap), wired consumers, whole-genome pileup, pipe-native; lay the `MemoryBudget` /
space-accounting hooks. (B1 streaming readers + B2 multi-contig FM-index ✅ done; B3 zero-copy
persistence + B4 wiring remain.) Detailed in `2026-05-27-phase-b-genome-scale-design.md`.
3. **Phase C — memory as a verifiable contract.** Declared `MemoryBudget` across every streaming
stage + mmap-query; `rosalind plan` (envelope + space/time curve); honor-or-refuse + graceful
degradation; **real RSS gates** in CI; deterministic (thread-count-invariant) execution +
receipts + `rosalind verify`.
4. **Phase D — sublinear-space index construction (beachhead).** The √t-family knob: build the index
under the declared budget across the full curve (in-RAM-fast → checkpointed ~√n →
external-memory spill), time overhead characterized; theory layer load-bearing; erases the
O(reference) build-RAM caveat. The headline space-complexity contribution.
5. **Phase E — reach + substrate.** Fast deterministic-parallel aligner + germline indels + richer
read QC; the Python/tensor **pileup-feature substrate** + pluggable models; rigorous eval +
calibration validation + the open-problems benchmark. (Deterministic-parallel *execution* is part
of C's contract; aligner-*algorithm* quality + micro-optimization are E.)

The open research problem and the intermediate-state-vs-input lens that order these phases are in
[`docs/OPEN_PROBLEMS.md`](../../OPEN_PROBLEMS.md).

---

Expand Down
Loading