Merge pull request #5 from peterrrock2/1.0.0

1.0.0

Author: peterrrock2

.gitattributes

@@ -1 +1,2 @@
 example/100k_CO_chain.jsonl.xben filter=lfs diff=lfs merge=lfs -text
+example/50k_CO_chain.xben filter=lfs diff=lfs merge=lfs -text

.github/workflows/ci.yml

@@ -1,10 +1,12 @@
 name: CI
 
-# Lightweight quality gates on every PR: formatting and lints for both languages. The heavier
-# gates (full test suite, big-endian emulation) live in full-tests.yml and run on demand — either
-# from the Actions tab or via a `/ci-full` / `/ci-endian` PR comment.
+# Quality gates on every PR: formatting, lints, and the fast test suites for both languages. The
+# heavier gates (the `#[ignore]`-gated stress suite, big-endian emulation, fuzzing) live in
+# full-tests.yml and run on demand: from the Actions tab or via a `/ci-full` / `/ci-endian` /
+# `/ci-fuzz` PR comment.
 #
-# These mirror `task format` / `task lint`; keep the two in sync.
+# These mirror `task format` / `task lint` / `task test-rust-fast` / `task test-python`; keep them in
+# sync.
 
 on:
   pull_request:
@@ -45,3 +47,31 @@ jobs:
       - name: ruff check
         working-directory: ben-py
         run: uvx ruff check .
+
+  rust-test:
+    name: rust tests (fast)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      - name: cargo test
+        run: cargo test
+
+  python-test:
+    name: python tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      - uses: astral-sh/setup-uv@v5
+      - name: sync environment
+        working-directory: ben-py
+        run: uv sync --all-groups
+      - name: build extension
+        working-directory: ben-py
+        run: uv run maturin develop
+      - name: pytest
+        working-directory: ben-py
+        run: uv run pytest tests/

.gitignore

@@ -7,4 +7,5 @@ dev_files
 demo
 
 __pycache__
-*.so
+*.so
+/ben-py/docs/code-theme-preview.md

.readthedocs.yaml

@@ -9,10 +9,13 @@ build:
 python:
   install:
     - method: pip
-      path: ./pyben
+      path: ./ben-py
       extra_requirements:
         - docs
 
 sphinx:
   builder: dirhtml
-  configuration: pyben/docs/conf.py
+  configuration: ben-py/docs/conf.py
+  # Notebook execution stays off here (NB_EXECUTION_MODE defaults to "off"), so the
+  # hosted build renders the committed notebook outputs. CI executes the notebooks.
+  fail_on_warning: true

CONTEXT.md

@@ -0,0 +1,144 @@
+# Context
+
+Orientation for anyone (human or agent) landing in the `binary-ensemble` workspace. It explains what
+the project is, how the code is shaped, and the invariants that aren't obvious from any single file.
+
+## What this project is
+
+`binary-ensemble` compresses **ensembles of districting plans**. A redistricting sampler (MCMC
+ReCom, SMC, etc.) emits thousands to millions of plans as canonicalized JSONL: one
+`{"assignment": [...], "sample": n}` line per draw. Those files are enormous and highly redundant.
+This workspace turns them into compact binary formats and provides the tooling to encode, decode,
+inspect, relabel, and bundle them.
+
+It is the spiritual successor to [PCompress](https://github.com/mggg/pcompress) and interoperates
+with it.
+
+The formats, in increasing capability:
+
+- **`.ben`**: a banner plus bit-packed, run-length-encoded frames. One frame per sample.
+- **`.xben`**: a BEN stream's payload wrapped in LZMA2 for maximum size reduction.
+- **`.bendl`**: a self-describing _bundle_ holding a header, optional assets (dual graph, metadata,
+  node-permutation map), an embedded BEN/XBEN assignment stream, and a trailing directory. Feels
+  like one file; supports interrupted writes and post-finalize appends.
+
+## Domain model (in brief)
+
+`docs/glossary.md` is the **source of truth** for terminology and is worth reading before making
+changes. The essentials:
+
+- **Plan**: a partition of dual-graph nodes into districts (the mathematical object).
+  **Assignment**: its vector encoding, `Vec<u16>` where index _i_ is the district id of node _i_.
+  One plan has many assignments.
+- **Sample**: `(sample_number, assignment)`. **Ensemble**: an ordered stream of samples from one
+  sampler run; the thing every format wraps.
+- **Variant**: `Standard` | `MkvChain` | `TwoDelta`. Fixed per stream by its banner. `Standard`
+  stores each sample independently; `MkvChain` collapses repeated consecutive samples with a count;
+  `TwoDelta` delta-encodes single-ReCom-step transitions. Variant fitness depends on the sampler;
+  see the glossary.
+- **Dual graph**: the geographic adjacency graph that gives a node ordering meaning.
+  Relabeling/reordering operations are defined against it.
+
+The glossary also nails down deliberately-disambiguated words ("header", "extract", "payload",
+"canonical\*") and the relabeling taxonomy. Honor those distinctions in code and prose.
+
+## Architecture
+
+A Cargo workspace with two members:
+
+- **`ben/`**: package `binary-ensemble`, library `binary_ensemble`, plus two thin CLI binaries.
+- **`ben-py/`**: PyO3 bindings (cdylib) published as the `binary_ensemble` Python package. Depends
+  on `ben/` by path; the core library has no Python dependency.
+
+### CLI binaries (`ben/src/bin/*.rs`)
+
+Each is a one-line wrapper over `cli::<tool>::run()`. `ben` is a subcommand tree; `bendl` owns the
+bundle container role:
+
+| Binary | Role | Does | | ------- | -------- |
+------------------------------------------------------------------------- | | `ben` | codec |
+encode/decode BEN/XBEN + xz; relabel/canonicalize/reencode; pcompress bridge | | `bendl` | bundle |
+create / inspect / extract / append `.bendl` containers |
+
+`ben` subcommands: `encode`, `xencode`, `decode`, `xdecode`, `lookup`, `xz-compress`,
+`xz-decompress`, `relabel`, `canonicalize`, `reencode`, `sort-graph`, and `pcompress` (`from-ben` /
+`to-ben` / `to-xben`). The relabel pipeline (decode → transform → re-encode) backs
+`relabel`/`canonicalize`/`reencode`; the PCompress bridge backs `pcompress`.
+
+### Library modules (`ben/src/`)
+
+- **`codec/`**: the heart. `encode`, `decode`, `frames` (`BenEncodeFrame` / `BenDecodeFrame`), and
+  `translate` (BEN ↔ ben32 wire form). Frames keep their `raw_bytes` so they can be moved/subsampled
+  without eager unpacking.
+- **`io/`**: streaming `reader` / `writer` over buffered, generic IO, and `bundle` (the `.bendl`
+  reader/writer/verify/format machinery).
+- **`format/`**: on-disk metadata shared across streams (banners and `FormatError`).
+- **`ops/`**: the higher-level operations `relabel` (the single `relabel_ben_file` driver
+  parameterised by `RelabelOptions`) and `extract`.
+- **`json/`**: dual-graph utilities (NetworkX-adjacency IO, MLC and RCM node-ordering algorithms)
+  used by the relabel pipeline.
+- **`progress/`**, **`logging/`**, **`util/`**: spinners (`indicatif`), `tracing` setup, and small
+  shared helpers (RLE).
+
+### Data flow
+
+```mermaid
+flowchart LR
+    JSONL -->|encode| RLE
+    RLE -->|bit-pack| frame
+    frame -->|concat| ben["stream(.ben)"]
+    ben -->|LZMA2 wrap| xben[".xben"]
+    ben -->|bundle| bendl[".bendl"]
+```
+
+Decode reverses this. The relabel subcommands run decode → transform → re-encode in one streaming
+pass. The encoding stack has five named layers (bit-packing, RLE, frame, stream, container); see the
+glossary's "Encoding Stack" table.
+
+## Invariants and cross-cutting concerns
+
+These hold across the codebase and are easy to violate by accident:
+
+- **Format stability is a contract.** Committed fixtures under `ben/tests/fixtures/v<n>/` must keep
+  decoding forever within a major version. Never regenerate fixtures in place. See
+  `docs/format-stability.md`.
+- **Frames decode lazily.** Keeping `raw_bytes` without unpacking runs is what makes
+  subsample-by-skip and random-access reads fast. Don't force eager bit-unpacking on read.
+- **Integrity is checked with CRC32C.** Verifying read paths are the default; checksum-skipping
+  variants are explicitly named with an `_unverified` suffix.
+- **Terminology is disciplined.** The glossary governs identifiers and prose; when they disagree,
+  the glossary wins and the code changes.
+- **Streaming, not slurping.** Ensembles are too large to hold in memory.
+- **64-bit only** (enforced with `compile_error!` in `lib.rs`).
+- **Illegal states are unrepresentable where practical**: e.g. `XBenVariant` cannot hold `TwoDelta`,
+  so BEN32-only paths reject it at compile time.
+
+## Building and testing
+
+The workspace uses a `Taskfile.yml` (the `task` / `go-task` runner) as the single entry point for
+local workflows. CI runs the lightweight gates (formatting + lints) on every PR; the heavy gates
+(full test suites, big-endian emulation) run on demand via the Actions tab or a `/ci-full` /
+`/ci-endian` PR comment from a collaborator. The wheel-publishing workflow is separate and
+tag-triggered.
+
+- `task test`: Rust fast suite + `#[ignore]`-gated slow/stress suite + Python `pytest`.
+- `task format`: `cargo fmt --all` + `ruff format`.
+- `task lint`: `cargo clippy --workspace --all-targets` (warnings denied) + `ruff check`.
+- `task coverage-summary`: combined Rust + Python coverage.
+- `task test-endian`: full ben suite on one big-endian and one little-endian target via `cross`
+  (Docker + QEMU), proving wire-format endianness regardless of the development machine.
+  `task check-endian` is the no-Docker compile-only tier.
+- `task fuzz`: time-boxed coverage-guided fuzzing (cargo-fuzz/libFuzzer, nightly) of every read
+  surface, seeded from the committed fixtures. `FUZZ_SECONDS` bounds each target (default 60).
+
+Python development uses `uv` + `maturin` (`task ben-py-develop`).
+
+## Document map
+
+- **`docs/glossary.md`**: terminology, the source of truth.
+- **`docs/coding-standards.md`**: how code in this repo is written (errors, logging, naming,
+  testing, modules, PyO3).
+- **`docs/bendl-format-spec.md`**: the `.bendl` on-disk binary layout.
+- **`docs/format-stability.md`**: the wire-format stability policy.
+- **`README.md`**: user-facing CLI and library usage.
+- **`docs/*-plan.md`**: active design plans, written before implementation.