Phase A: manifest generations, rebuildable rollups, correction workflow

[pbudzik]

Summary

Three correctness-layer additions from the expanded roadmap. After this, items 1–4 of Phase A from the external reviewer's framing are closed (persistent dedupe index deferred to a perf-focused PR — current recovery scan is asymptotically the same as on-disk lookup at MVP scale).

Changes

Manifest generations

Replaces the single manifest.json with a generation directory:

db_root/manifest/
  CURRENT                  (text: latest valid generation u64)
  manifest-000001.json     (last 10 retained)
  manifest-000002.json
  ...

• Manifest::save bumps the generation, writes the new file via tmp+rename+fsync, atomically advances CURRENT, fsyncs the directory, and prunes old generations beyond KEEP_GENERATIONS = 10.
• Manifest::load reads CURRENT and tries that generation; on corruption it walks backwards until it finds one that parses, logs a warning, and uses it. Fails closed only if no generation is valid.
• Legacy single-file manifest.json is auto-migrated on first load: read → save as generation 1 → delete the old file. Existing DBs upgrade transparently.
• Recovery::run_startup_recovery uses Manifest::load instead of inline serde_json — happy-path behavior unchanged.

This is what the external reviewer asked for under "Strict manifest generations — Corrupt manifest must not start empty." The previous code did fail closed, but had no recovery path; now operators can roll forward through one bad write without intervention.

Rebuildable rollups

RollupWorker::rebuild_rollups(from_ms, to_ms) drops rollup segments overlapping the range, rewinds manifest.watermarks.hourly_rollup_ms to from_ms, and saves the manifest. The next worker tick refills the gap from raw events.

Use cases: rollup-builder bug fixes, late events arriving for a sealed hour, operator drift verification. Surface for a future admin CLI to call.

Correction event workflow

The infrastructure was already there:

• validate_event requires correction_ref for Correction/Retraction (shipped in P1 bundle)
• The rollup builder sums quantities uniformly, so a Correction event with negative quantity naturally subtracts from rollup SUM

What this PR adds:

• matches_plan supports kind as a filterable field
• group_key_value supports kind as a group key

So operators can isolate or break out adjustments in forensic queries. The rollup key intentionally doesn't include kind — that would split corrections into separate rollup rows and break the "net total" semantics the billing layer wants.

Tests

11 new tests in tests/phase_a.rs:

Manifest generations:

• save_creates_generation_files — CURRENT advances, files are numbered
• load_rolls_back_through_corrupt_generations — corrupt gen 3, loader uses gen 2
• load_fails_closed_when_no_generation_is_valid — every gen corrupt → InvalidData
• load_migrates_legacy_manifest — old single-file format → gen 1, old file removed
• load_returns_none_for_fresh_db — no manifest at all → Ok(None)
• old_generations_get_pruned — 12 saves → 10 files remain
• recovery_picks_up_generations — round-trip through Recovery

Rebuildable rollups:

• rebuild_rollups_drops_segments_and_rewinds_watermark — full cycle: tick → rebuild → tick → restored

Correction workflow:

• correction_event_subtracts_from_sum — 100 + (-40) → SUM = 60
• query_filter_by_kind_isolates_corrections — filter kind=Correction → only -40
• group_by_kind_splits_originals_from_corrections — Usage=100, Correction=-40

Test plan

• cargo build --all-targets clean with -D warnings
• cargo test --all-targets — 77 tests pass (was 66; +11)
• CI green

Still on the Phase A backlog

• Persistent dedupe index (per-segment Bloom of event_id_hash) — pure perf at MVP scale, deferred to Phase B where it lives alongside other segment-format work.

Next

After this lands, the natural next slice is Phase B (per-column encodings — dictionary for IDs, delta for timestamps, zigzag-varint for quantities) or DST (deterministic simulation testing).

🤖 Generated with Claude Code