Background compaction scheduler with reader-safe deferred deletion

[pbudzik]

Summary

Closes the compaction-scheduler gap. The compaction engine has existed since the durability PR but wasn't reached by any background task. This PR adds the scheduler, makes the manifest swap atomic, and implements a reader-safe deferred-deletion grace window (spec §15.3) so queries that snapshotted the manifest just before a compaction commit don't race with file unlinks.

Each tick (default 60s)

1. Sweep pending deletions. For every compacted_replacements entry older than compaction_grace_ms (default 30s), delete the old segment files and drop the record from the manifest.
2. Plan + execute. Per-bucket plans (planner triggers when a bucket has > max_small_segments segments) are merged: sort by (account, product, meter, model, ts), cold-dedupe by event_id, write a single output segment. Manifest swap is atomic — old metas dropped from raw_segments, new meta added, ReplacementRecord appended, save.

The plan is defensively aborted if the input set isn't fully present in raw_segments at commit time. Not reachable today (one worker), but it future-proofs against concurrent compaction.

Key design choices

• ReplacementRecord.committed_at_ms is the deferred-delete clock. #[serde(default)] means existing manifests read back as 0; pre-existing records (none, currently) never get finalized by the new code, matching the prior "leave alone forever" behavior.
• CompactionPlanner returns Vec<CompactionPlan> with an explicit bucket field per plan instead of an untagged Vec<Vec<String>>. The bucket flows through to build_segment_meta so the compacted output keeps its label.
• Config gains compaction_tick_interval_secs, compaction_grace_ms, compaction_max_small_segments.
• Shutdown: same Notify pattern as the rollup worker.

Test plan

• cargo build --all-targets clean with -D warnings
• cargo test --all-targets — 32 tests pass (was 27; +5 in tests/compaction.rs)
• CI green

New tests:

• Per-bucket plans trigger only when the bucket exceeds the threshold
• The compacted segment contains every input event (no loss, exact event_id dedup)
• Re-tick post-compaction is a no-op (no extra segments, no premature deletion)
• Old files survive the grace window, then disappear on the next post-grace tick; replacement record drops out of the manifest
• Independent buckets are processed in one tick

Storage layer status

With this PR, the storage path is end-to-end complete:

ingest → WAL → memtable → raw segment → rollup segment + compaction
                                                          ↓
                                                  deferred file deletion

Still on the backlog

• Phase D operability: explain + verify endpoints #14 event validation / rejected counts
• Parquet export for raw events (Phase D operability) #17 buffered WAL
• Admin CLI: check / rebuild-rollups / inspect-segment / verify-period / export-parquet #18 durability mode setting
• Polish bundle: file locking, sort-on-flush, RLE for kind #19 endpoint path alignment with spec
• Bump Cargo version to 0.4.0 #20 first_seen_ms consistency between live ingest and replay
• Persistent dedupe sidecar: per-segment .idx for fast recovery (closes last Phase A item) #21 dead defensive branch in evict_expired
• Period lifecycle (minimal: Open ↔ Closed) #22 unused parking_lot dep
• Frozen-snapshot semantics for closed periods #23 hot-path event clones
• Per-column dictionary / delta / RLE encodings
• Block-level metadata for in-segment skipping
• Rollup segment columnar format

These are smaller items suitable for a single "polish" PR.

🤖 Generated with Claude Code