Columnar segment format with per-column zstd + blake3 checksum

[pbudzik]

Summary

Closes review item #6. Replaces the length-prefixed bincode-row segment format with a real columnar layout (spec §6.1) — per-column zstd compression, blake3 footer checksum, format-versioned headers, and readers that reject corrupt or truncated files.

File format

header:
  magic        b"UDBRAW1\n"  (8 bytes)
  version      u8 = 1
  row_count    u32 LE
  num_columns  u16 LE

per column (×14):
  name_len         u16 LE
  name             utf-8 bytes
  encoding         u8  (0 = Plain)
  codec            u8  (0 = None, 1 = Zstd, 2 = Lz4)
  compressed_len   u32 LE
  compressed_bytes bytes

footer:
  checksum   u64 LE  (low 8 bytes of blake3 over everything above)
  magic_end  b"UDBEND01"  (8 bytes)

Each column payload before compression is a bincode-serialized Vec<T> of the column's native type (Vec<String>, Vec<i64>, Vec<i128>, Vec<Option<String>>, Vec<SmallDimensions>, etc.). Zstd is applied per-column. Co-locating homogeneous data makes zstd's history window dramatically more effective — the new compression test packs 10k repetitive events into under 500 KB.

Extensibility

• Encoding enum reserves discriminants for Dictionary / Delta / Zigzag / RLE — these can be added without bumping the file format version, and the reader rejects unknown values with a corrupt-segment error.
• Codec enum covers None / Zstd / Lz4; the reader picks the decoder per column header.
• Adding a column is additive; old readers fail loud ("missing column").

Validation

Every open verifies start magic + version + end magic + checksum. Any deviation surfaces as InvalidData. Two integration tests verify this:

• checksum_mismatch_is_rejected — flips a byte inside the body
• missing_end_magic_is_rejected — truncates the tail

Plumbing

• RawSegmentWriter::finish() now returns (row_count, checksum).
• Flusher and compaction worker thread the checksum into SegmentMeta.checksum, populating a field that was previously hardcoded to 0.
• Reader API unchanged (read_next(), scan_by_account()), so the query executor needed no edits.

Test plan

• cargo build --all-targets clean with -D warnings
• cargo test --all-targets — 23 tests pass (6 unit + 8 durability + 4 bucketing + 5 segment_format)
• CI green on this PR

Still on the backlog

• Per-column dictionary / delta / zigzag / RLE encodings (format reserves the discriminants)
• Block-level metadata for in-segment skipping
• Bump Cargo version to 0.2.2 #9 rollup builder scheduler
• Phase D operability: explain + verify endpoints #14 event validation / rejected counts
• Parquet export for raw events (Phase D operability) #17 buffered WAL
• Admin CLI: check / rebuild-rollups / inspect-segment / verify-period / export-parquet #18 durability mode setting
• Polish bundle: file locking, sort-on-flush, RLE for kind #19 endpoint path alignment with spec
• Period lifecycle (minimal: Open ↔ Closed) #22 unused parking_lot dep
• Frozen-snapshot semantics for closed periods #23 hot-path event clones

🤖 Generated with Claude Code