Polish bundle: validation, BufWriter WAL, durability mode, spec-aligned routes

[pbudzik]

Summary

Knocks out the long tail of small review items in a single PR. After this, the codebase is in a clean state where the only remaining moves are performance work (per-column encodings, block metadata) or testing infrastructure (proptest invariants).

What's in

• Phase D operability: explain + verify endpoints #14 Event validation. handle_ingest runs each event through validate_event before WAL append: rejects empty IDs, non-positive timestamp, >16 dimensions (spec §21), Correction/Retraction without correction_ref. Rejected events never reach the WAL or dedupe. IngestBatchResponse.rejected is finally a real count. 5 unit tests cover the rules.
• Parquet export for raw events (Phase D operability) #17 Buffered WAL. Wal.file is now BufWriter<File>; writeln! coalesces in userspace, sync() flushes then fsyncs, rotate() flushes + fsyncs the sealed file before swapping.
• Admin CLI: check / rebuild-rollups / inspect-segment / verify-period / export-parquet #18 Durability mode. DurabilityMode::{Strict, Fast} in Config. Strict = flush + fsync (default, billing-safe). Fast = flush only (page cache; for at-least-once upstream retry pipelines). Balanced (group commit) is documented as future work.
• Polish bundle: file locking, sort-on-flush, RLE for kind #19 Spec-aligned routes.
  • POST /v1/usage/batch (was /v1/ingest) — spec §9.1
  • GET /v1/accounts/{account_id}/usage?from&to&group_by&product_id&meter_id&model_id&source — spec §12.2
  • GET /v1/accounts/{account_id}/usage/events?from&to&meter_id&product_id — spec §12.3 raw audit
  • POST /v1/query/json and /v1/query/sql retained for flexible callers
• Bump Cargo version to 0.4.0 #20 ingested_at_ms consistency. Ingest handler stamps event.ingested_at_ms = now_ms() server-side before hashing and WAL persistence — a client with a wrong clock can't poison the dedupe TTL. Replay still uses the persisted server-stamped value, so live and replayed entries are consistent.
• Persistent dedupe sidecar: per-segment .idx for fast recovery (closes last Phase A item) #21 evict_expired simplification. The dead defensive branch ("only remove if entry hasn't been replaced") was unreachable since cache entries are never updated.
• Period lifecycle (minimal: Open ↔ Closed) #22 Drop unused parking_lot dependency.
• Frozen-snapshot semantics for closed periods #23 Zero clones on the accepted ingest path. wal.append_batch takes IntoIterator<Item = &UsageEvent>, so the handler streams refs from new_events.iter() without building an intermediate Vec of clones. The memtable still owns its events (move, not clone).
• Bump Cargo version to 0.3.0 #16 Manifest fsync portability. Switch from fs::write + reopened-RO fsync (Linux-only contract) to the standard atomic-write recipe: open + write + sync_all on a single fd, drop, rename, parent dir fsync.

Test plan

• cargo build --all-targets clean with -D warnings
• cargo test --all-targets — 37 tests pass (was 32; +5 validation unit tests)
• CI green

Still on the backlog

• Per-column dictionary / delta / RLE encodings (the format reserves discriminants)
• Block-level metadata for in-segment skipping
• Rollup segment columnar format
• DurabilityMode::Balanced (group commit)
• proptest property tests for spec §19 invariants

🤖 Generated with Claude Code