P1 bundle: flusher retry, fail-closed manifest, strict SQL, full pruning, [from,to) ranges, shutdown drain

[pbudzik]

Summary

Closes the six P1 items from external review in a single PR. Each is a path to either silently-wrong query answers or data getting stranded between subsystems.

What's in

#1 — Flusher failure recovery. attempt_flush returns the unflushed events to handle_message on any pre-commit failure (segment create / write / finish). The handler re-inserts them into the memtable so the next size or age trigger retries. Without this they sat in the sealed WAL file, invisible to queries until restart. Post-commit (manifest save) failures still surface as a retry but with an empty event list — the events are durable in the still-sealed WAL and recovery replays them on next start; returning them here would risk a double-flush.

#2 — Fail-closed corrupt manifest. recovery.rs no longer silently falls back to Manifest::default() on a parse error — for a billing DB, orphaning every committed segment is the worst-case outcome. Returns InvalidData and the process refuses to start. Fresh DBs (no manifest file at all) still start cleanly.

#3 — Strict SQL parser. Rewritten end-to-end. Rejects:

• SUM(anything but quantity) (was: silently mapped to SUM(quantity))
• COUNT(col) (was: silently treated as 1)
• OR / HAVING / SELECT * / aliases / unknown tables
• Unknown FROM aliases, JOINs, WINDOW, DISTINCT, etc.

Distinguishes < from <= and > from >= — each adjusts the half-open [from_ms, to_ms) correctly. Range comparisons only on timestamp_ms/hour_start_ms. WHERE is AND-only.

#4 — Full segment pruning. collect_raw_events (and the rollup-then-tail path) now uses every relevant SegmentMeta field before opening a segment file:

• time range, half-open
• bucket = blake3(account_id) % bucket_count when an account is specified
• min_account_id / max_account_id for further account-range narrowing
• per-segment product_ids / meter_ids / model_ids intersected against query filters

#5 — Half-open [from, to) semantics. matches_plan excludes events at exactly to_ms; segment overlap is s.min < to && s.max >= from; rollup_upper is the exclusive boundary min(to, watermark). Adjacent monthly queries no longer double-count boundary events.

#6 — Shutdown drain. main.rs drains the memtable + rotates the WAL on ctrl+c before letting the background workers exit. Combined with the age-based force-flush from the rollup worker (already shipped), the memtable can no longer hold events indefinitely.

Tests

tests/p1_fixes.rs — 15 new tests, each shaped to verify the fix:

• recovery_refuses_to_start_on_corrupt_manifest
• recovery_starts_fresh_when_no_manifest_exists
• sql_tests::rejects_or_in_where
• sql_tests::rejects_sum_on_non_quantity
• sql_tests::rejects_count_with_column
• sql_tests::rejects_select_star
• sql_tests::rejects_unknown_table
• sql_tests::rejects_alias
• sql_tests::distinguishes_lt_from_lte
• sql_tests::distinguishes_gt_from_gte
• sql_tests::accepts_canonical_form
• adjacent_queries_dont_double_count_boundary_event
• account_query_results_are_correct_with_bucket_pruning
• product_filter_prunes_segments_without_that_product
• build_segment_meta_unchanged_by_failure_refactor

Test plan

• cargo build --all-targets clean with -D warnings
• cargo test --all-targets — 59 tests pass (was 44; +15)
• CI green

Still on the backlog

• Per-column dictionary / delta / RLE encodings (format reserves discriminants)
• Block-level metadata for in-segment skipping
• Rollup segment columnar format
• DurabilityMode::Balanced (group commit)
• proptest property tests for §19 invariants

After this lands the codebase has no known correctness gaps from either review. Remaining work is purely performance + testing infrastructure.

🤖 Generated with Claude Code