Log-store: narrow per-segment key pruning so reads don't need the full cap

[rjpower]

🤖 Filed from investigation of the 2026-04-16 log-store outage (see `/tmp/iris-outage/log_store_audit.md`).

Context

`DuckDBLogStore` segments hold logs for every user on the cluster. `_LocalSegment.min_key` / `max_key` are the Parquet row-group min/max of the `key` column, which currently spans roughly `/alice/…` → `/zoe/…` for every file. As a result, `_segment_overlaps_key` and `_segment_overlaps_prefix` rarely narrow anything — almost every segment is included in almost every read, regardless of whose logs the user asked for.

That forces the per-read working-set cap (`_MAX_PARQUETS_PER_READ` / `_MAX_PARQUET_BYTES_PER_READ`) to do the narrowing, which is what produced the outage: keys whose rows happened to live outside the newest N segments were invisible.

#4820 (this change) lifts the cap to 25 files / 2.5 GB and adds a newest-first early-stop loop. Benchmark results at 4.5 GB / 46 segments: p95 now sits at ~280-500 ms for realistic tail queries and drops to ~30 ms when matches cluster in the newest segment. That's a workable ceiling, not a permanent answer — the underlying issue is that segment-level stats don't actually prune.

Proposal: make `min_key` / `max_key` tight enough to prune

Options, roughly in order of implementation cost:

1. Sort each flushed parquet by `(user_prefix, key, seq)`. Segments already sort by `(key, seq)` at flush time (`_flush_sealed_buffer` line 653). The row-group min/max on `key` is still wide because a single file spans many users. If we instead wrote one parquet per user-prefix bucket — or at least partitioned row groups by user inside the file — row-group pruning on `key` would eliminate most segments for any exact-key or prefix query.

2. Per-user sub-segments within one file. Keep the single-file-per-flush layout but control row-group boundaries so each row group covers a contiguous user range. Parquet predicate pushdown already uses per-row-group min/max; this would cut segments scanned without changing the on-disk file count.

3. Maintain an auxiliary per-segment bloom/roaring index on `user_prefix`. Cheap in-memory side structure keyed by segment path. `_segment_overlaps_key` / `_overlaps_prefix` consult it instead of `min_key`/`max_key`. Update on flush/consolidation. Avoids changing the parquet layout.

4. Route writes to per-user segments from the start. Largest change — maintain separate rolling segments per user (or per bucket). Cleanest pruning semantics, biggest refactor; would also change GC, consolidation, and GCS layout.

Option 1 or 2 seems like the right first step — both preserve the single-store design and make segment-level pruning actually do its job, at which point the `_MAX_PARQUETS_PER_READ` cap becomes a safety net rather than load-bearing.

Acceptance criteria

• `_segment_overlaps_key` rejects ≥90% of segments for a realistic exact-key query on the production corpus (~45 files).
• Tail p95 for `get_logs(exact_key, max_lines=1000)` stays <150 ms when no matches are in the newest segment but exist in older ones (currently ~300-700 ms with the cap-lifted early-stop path).
• No regression on mixed-write throughput or GCS offload volume.

References

• `lib/iris/src/iris/cluster/log_store/duckdb_store.py` — `_segment_overlaps_key` (lines 190-194), `_flush_sealed_buffer` sort (lines 653, 666), `_cap_segments` (1011-1027).
• Outage audit: `/tmp/iris-outage/log_store_audit.md` (local).
• Benchmark: `/tmp/iris-outage/bench.py` (local).

[claude]

🤖 Plan:

• Fetch issue
• Read relevant code in duckdb_store.py
• Duplicate-work preflight (no open PR for this issue)
• Post Research + Proposed Approach comment
• Await human review before implementation (scope = design work, not <100-line bug fix)

Research

Segments are already sorted by (key, seq) at flush (duckdb_store.py#L653, L666), so row-group min/max on key inside each file are already tight. The problem is that _LocalSegment only keeps the aggregate bounds — _read_key_bounds reduces every row group to a single (min_key, max_key) pair (L168-L187), and the flush/consolidate paths store just key_col[0]/key_col[-1] (L689-L690, L752-L753). That collapses to /alice/... → /zoe/... on any multi-user file, so _segment_overlaps_key / _segment_overlaps_prefix (L190-L217) almost never prune. DuckDB still does per-row-group skipping inside a file, but the Python-side filter decides which files to open — and it currently opens all of them, forcing _cap_segments to be load-bearing.

Relevant code

• duckdb_store.py#L168-L187 — _read_key_bounds collapses to one pair
• duckdb_store.py#L190-L217 — segment overlap checks use only seg.min_key/max_key
• duckdb_store.py#L280-L289 — _LocalSegment dataclass
• duckdb_store.py#L653 / L666 — flush/consolidate sort by (key, seq)
• duckdb_store.py#L1007-L1023 — _cap_segments, the load-bearing safety net

Proposed Approach

A variant of the issue's Option 2, but with no changes to the on-disk layout: keep per-row-group key bounds in _LocalSegment and check all of them in _segment_overlaps_*.

Concretely:

1. Replace min_key/max_key on _LocalSegment with key_row_groups: tuple[tuple[str, str], ...] (or add alongside and drop the scalars — no back-compat needed per AGENTS.md).
2. _read_key_bounds returns the per-row-group list (trivial loop change).
3. _segment_overlaps_key(seg, key) returns True iff any rg_min <= key <= rg_max.
4. _segment_overlaps_prefix(seg, prefix) returns True iff any row group overlaps [prefix, prefix_upper_bound).
5. Populate from pq.read_metadata in the recovery path (L450-462); on flush/consolidate, compute per row group from the sorted combined/new_table by slicing key_col at _ROW_GROUP_SIZE boundaries (avoids re-reading the file we just wrote).

Why this and not the issue's Options 1/3/4:

• Option 1 (one parquet per user bucket) and Option 4 (per-user segments) balloon the file count and touch GC/consolidation/GCS layout — large refactor, same pruning win.
• Option 3 (auxiliary bloom/roaring index on user_prefix) duplicates information already encoded in the existing sort; extra side structure to maintain and persist across restarts.
• Per-row-group bounds are already written into the parquet metadata by DuckDB's row-group stats — we're just caching them in Python so file-level pruning has the same fidelity as DuckDB's intra-file row-group pruning.

Expected effect vs. acceptance criteria:

• A 100MB segment at _ROW_GROUP_SIZE = 16_384 rows × ~300B/row = ~300 row groups per file. Each row group's key range is very narrow post-sort, so _segment_overlaps_key on an exact key should reject any segment whose entire key range doesn't include the target — ≥90% rejection on the stated corpus is plausible.
• _cap_segments becomes a true safety net because the filter, not the cap, does the narrowing.

Scope estimate: ~60-90 lines of src + ~40-60 lines of tests (parametrized over exact-key and prefix queries with synthetic multi-user segments). Close to the 100-line threshold but self-contained to duckdb_store.py.

Open questions for reviewer before I implement:

1. OK to drop the scalar min_key/max_key fields (no back-compat shim), or keep them as convenience aggregates?
2. Per-row-group bounds for ~300 row groups × ~45 segments = ~13.5k string pairs held in memory. Fine, or should this also cap to a coarser N buckets per segment (e.g. every Kth row group)?
3. Is the benchmark at /tmp/iris-outage/bench.py runnable as a regression check, or should I reproduce the perf claim with a new synthetic bench in lib/iris/scripts/?

[rjpower]

@claude do we sort keys at flush time? E.g. if we sort over 100mb we should have much tighter key bounds than over 1mb.

The normal log query is a prefix scan for a given key + a like query on the body. Is this effectively turned into a duckdb query? Ddb should be able to efficiently ignore most row groups in that setup.

Should we split the ddb contexts between memory (reset on each query) and disk (reset on flush)? Presumably ddb could amortize the row group analysis in that case.

[claude]

Claude finished @rjpower's task in 1m 55s —— View job

---

Answers

1. Do we sort keys at flush time? Would a 100MB segment have tighter bounds than a 1MB one?

Yes — both the new-segment write (duckdb_store.py#L663) and the consolidate-with-latest path (L676) sort by (key, seq) before pq.write_table(..., row_group_size=_ROW_GROUP_SIZE, write_page_index=True) at L690-L691 / L747-L748. So inside any one parquet, the _ROW_GROUP_SIZE = 16_384-row groups have very tight key bounds.

But the relationship to file size is the opposite of the intuition: a bigger file's aggregate bounds are wider, not tighter. A 100MB segment is more likely to span all users (/alice/... → /zoe/...) than a 1MB one. Larger files improve intra-file row-group selectivity (more groups, narrower per-group ranges) but make the file-level scalar bounds we currently store useless. Reducing segment size to ~1MB to tighten the aggregate would explode file count and footer-read overhead — wrong direction.

The structural fix is to stop summarizing to one (min_key, max_key) per file. Either:

• Cache per-row-group bounds in _LocalSegment (my Option 2 / proposed approach above) — same on-disk layout, file-level pruning gets the same fidelity as DuckDB's intra-file pruning.
• Bucket on flush by user-prefix so the file-level aggregate is itself narrow (issue's Option 1) — bigger refactor, touches GC/consolidation/GCS layout.

2. Is the prefix-scan + body-LIKE turned into a DuckDB query? Does DDB skip row groups?

Yes, both halves are pushed into DuckDB and yes, it skips row groups — but only inside files we choose to open.

• Exact key: WHERE key = $key AND seq > $cursor (L535-L537).
• Pure prefix: rewritten to key >= $prefix_lo AND key < $prefix_hi (L1093-L1098) so parquet stats can prune (regex isn't pushed through parameterized queries, see L122-L135).
• Body LIKE: contains(data, $substring) (L1129). Pushed down, but body has no useful row-group min/max so it's a real scan.

Combined with write_page_index=True and the (key, seq) sort, DuckDB does efficiently skip row groups within an opened file. The bottleneck is upstream: _segment_overlaps_* (L200-L227) consults only the file-level scalar (min_key, max_key), which is wide for the reason in (1), so almost every file is opened. Each opened file still costs a footer read + planning over all its row groups, even when zero rows match — at ~45 files that's ~45 footer reads per query, not free.

So fixing (1) — narrow per-row-group bounds at the Python filter — and (2) work together: it stops us from opening files DuckDB would only fully prune inside.

3. Should we split DuckDB contexts between memory (per-query) and disk (per-flush) so DDB can amortize row-group analysis?

Worth pursuing, but the framing should be caching parquet metadata across queries, not splitting connections — they're the same connection today (_ConnectionPool at L375-L411) and that's already the right shape: one duckdb.connect(...) shares the buffer pool, RAM tables register/unregister per cursor (L399-L408), parquet files appear inline in each query's SQL (L1146).

What's missing is metadata caching. Two cheap things that would help amortize:

• SET enable_object_cache=true on the connection (L386-L387). Default is off; turning it on caches parquet footers/row-group stats across queries. Every query re-reads ~45 footers today; with this on, we read each footer once until the file changes. Truly orthogonal to the per-row-group fix and probably the highest-ROI one-liner here.
• Stable view over the disk segment set, refreshed on flush/GC. CREATE OR REPLACE VIEW disk_logs AS SELECT * FROM read_parquet([...]) after each segment-list mutation. With per-row-group bounds (1) plus enable_object_cache (above), DuckDB's own predicate pushdown could carry essentially all the file-level pruning, and _segment_overlaps_* + _cap_segments could shrink to a real safety net.

The "split contexts" reading I'd push back on: you don't want a separate disk connection because RAM and disk need to UNION ALL in one query (see the tail early-stop at L1004-L1035 — RAM seqs are strictly newer, but the same cursor walks both). Keep the single connection, give it metadata caching, and refresh a stable view on segment changes — that gets the amortization you're after without losing the unified read path.

Suggested ordering

1. SET enable_object_cache=true — one line, immediate win, no design risk.
2. Per-row-group (min_key, max_key) cached on _LocalSegment — what I proposed above; ≥90% file rejection on exact-key tail queries.
3. Optional: stable disk view + drop file-level filter to DuckDB. Defer until (1)+(2) are measured; may not be needed.

If you want me to proceed, I'd start with (1)+(2) in one PR and bench against /tmp/iris-outage/bench.py. Let me know.
· claude/issue-4833-20260417-0130