Cold tier: predicate pushdown inside Parquet objects + streaming scans - #12
Open
kvey wants to merge 6 commits into
Open
Cold tier: predicate pushdown inside Parquet objects + streaming scans#12kvey wants to merge 6 commits into
kvey wants to merge 6 commits into
Conversation
Cold reads now push their predicate down three levels before a row is materialized: partition pruning (existing), row-group pruning from Parquet column statistics, and a decoder-level Arrow RowFilter. Objects are written with bounded row groups (16k rows) so the statistics have pruning granularity; spans stay trace_id-sorted and logs/metrics (service|name, ts)-sorted, which is what makes the stats tight. Scans stream one time partition at a time, newest first, and a visitor can stop at a partition boundary — a limit-bounded query no longer reads history it would discard. Every scan reports ColdScanStats (objects listed/pruned/ read, row groups pruned, rows decoded), surfaced in `query traces --explain` as `cold_scan`. get_trace, query_traces/logs/metrics cold fills, and the summarize/anomalies aggregation window all use the pushed-down scans. An oracle test pins the pushdown against full-scan-plus-filter across a predicate grid. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RBdJtXn5RHXyUsmzNGv2Gh
TAEL_TENANT_ISOLATION=1 turns tenancy from query-layer authorization into physical isolation. TenantShardedStore gives every tenant its own complete TaelBackend — WAL namespace, hot tier, cold tier, text index, comments file — under <data_dir>/tenants/<tenant>/, with the tenant as the top-level shard key: writes route by the stamped tael.tenant attribute, scoped reads open only the scoped tenant's engine, unscoped (admin) reads fan out and merge via the shard-merge helpers. WAL replication composes (each tenant engine ships to the same standbys; an isolated standby re-routes by the stamp), hostile tenant names percent-encode so they cannot escape the tenants root, and existing tenant engines re-open at startup. Ingest now actually stamps the writer's tenant server-side — the tenancy doc claimed this but no ingest path did it, meaning a client could forge tael.tenant to read or write another tenant's data even under plain multi-tenancy. Both OTLP transports, Datadog intake, remote-write, and the eval span/score endpoints stamp from the authenticated principal (attached by the auth middleware / a new gRPC interceptor extension), overriding any client-supplied value. Payload-text indexing routes to the writing tenant's engine index via PayloadIndexes; the compactor drives either topology through the new EngineMaintenance trait; the shared blob store's GC unions live hashes across tenant engines. SQL is refused under isolation (cannot be scoped or merged across engines), Postgres comments + isolation is rejected at startup, and enabling isolation over a pre-isolation data dir warns that the old engine's data is not served. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RBdJtXn5RHXyUsmzNGv2Gh
TAEL_NODE_ROLE=ingest turns a node into the stateless ingest tier the
scaling design's §4 describes: it terminates OTLP on both transports,
splits each export batch at the protobuf layer by the same shard key the
query fan-out uses (hash(trace_id) for spans/logs, hash(name) for metrics),
and re-posts per-shard OTLP requests to TAEL_INGEST_SHARDS. No local
engine: blobs, text indexing, and tenant stamping happen on the owning
shard, and the client's Authorization header is forwarded verbatim so
shards authenticate the original principal. Delivery is at-least-once with
idempotent replay (content-derived keys), queries answer with an
explanation, and readiness reflects shard reachability. The OTLP transports
now serve trait objects so the storing and forwarding services are
interchangeable. An end-to-end test runs a forwarder against two real
shard servers and checks whole-trace placement plus fan-out completeness.
scripts/failover-drill.sh is the multi-process network failover drill the
roadmap called for: real processes, real TCP, SIGKILL the leader, gossip
election, zero acked-write loss — now a CI job. It immediately caught a
real bug the in-process tests could not: reqwest::blocking panics inside
the server runtime ("Cannot drop a runtime…"), so TAEL_WAL_STANDBYS
panicked at startup and WAL shipping/query-tier reads panicked per call.
RemoteStore/RemoteWalSink now drive an async client on a dedicated IO
runtime behind the same sync facade (the objstore pattern), safe from any
context.
docs/running-tael-for-a-team.md is the operations guide: the five safety
rules, topologies 0-3 (single node, leader+standby, sharded, +ingest tier)
with exact env configs, tenancy levels, object storage, an operations
checklist, and what is deliberately unsupported.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBdJtXn5RHXyUsmzNGv2Gh
TAEL_KAFKA_MODE=produce turns a node into a buffering edge: it terminates OTLP on both transports, splits each export batch by the shard key (hash(trace_id) for spans/logs, hash(name) for metrics — the same splitters the ingest tier uses), and publishes one OTLP protobuf slice per topic partition, acking the producer only after the broker acked. The writing principal's tenant travels as a record header; credentials never do. TAEL_KAFKA_MODE=consume makes a storage node drain an explicit set of partitions (TAEL_KAFKA_PARTITIONS — ownership is an operator decision, one consumer per partition, preserving the single-writer rule) and apply each record through the same OTLP services the listeners use, so blob extraction, text indexing, tenant stamping, backpressure, and ingest stats behave identically. Offsets persist in <data_dir>/kafka_offsets.json after apply: at-least-once, safe because hot-tier keys are content-derived and replay overwrites. Undecodable records are skipped loudly; transient apply failures retry without advancing. Pure-Rust rskafka keeps the optional feature free of C toolchains. Default builds refuse TAEL_KAFKA_MODE with an error naming the feature. Per the scaling doc, the broker stays out of the HA story (walrus + WAL shipping own that); this buffer is for scaling ingest and storage independently, and the docs say so. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RBdJtXn5RHXyUsmzNGv2Gh
The gap-closure plan's D3 recommendation, now pursued: the WAL (walrus) is unix-only, so on Windows the tael-server dependency is dropped via a target-gated dependency and the crate builds as a pure client — every query, eval, watch, and annotation command works against a remote server via --server/TAEL_SERVER. The four commands that need the local engine (serve, auth, config, server migrate) refuse on Windows with a pointer to running the server on Linux/macOS/WSL/Docker and managing keys on its host, instead of failing to compile or link. cargo check and clippy pass for x86_64-pc-windows-msvc. CI gains a windows-latest job building the client so unix-only code cannot sneak back into the client half, and the release workflow now ships a Windows binary (the binstall metadata already handled binary-ext). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RBdJtXn5RHXyUsmzNGv2Gh
eval report gains --group-by <label> (the last B4 sub-item): per-group case counts and score means over the run's case labels, matching tael.-prefixed forms, in JSON and the table renderer — the report-side half of the git/prompt span conventions. The planning docs now match the code. The gap-closure roadmap's residual list is empty: a third-pass section records the five architecture projects (Parquet predicate pushdown, ingest-only mode + ops doc + network failover drill, Windows client build, Kafka buffer, tenant-as-shard-key isolation) with what each actually shipped, including the two real defects the work surfaced (reqwest::blocking panicking on the replication path; ingest never stamping the writer's tenant). The scaling/HA doc's rollout items 5-7 are marked landed with their real shapes, backend-design B5 checkboxes close, the impl plan's deferred note flips to landed, and README documents the Windows client-only build, links the team operations guide, and checks off M4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RBdJtXn5RHXyUsmzNGv2Gh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cold reads now push their predicate down three levels before a row is
materialized: partition pruning (existing), row-group pruning from Parquet
column statistics, and a decoder-level Arrow RowFilter. Objects are written
with bounded row groups (16k rows) so the statistics have pruning
granularity; spans stay trace_id-sorted and logs/metrics (service|name,
ts)-sorted, which is what makes the stats tight.
Scans stream one time partition at a time, newest first, and a visitor can
stop at a partition boundary — a limit-bounded query no longer reads history
it would discard. Every scan reports ColdScanStats (objects listed/pruned/
read, row groups pruned, rows decoded), surfaced in
query traces --explainas
cold_scan.get_trace, query_traces/logs/metrics cold fills, and the summarize/anomalies
aggregation window all use the pushed-down scans. An oracle test pins the
pushdown against full-scan-plus-filter across a predicate grid.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01RBdJtXn5RHXyUsmzNGv2Gh