WIP experimenting with compressing spills - #1343
Draft
joosthooz wants to merge 80 commits into
Draft
Conversation
Adds the foundational pieces for Phase 3 (spill-path compression): - plan_register: new spill plan map keyed by shared_data_repository* (one plan per operator output edge, discovered lazily on first spill) - convertible_data_batch: carry source_repo pointer so convert() can look up / store a spill plan for the originating query-graph edge - sirius_config: spill-compression knobs (enable flag, explorer beam_width, per-column byte cap) - compressed_disk_representation: DISK-tier idata_representation backed by a .hpln file with RAII unlink on last-owner drop - simpatico_bridge: initialize_simpatico_jit() placeholder + make_compressed_temp_path() for disk-tier spill paths - docs/compression/spill-compression-plan.md: work-item tracker Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Wires Simpatico compression into the GPU->HOST and GPU->DISK downgrade paths. The plan is discovered once per query-graph edge and reused. Plan lookup keys off the source shared_data_repository* (one plan per operator output port). cuCascade's converter signature cannot carry that key, so convertible_data_batch::convert() installs a thread-local spill_context for the duration of the convert_to<> call; the converter reads it back to resolve — or, on the first batch to spill from an edge, explore and cache — the plan. New converters: gpu_table -> compressed_host (compress, stage to pinned blob) gpu_table -> compressed_disk (compress, write .hpln) compressed_host -> compressed_disk (flush blob verbatim, no re-compress) compressed_disk -> gpu_table (read .hpln, decompress) The host->disk cascade is a straight file write: a pinned_compressed_blob is already in .hpln layout (header ++ payload), so no decompress / re-compress round trip is needed. Fallback is total: a converter that throws — no context, no plan, poor ratio, GPU error — leaves the batch untouched (convert_to only installs the result after the converter returns), so convert() falls through to the existing uncompressed spill. Settings: SET spill_compression, SET spill_compression_explore_beam_width. Off by default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Notes the thread-local spill_context decision (and why set_data() inline was rejected), and adds the reservation-oversizing follow-up: the compress converter receives a reservation sized for the uncompressed batch, so the host budget is over-charged until it can be resized. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ports the spill tests from the old compress-spills branch and extends them for the per-edge plan register and the blob-backed host tier. Roundtrips (compress -> restore, verified by column SUM): GPU -> compressed_host GPU -> compressed_disk (also asserts the .hpln file exists) compressed_host -> compressed_disk The host->disk case asserts the written file is exactly header + payload bytes, pinning the contract that the cascade is a verbatim blob flush and not a decompress/re-compress round trip. New coverage beyond the old branch: first spill from an unseen edge runs the explorer, caches the plan, leaves other edges untouched, and reuses the cached plan on the next batch. Fallbacks (each must spill uncompressed, never throw): feature disabled, batch with no source edge, plan/table column-count mismatch, and a compression ratio that misses the threshold. Note on the test DSL: the spill cases bitpack rather than "delta -> differences" — the latter emits full-width output and so is rejected by the ratio gate (test_compression.cpp already uses it as its "saves too little" fixture). It is kept here for exactly that test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
max_compressed_fraction governs both the pin and spill paths, but was named
and documented as if it were pin-only, and two wiring gaps meant the spill
path never actually saw the configured value:
- YAML config was never pushed to the converter globals. The cuCascade
converters run without a SiriusContext, so they read process globals;
only the DuckDB SET handlers pushed to them, and only on change. A
deployment configuring spill compression purely through YAML got the
built-in defaults. Worse, the YAML reader rejects unknown keys, so
`enable_spill_compression` in a config file was a hard error.
SiriusContext::initialize() now performs the push (also covering
column_threads, which previously relied on a pin_table bind to land).
- SET max_compressed_fraction did not propagate. The handler wrote the
config field but never pushed, so the spill converters kept the stale
value while pinning honoured the new one.
Renames the DuckDB setting pin_table_compression_max_compressed_fraction ->
compression_max_compressed_fraction, matching the existing
compression_column_threads precedent for a setting shared across paths. The
C++ field and YAML key were already path-neutral and are unchanged.
Also corrects the compression_config doc comment, which claimed the struct
had "no effect on spill-path compression".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ically A plan that missed max_compressed_fraction was still cached, so every later batch from that edge repeated the full compress and threw the result away. The register now records a per-edge verdict and the spill path skips the attempt outright, before entering convert_to. The verdict is not permanent. Each entry counts spill attempts, and after spill_replan_after_uses (new setting, default 128; 0 = never) it expires and the edge is explored afresh. Expiry overrides both the cached plan and an unviable verdict, so a plan whose data drifted is re-planned and an edge written off from an unrepresentative early batch gets another chance. Uses are counted once per attempt including skipped ones — otherwise a skipped edge would never age and could never be retried. plan_register's spill API grows accordingly: decide_spill_plan() returns an explore/use/skip verdict, with mark_spill_plan_unviable() and note_spill_plan_use() maintaining the state. resolve_spill_plan() now returns the full entry rather than just the DSL. Settable via SET spill_compression_replan_after_uses or the spill_replan_after_uses YAML key. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s stop paying
Re-exploring an edge costs a beam search per column, so a fixed interval keeps
paying that price even for an edge where nothing ever changes. Each entry now
carries its own replan_interval, seeded from spill_replan_after_uses and adapted
after every re-explore cycle:
reset to the configured value when the cycle produced a change that actually
compresses (new plan that works, or viability
recovered on the same plan)
doubled otherwise — same plan, or a new plan that still
misses the threshold
So a stable good plan and a stubbornly incompressible edge both stop paying for
explores they learn nothing from, while an edge that is genuinely moving stays on
the frequent schedule. A stretched interval takes precedence over the configured
one when deciding; doubling saturates rather than wrapping.
mark_spill_plan_unviable() is replaced by conclude_spill_attempt(), which records
viability and drives the backoff from one call. The converter reports through an
RAII guard so every exit path concludes exactly once — including a hard
compression failure, which previously went unrecorded and so was repeated by
every later batch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Widening the unviable verdict to cover any exception went too far. Compression runs under memory pressure, so compress_with_plan can throw rmm::out_of_memory for reasons that say nothing about the data — yet that would disable compression for the edge for a whole replan interval and, at the next re-explore, count as "still failing" and double that interval. A passing blip degraded the edge lastingly. Attempts now report one of three outcomes instead of a bool: compressed / not_worth_it measurements — applied immediately, as before failed an error, which may well be transient A failed outcome only increments a per-edge error streak, leaving viability, the replan interval and any pending replan comparison untouched, until spill_error_tolerance (new setting, default 3) consecutive failures make it durable. Any measurement resets the streak. Missing max_compressed_fraction is still a measurement and still applies on the first occurrence — it is real evidence about the data, unlike an allocation failure. A tolerance of 1 restores the previous write-off-on-first-error behaviour. Settable via SET spill_compression_error_tolerance or the spill_error_tolerance YAML key. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Compressibility is a property of a column, not of a batch: a wide operator output routinely mixes columns that shrink 10x with ones that do not compress at all. Tracking one plan and one verdict per edge got both cases wrong — a single incompressible column could disable compression for all of them, while an aggregate that passed kept paying to compress a column that never shrank. The explorer already works one column at a time, so its per-column results are now stored per column instead of being flattened into a "---"-joined plan. Compression runs column by column via simpatico::compress_column (the same loop compress_with_plan performs internally), assembling the compressed_table directly, and each column is measured against its own original bytes. A column that does not pay is stored with a passthrough plan (input -> identity) on later batches rather than being re-compressed and discarded every time. identity is safe for every dtype: on STRING it decomposes via str_split and round-trips through both the in-memory and the file path. The skip verdict now means no column is viable — a partially viable edge still compresses the columns that pay. The replan schedule stays per edge, since batches arrive per edge and all its columns are re-explored together; its backoff comparison generalises to "plans changed, or the viable-column count changed, and at least one column compresses". A cached entry whose column count does not match the batch describes a different schema, so it is now discarded and the edge explored afresh rather than applied blindly (which threw inside Simpatico and fell back uncompressed). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ently The explorer is a beam search over a large space and readily returns a differently spelled plan that performs identically. Comparing plans by DSL text meant those counted as a change, which reset the replan backoff and locked the edge into re-exploring for the rest of the query — the exact churn the backoff exists to prevent. Plans now carry the measurements the explorer already reports (compression ratio, compress and decompress throughput), and a candidate is adopted only when one of them differs from the cached plan's by more than spill_replan_change_threshold (new setting, default 0.20). The comparison is relative to the larger of the two values, so it stays symmetric and reads a previously unmeasured (zero) value as a full change. Adoption is decided per column. An adopted column resets to viable with a clear error streak; a column that keeps its cached plan keeps its verdict too, since an equivalent plan will not compress any better than the one already judged — so a written-off column is not resurrected by a cosmetic re-explore. Only genuinely adopted columns mark the entry as changed, so an all-equivalent re-explore now backs off instead of resetting. Settable via SET spill_compression_replan_change_threshold or the spill_replan_change_threshold YAML key; 0 adopts every re-explored plan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…regression
Adds the A/B configs and driver, and records the first measurements in the plan
document.
TPC-H q21 at SF100 on a 12 GB card (usage_limit_fraction 0.5), 2 iterations:
14.7s with spill compression off, 68.3s with it on. nsys attributes 18.07s of
the 23.45s query to explore_spill_plan — 77% of query time, and 99.998% of the
compress path, since actual compression accounts for 0.35ms of the 18.0685s
spent in the converter.
Two problems surfaced, both recorded in the plan doc:
- The beam search allocates GPU memory during a downgrade, i.e. exactly when
the GPU is out of memory: 2,430 bad_alloc exceptions from trial encodes, and
only 1 of 42 spill attempts produced a compressed batch.
- A failed exploration is not memoized. resolve_or_explore_spill_plan throws
before set_spill_plan creates the register entry, so conclude_spill_attempt
finds nothing to record against and the outcome_guard is never constructed.
No error streak accumulates, the edge is never written off, and every later
spill re-runs the full beam search — 41 times in this one query. The
existing memoization covers a failing compression but not a failing
exploration, which is the more expensive case by far.
The driver is standalone rather than using performance_test.py, which calls
drop_os_cache() unconditionally and so requires passwordless sudo.
spill_compression remains off by default.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two fixes for the problems the SF100 benchmark surfaced, taking TPC-H q21 from 4.6x slower with spill compression to 2.3x, with compression actually working (18 compressed spills vs 1, and zero allocation failures vs 2,430). 1. A failed exploration was not recorded anywhere. resolve_or_explore_spill_plan throws before set_spill_plan creates the register entry, so conclude_spill_attempt found nothing to record against and the outcome_guard was never constructed — no streak accumulated, the edge was never written off, and every later spill re-ran the whole beam search (41 times in one query). spill_plan_state now carries an explore-failure streak that note_spill_explore_failure() creates the entry for when absent; once it reaches spill_error_tolerance the edge reports `skip` until its entry expires on the normal replan schedule. 2. The explorer allocated unboundedly at exactly the wrong moment. It runs during a downgrade — i.e. when the GPU is out of memory — and threw bad_alloc on most trial encodes. spill_explore_sample_rows (new setting, default 65536) trims the beam ranking. Note this alone was not enough, and is worth remembering: sampling made exploration *slower* per call (1.29s -> 2.13s) because sample_rows only trims the ranking phase, while finalists are still re-measured on the full column. The OOMs had been acting as an accidental cost limiter. The benchmark config therefore also lowers max_explore_bytes to 8 MiB, which bounds both phases and is what removed the failures outright. What remains is the cost of running a beam search inside a query at all — nsys still puts ~81% of query time in explore_spill_plan. Tuning cannot remove that; seeding plans from column lineage can, and is the next step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Groundwork for seeding spill-compression plans from the plans already explored
offline per table, instead of running a beam search inside the query — which the
SF100 benchmark showed costs ~81% of query time.
DuckDB's ColumnBindings already form the lineage graph, so column_origin_resolver
walks the plan once: it seeds at each LogicalGet with the real table and column,
and propagates through the operators that introduce their own table_index
(projections keep an origin for a bare column reference, aggregates for their
group keys). Filter, join, order and partition re-expose their children's
bindings unchanged, so they need no handling. Results are carried on
sirius_physical_operator alongside `types` and reported per repo at wiring time.
Two ordering constraints worth recording:
- The walk must run BEFORE ColumnBindingResolver, which rewrites
BoundColumnRefExpression into positional BoundReferenceExpression and so
erases the bindings it follows.
- Operators inserted by the later rewrites (GPU pipeline wrappers, partitions,
merges) never pass through the create_plan dispatcher and start with no
lineage. propagate_column_origins() lets a pass-through operator inherit its
child's origins, gated on matching output arity.
Not yet effective for parquet scans: LogicalGet::GetTable() returns null for
read_parquet (a table function, not a catalog table), so nothing seeds. Measured
on q21 at SF100: 47 columns reach the spilling edges, 0 resolved. Deriving the
table identity from the resolved file paths — the convention the pin plans are
already keyed by — is the next step, and is written up in the plan doc.
No behaviour change: nothing consumes the origins yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
read_parquet is a table function with no catalog entry, so LogicalGet::GetTable() returns null and nothing seeded the lineage walk. The table identity now comes from the directory holding the scan's files (<root>/lineitem/part.0.parquet -> lineitem), which is both the layout pin_table(name=...) is given and the key the pin plans are stored under. Deliberately the parent directory rather than the file stem: in these datasets every file is named part.N.parquet, so the stem would resolve every table to "part" — which is itself a TPC-H table, so it would silently attach the wrong plans rather than fail. This works: q21 at SF100 now resolves l_orderkey and l_suppkey to lineitem.0 and lineitem.2. But overall coverage is still 2 of 47 columns there, and 0 of 3 on a plain lineitem/orders join, so it is not yet usable. Two distinct failures remain downstream — edges whose source operator has an empty origins vector (pipeline sinks that propagate_column_origins is not reaching), and edges that carry a vector but resolve every entry to nullopt (bindings recorded pre- ColumnBindingResolver not matching those read back post-resolution). Both are written up in the plan doc. Still no behaviour change: nothing consumes the origins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Takes lineage coverage from 2 of 47 columns to 59 of 64 on TPC-H q21, and to 100% on a plain two-table join. Two bugs, neither where the symptom pointed. The GPU scan dropped the lineage. insert_gpu_pipeline_operators replaces the table scan with a sirius_gpu_scan_operator built from scratch, which did not copy column_origins. The scan is a leaf, so propagate_column_origins() had nothing to inherit from and the loss at the leaf silently emptied the whole plan — it presented as propagation failing when it was one missing assignment at the point where every origin enters the tree. Expressions arrive already binding-resolved. The plan Sirius receives has been through ColumnBindingResolver upstream, so a pass-through projection column is a positional BOUND_REF into the child's output rather than a named BOUND_COLUMN_REF; resolve_expression only handled the latter and so recorded nothing for projections and aggregate groups. Running our walk earlier in create_plan does not help — that assumption was wrong. It now handles both forms, mapping a position back through the child's bindings. LogicalGet had masked the second bug: it reads column_ids directly and never consults an expression, so scans resolved correctly while everything above them missed, which is why the two q21 hits looked like partial success rather than a systematic failure. The residual ~8% on q21 are computed columns — aggregate results and expressions — which have no single base column by definition. Still no behaviour change: nothing consumes the origins yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Columns that trace back to a base table now take that table's offline plan instead of running a beam search mid-query. Exploration on the spill path drops to zero on TPC-H q21 at SF100 — previously 81% of query time. plan_register gains per-repo column origins (recorded at plan-wiring time) and seed_plans_from_lineage(). Columns with no lineage, or whose table has no plan, are stored raw until the edge's next scheduled replan. Two fixes were needed for the plans to be reachable: input_plan_dir was read lazily inside pin_table()'s bind (so a query that never pinned never loaded a plan), and that load was gated on enable_pin_table_compression. Plans now load once at SiriusContext::initialize(), gated only on input_plan_dir. Wall clock barely moves (36.4s -> 31.3s, ~2.4x) because the cost relocated: all 20 declines are now std::bad_alloc from compress_column itself. That isolates the real problem — compressing on the spill path needs GPU memory at exactly the moment the GPU has none. Options in the plan doc. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d to q21 Re-runs the SF100 comparison against the real working config from ~/.sirius/sirius.yaml instead of the one I invented. The earlier 2.3-4.6x numbers were substantially an artefact of that config: usage_limit_fraction 0.5 (half the device unused), no disk tier, and Sirius's default downgrade_trigger_fraction of 1.0 — spill only once the GPU is completely full, which is precisely why compression had nothing left to allocate in. All 22 queries, 1 iteration, both arms identical except the compression block: sum of per-query time off 34.97s on 40.15s 1.15x q21 off 4.65s on 10.22s 2.20x other 21 queries 0.79x - 1.06x The 1.15x is entirely q21; everything else is within measurement scatter. Whole-sweep spill activity: 1 seeded from lineage, 2 explored, 3 compressed, 3 declined, and zero OOM declines against 20 in a single query before. So the allocation failures were a configuration problem, not an architectural one — the earlier conclusion that they were inherent to compressing on the device being evacuated was wrong. They were inherent to spilling only once it is already full. The flip side is that at SF100 on a 12GB card with 0.95 usage, almost nothing spills: six spill events across 22 queries. Judging this feature needs a workload that genuinely spills. Lineage seeding fired only once, so it is not yet carrying the load it was built for. Adds run_spill_sweep.sh (standalone; performance_test.py needs passwordless sudo) and rebases both benchmark configs on the working config, dropping two keys this build's reader rejects: executor.duckdb_scan and operator_params.default_scan_task_varchar_size. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…er-second Answers whether the downgrade thresholds are relative to the usage limit or the device: they are relative to the limit. usage_limit_fraction feeds _gpu_capacity -> memory_capacity, and downgrade_trigger/stop_fraction multiply that. Confirmed empirically — dropping the limit from 0.95 to 0.30 took spill activity from 6 events to 322 with the fractions untouched. No adjustment needed. All 22 queries at SF100, usage_limit_fraction 0.30: sum off 115.81s on 135.67s 1.17x q21 off 82.12s on 80.79s 0.98x q22 off 0.48s on 8.70s 18.02x q10 off 1.81s on 9.05s 4.99x q4/q11 ~4.5x other 17 0.92x - 1.06x The overhead is per-edge and roughly constant, not proportional to query time. q21 spills heavily for 80s and compression is free there — it pays for itself. q22 runs in 0.48s and pays 8.2s of setup. That argues for gating compression on expected spill volume per edge rather than enabling it globally. OOM declines also reappeared (3) once the free margin fell from ~4.6GB to ~1.44GB, so there is a floor on the headroom compression needs that a fraction cannot express — a minimum-bytes floor alongside it would. Lineage seeding is now doing real work: 37 seeds against 13 explores, versus 1 seed in the 0.95 run. It engages once there is enough spilling to matter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…oring An edge's first spill no longer runs the explorer. It installs plans straight away — seeded from the base table's offline plan where column lineage reaches, and a fixed default everywhere else — and defers exploration to the edge's first expiry on the normal spill_replan_after_uses schedule, so a beam search is only paid for once an edge has proven it spills enough to amortize it. This targets what the 30% sweep showed: the overhead is a fixed per-edge setup cost, not proportional to query time. Measured across all 22 queries at SF100: explore on first spill 1.17x overall, q22 18.0x, q10 5.0x, q4/q11 4.5x bitcomp 2.69x overall, q21 8.57x bitpack 0.95x overall, q21 0.92x <- chosen delta -> bitpack 1.07x overall, q21 0.98x Deferring exploration removes the short-query regressions outright (q22 18.0x -> 1.0x). Choosing the default then mattered more than expected: it is applied to every un-seeded column on every spilling edge, so its speed matters more than its ratio. bitcomp compresses well but is an entropy coder, and on q21 — the one query that spills heavily — cost 8.6x, because the explored plans it displaced were cheap bitpack/delta cascades. delta -> bitpack was expected to win on the monotonic key columns that dominate TPC-H spill traffic. It does not: partitioning has already narrowed those columns by the time they spill, so the extra pass costs more than the width it recovers. STRING and nested columns are stored raw. A str_split cascade was written and reverted — its cost on real data is unmeasured, and a wrong blind default is expensive on exactly the heavily-spilling queries this is meant to help. The margins are not solid: run_spill_sweep.sh runs the off arm first, so it reads colder and the bias flatters compression, and the off arm swung 43.72s -> 118.84s across runs on identical config. Fixing the harness is the next thing worth doing. The shape of the result is what is solid, not the 5%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_spill_plans and _spill_origins are keyed by shared_data_repository*, and SiriusContext::QueryEnd destroys every repository. Nothing cleared them in production — clear_all() is only ever called from tests — so the maps grew without bound holding entries keyed by freed pointers, and a repository later allocated at a recycled address would inherit plans and verdicts belonging to an unrelated edge. The column-count guard catches most such collisions; one that slips through applies a wrong plan and fails or compresses badly. Not a correctness problem for the data (the .hpln header is self-describing, so decompression stays correct) but wasted work and verdicts attributed to the wrong edge. clear_spill_state() now runs at query end, immediately before clear_all_repositories(). The offline table plans are deliberately left alone: they come from input_plan_dir at startup rather than from a query, and are what lineage seeding reads, so the next query re-seeds from the same source. No write-back of explored plans to the per-column store. An exploration is evidence about one spilling edge's data, not about the base column in general, and promoting it would let one query's intermediate distribution silently redirect every later query's plans for that column. Documents the consequence: spill_replan_after_uses counts uses within a single query, few edges reach 128, so exploration rarely fires and the machinery around it is mostly dormant. Lineage seeding plus the fixed default is what carries the work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…downgrade
Adds two ways to hold data compressed while it is still on the GPU, and the
plumbing to make either safe to measure against an uncompressed baseline.
Task-output compression (off by default, `enable_output_compression`).
The operator sink compresses a finished task's output before publishing it.
Gated three ways, each for a measured reason:
- FULL barrier only. A PIPELINE/PARTIAL consumer starts on the batch almost
immediately, so compressing it buys an immediate decompress. q9 showed the
failure mode exactly: 24 batches compressed, all 24 decompressed, zero
downgrade requests, 1.15x slower.
- 64 MiB minimum. Compression costs ~2.95 ms/batch largely independent of
size - measured at 1-2% of the codecs' rated throughput, i.e. dominated by
per-column, per-plan-node stream syncs. Gating took the SF100 sweep from
1861 batches to 117 (16x fewer) while keeping 61% of the bytes saved, and
turned a 1.16x regression into 0.92x.
- Per-column plan quality. A column is compressed only where lineage reaches
an offline plan whose *measured* ratio and throughputs clear a gate
(default 3x / 250 / 250 GB/s). Those numbers live in `#` comments that
split_plan_dsl strips, so they are now parsed at load time and kept
index-aligned with the plan blocks.
Compress-to-device as a downgrade target (off by default,
`enable_device_compression_downgrade`). A third option beside HOST and DISK:
the batch stays on the GPU and stays usable, just smaller, so there is no D2H
copy now and no readback later. Only taken when the candidate *set* can satisfy
the request - compressing frees size*(1 - 1/ratio), so candidate bytes C
against request R need ratio >= C/(C-R), and C <= R cannot be satisfied at any
ratio. All-or-nothing: a partial compression would spend GPU time, still miss
the request, and leave batches needing a decode.
Spilling an already-compressed batch. Every converter out of
compressed_device_representation targeted the GPU, so the downgrade executor
could not evict one at all - convert() threw "No converter registered" and the
batch stayed resident. Adds device->host and device->disk staging, which is the
cheapest spill in the system: the bytes are already compressed and already laid
out as .hpln wants, so it is a straight D2H of the payload with no compression,
decompression or re-layout.
Memory accounting. Three places assumed "same tier + same space => nothing to
do", which the GPU tier holding two representations breaks:
- lock_or_prepare_batch returned a compressed representation straight to an
operator that casts to gpu_table_representation.
- get_estimated_bytes_to_materialize_input counted a GPU-resident compressed
batch as zero, so its decode allocated unreserved.
- The same function ignored the decode transient entirely: reconstruct stages
the compressed payload on device *alongside* the table being built, so the
peak is both at once, not just the final table.
Logical vs physical size. Sites that *size work* - CONCAT's batching threshold,
task-creator placement scoring, the execution-history basis, partition and
sort-sample sizing, working-set estimates - now use
get_uncompressed_data_size_in_bytes(). Using the compressed number made
compression change the plan rather than the footprint (CONCAT packed ~3x more
rows per batch), which both distorts the engine's decisions and makes an
on-vs-off benchmark incomparable. Sites that ask what a batch occupies *now*
(reservations, eviction candidates, telemetry) keep get_size_in_bytes().
Also extracts the device-blob staging out of pin_table.cpp so the pin path and
the new converter share one copy of the slab/alignment rules.
Docs record the 3-way measurement (spill compression is neutral at 0.30 budget;
output compression 0.90-0.94x) and withdraw two earlier conclusions as
artefacts of fixed arm ordering - including one of my own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Work in progress, not yet validated end to end. - the device tier now runs for every downgrade request, not only for an explicit target - compressed batches are deprioritised in the candidate search, via is_device_compressed + stable_partition - fix: to_read_only() blocked in estimate_device_compression, deadlocking the downgrade thread - fix: to_read_only() blocked in is_device_compressed, same shape - fix: reserved the full uncompressed size on an already-OOM GPU, so the reservation could never succeed - release_table() moved after the threshold check, with a bounded retry (10x, 50ms*n) - incremental per-column tree release during staging, via the on_buffer_copied hook - the payload memset is removed; this assumes the compressor bakes the over-allocation into what it emits (see feat/simpatico-int16 eb32fda) Last green validation was 47/47 [compression] BEFORE the memset removal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
build_device_compressed_blob eagerly re-read the compressed_table at staging time. That is right for pin_table — a pinned chunk is read by many queries, so the parse amortises — and wrong for the output and downgrade tiers, where a batch is decompressed at most once. There the re-read is pure cost, paid at the moment the device is most constrained: it takes decode scratch from `mr` for every non-fused codec (ans, lz4, snappy, dictionary, str_split, ALP), and it runs a make_numeric_column dtype check that rejects DECIMAL/TIMESTAMP. The blob now retains the (small, host-side) header and builds its table on first use, behind a once_flag — compressed_device_representation::table() is reached from the parallel decode path, and a blob is shared by every projection and clone of the same chunk, so it has to be thread-safe. This mirrors what compressed_host_representation already does in decompress_host_to_gpu. build_device_compressed_blob takes reconstruct_now; only pin_table passes true. The stored header stays valid however late it is used: the slab hands out its own aligned offsets positionally and ignores the dense ones in the header, so the header describes the pre-staging layout and the slab the post-staging one, exactly as in the eager case. This also removes the one throw that could not be retried. With the reconstruct gone from compress_gpu_to_device, the only remaining failure after release_table() is the payload allocation, which fails before any byte is copied and so leaves `ct` intact — every attempt is repeatable. staged_release_began and the terminal "source already released" branch that tracked the unrepeatable case are deleted. The bounded retry stays: the source is already released by then, so without it a transient OOM on that allocation is fatal for the batch rather than a decline. [compression] 47/47 (598 assertions), including both cases from the known flaky pair. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
downgrade_executor::start() installs a per_thread_init that calls cudaSetDevice, but only on the worker pool. _processing_thread never got it — and the in-place compression pass runs there, in processing_loop(), not on a pool worker. cudaSetDevice is what makes the device's primary context current. Simpatico derives its JIT CUfunction lazily on whichever thread first asks for it and caches it by device id rather than by context (CompiledKernel::func_for_current_device, nvrtc_compiler.cpp:96). When the unbound processing thread won that race, cuKernelGetFunction returned a handle that cuLaunchKernel then rejected with CUDA_ERROR_INVALID_HANDLE, and every in-place compression attempt declined. Measured on q3/SF100, arm 'device' (output compression off), before -> after: batches compressed 0/78 -> 43/43 (7 passes) declines 75 -> 0 cuLaunchKernel failures 75 -> 0 freed per pass - -> 40-241 MB The failure was previously read as memory pressure, because it only ever appeared during a downgrade. It is not. Enabling task-output compression alongside it — which lets a task-executor thread populate the CUfunction cache first — takes the identical run from 0/78 to 76/76 with no other change. The variable was cache-warm order, not free memory. The real CUDA status was invisible: launch_encode_fused_tree discards its CUresult and reports only "launch_encode_fused_tree failed", with the detail going to stderr via fprintf rather than to the log. Propagating that status is worth doing upstream in simpatico. The same signature appears 26-40 times per run in the archived spill sweeps (spill_sweep, threeway_sweep, fourway_sweep), all on the spill arm, which runs on this same thread — very likely the same cause, though that is corroboration rather than a matched before/after. [compression] 47/47 (598 assertions). probe_both.yaml is the diagnostic config that isolated this: output compression and the device tier enabled together in one process. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ed" conclusion The device tier's launch_encode_fused_tree failures were a missing cudaSetDevice on downgrade_executor's processing thread, not memory pressure. Records the before/after (0/78 -> 43/43 batches on q3/SF100), the control that identified it (enabling output compression alongside took the unfixed run to 76/76), and the two lessons: the CUresult was discarded so the real error only reached stderr, and CUDA_ERROR_INVALID_HANDLE is not an allocation failure. Narrows the scope of the earlier "inherent to compressing on the device being evacuated" claim rather than deleting it — the std::bad_alloc declines it was written about are real and still stand; only the extension to the handle-error signature was wrong. Also notes that every previously reported device-arm timing measured a feature that never fired. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ight query sirius-db#1327 replaced the single data_repository_manager with a per-query registry. The TIER 1 spill loop was updated with it; the in-place compression pass above it was not, because it landed on a branch that predated the change. It still walked one manager, so the device tier only ever saw one query's repositories. Use the same traversal as TIER 1: newest query first (get_all() is ascending by query id, so iterate in reverse), then that manager's repositories. Memory pressure is a global condition, so candidates must come from every in-flight query, not just one. Caught by the compiler only because sirius-db#1327 also renamed the member. Had the name survived, this would have compiled and silently narrowed the candidate set. Verified after the rebase on q3/SF100, arm 'device': 9/9 batches compressed in the observed pass, 0 declines, 0 CUDA errors. [compression] 47/47. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cuKernelGetFunction binds the kernel to the context current on the calling thread, and the resulting CUfunction is only launchable from that context — cuLaunchKernel rejects it anywhere else with CUDA_ERROR_INVALID_HANDLE. The cache was keyed on cudaGetDevice(), so on one device the first thread to ask populated the entry and every other context got a handle it could not launch. The old comment stated the assumption that made this safe: "RMM/cuDF sets up the current device context correctly per-thread". That holds for threads bound at startup and not for a thread that was never bound. Sirius's downgrade processing thread had no current context, won the race to populate the cache, and every in-place compression attempt on it failed to launch — read for a long time as memory pressure, because it only ever appeared during a downgrade. Also make the unbound case work rather than fail: if no context is current, initialize the device's primary context via the runtime API and use that, instead of deriving a handle against no context at all. This makes the class correct on its own. The caller-side cudaSetDevice on the downgrade thread stays as defense in depth, and because the two are independent, either alone is sufficient. Upstream candidate: this file is shared with dev via sirius-db#937/sirius-db#1325, so it wants its own PR against simpatico rather than riding in on this branch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ream at a time The decompress converters have fanned columns across a stream pool since they were written; the three compress sites (spill-to-host, spill-to-disk, and the output/device tier) looped over columns on a single stream. Adds `simpatico::compress_columns(table, per_column_plans, threads, ...)` — the mirror of `decompress(table, column_threads, mr)`, taking one plan per column rather than a `---`-separated string, since every compress caller resolves plans per column — and routes all three sites through one helper that picks serial or parallel on `compression_column_threads()` and applies the same pre-barrier the decode path uses (pool streams do not observe the caller's stream). Also drops the `cudaStreamSynchronize` `compress_column`'s walk did after every generic op. Its comment claimed it was needed "before we read output column views/sizes", but everything read afterwards is host-side metadata of already-allocated columns, and every variable-output compressor reachable from make_compressor already syncs internally where it reads a size back from the device: nvcomp_batched_codec after both compress and scatter, plus dictionary, str_split, alp/alp_rd and bitextract. The fixed-output ones (identity) never needed one. All the barrier did was stall the walk once per plan node. `set_decompress_column_threads` becomes `set_compression_column_threads`: one knob, and it now governs both directions. Verified live in nsys at SF1000 — `simpatico::compress_columns[threads]` with overlapping `compress_column_worker[col=N/M]` ranges, ~19 ms for a 4-column batch against ~19 ms per column serially. Measured effect on end-to-end query time so far: none. On a 251 GB GPU, `gpu_to_device_compress` is 190 ms of an 11 s q9 trace (0.4%) against 9.6% in `decompress_column`, and a 6-iteration A/B of q9 in the output arm is 2.15 s patched vs 2.12 s unpatched. The change is a latency fix on a path that is not currently the bottleneck at this scale factor; it should matter where a downgrade's compression is on the critical path. [compression] 47/47 (598 assertions). The simpatico ctest `bitpack_layout_contract` fails, identically on unpatched HEAD — pre-existing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings in 22 commits, notably sirius-db#1357 (open compression issues), sirius-db#1337 (config defaults / cucascade memory API), sirius-db#1401 + sirius-db#1328 (S3 ETags, DuckDB 1.5.5). The substantive conflict is threading in the compression converters. sirius-db#1357 removes the worker-thread model (set_decompress_column_threads) in favour of a thread-local simpatico::stream_pool whose work is submitted from the calling thread, because cuCascade's reservation state is thread_local (reservation_aware_resource_adaptor: "Per-thread, per-instance reservation state"). This branch had gone the other way in 93e71e4, extending worker threads to the compress path. Worker threads carry no reservation, so their encode buffers bypass the reservation's headroom and are checked against the raw pool capacity, throwing LIMIT_EXCEEDED ("not enough capacity to allocate memory") regardless of free GPU memory. Measured on S3/SF300: 36-39 failures per run, flat across downgrade triggers 0.8/0.6/0.5 -- confirming it is not memory pressure. Resolution: - decode paths (host/device/disk -> GPU) all use dev's decode_pool() - compress path is serial on the caller's stream; there is no pool overload for per-column plans (compress_with_plan takes one whole-table DSL), so restoring column-parallel compression needs a pool-based compress_columns first - dropped compression_config::column_threads and its SET option accordingly - kept branch-side estimated_materialization_bytes (type-based, and accounts for GPU-resident compressed batches) over dev's size-heuristic lambda - combined both sides in the plan generator: column_origins lineage (branch) plus compressed_materialization_observer / set_physical_types (dev) Also fixed cleanly-merged files broken by the merge: sirius_context.cpp called the deleted setter; test_spill_compression.cpp used cucascade builder methods renamed by sirius-db#1337; test_compression.cpp used the pre-rename pin_table_compression_max_compressed_fraction option name. Build green; compression suite 59/59 (1391 assertions). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Compression strips a column's validity into a sidecar beside its plan tree, and the .hpln header serializes that record per column, so a compressed chunk can answer how many of a column's rows are null from the header alone. column_validity returns the sidecar, column_null_count the count. Both fail closed. A column with no plan tree, an index past the table's width and a negative recorded count all read as unknown rather than as no nulls; a blob written before the record existed never gets this far, since the reader rejects any header version it does not know. The two compacting per-column decodes gain an opt-in for validity. Their output holds only the selected rows while the stored bitmask still spans the whole chunk, so returning it verbatim would pair each value with another row's validity. A nullable column is refused unless the caller passes stripped_validity, which returns the values alone and hands back the full-width sidecar for the caller to compact against the selection it supplied. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
materialize_compressed decoded values only and threw on any decoded column that came back with nulls, since none of its routes wrote an output validity buffer. With the sidecar in place all four routes can carry one. The dense decode and the full-decode fallback get it for free: decompress_column reattaches the mask itself, and the fallback's gather propagates it. The two compacting routes cannot, because they return only the selected rows while the stored bitmask still describes every row of the chunk. So they take the sidecar and gather it by the selection's own batch-local row list, which is the same list the values were selected by: bit i of the output is bit local_indices[i] of the stored mask. Copying it verbatim would pair each value with another row's validity — right row count, wrong answer. gather_validity_bits is that gather, alongside the existing multi-source one. The all-valid and all-null shapes store no bitmask at all and need no kernel: one is a no-op and the other regenerates a constant mask. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ility Both install gates asked pinned_column_null_count first and treated its nullopt as unsafe, and it returned nullopt for every compressed chunk because it could not read a null count without decoding. That refused compressed origins outright whether or not they had any nulls. It reads the count from the chunk's validity sidecar now, and pinned_column_nulls_are_safe admits a compressed chunk on the same terms as an uncompressed one, since every decode route carries validity through. A chunk carrying no blob still answers nothing and stays unsafe, which is where resolve_pinned_column refuses too — the two checks have to agree. Every other refusal is untouched: host tier, multi-GPU, chunks disagreeing on their carrier width, and a filtered scan over a compressed pin, whose survivors are decided inside the fused decode where this path cannot see them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… columns materialize_compressed had no coverage at all: its route choice was never exercised, and no test reached it with nulls. Each of the four routes now gets a case, and each pins the route from below rather than trusting the cascade - it calls the simpatico decode the route rests on and asserts it serves or refuses for that plan. The plans classify differently: a bitpack root takes the sparse walk, a delta-over-bitpack root has no random access and takes the mask kernels, and an ANS root has no compacted route at all. The fixture holds row i at value i and is null on an irregular set of rows, so a materialized row names the row it was read from in both halves. That is what makes the repeated, unsorted selection case meaningful: dedup, materialize in table order and gather back gives validity three chances to part company with its value while the row count stays right. Also covered: the null count read from the header, an all-null column, a no-nulls compressed origin, and a nullable compressed pin resolved from a real pinned_entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The known-limits entry said a compressed origin is refused unconditionally because per-column nullability inside a compressed blob is opaque. It is not opaque any more: the validity sidecar makes the null count a header read, and the decode routes reattach the mask. Record what the compacting routes have to do about it, since that is the part that is easy to get subtly wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…p front Nullable columns compress now, so a filtered compressed scan can reach one whose stored bitmask spans the whole chunk while the fused decode returns only the survivors. Nothing on that path compacts one against the other. decompress_column already refuses it, but only as a per-column failure that unwinds the whole batch mid-flight. Check it with the other structural preconditions instead, before any device work is issued. Also picks up clang-format reflows in the new test file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A compacting decode already holds the answer — the tier-B route materializes the survivor index list to gather by, and every other route leaves the counted mask it balloted — but nothing outside the decode could read it, so a consumer addressing the chunk by position had no way to follow the compaction. pushdown_request::report_survivors asks for those positions as an ascending INT32 list on pushdown_outcome::survivor_rows. pushdown_outcome::compacted reports that rows were dropped at all, separately from row_filtered, which additionally claims the whole filter was carried. Off, the decode does neither. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A batch off a compressed pin can be restricted twice, and each stage reports positions in its own input: the decode's are chunk-local, the residual filter's index the decode's output. A pin-order rowid built from either alone addresses rows the batch no longer holds. A split carrying a deferral now asks its decode for the positions it kept and gathers the decode's list by the residual's. Fails closed at both ends: a compaction with no positions to account for it throws, and so does a survivor list whose length does not match the rows handed on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both stages that can drop rows on a compressed scan now report their survivors, so the install gate no longer has to refuse the shape. Host-tier pins and ingestibles that cannot report survivors stay refused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The merge at eed19f0 ("Merge remote-tracking branch 'origin/dev' into compress-spills-v2") resolved src/compression/compression_converters.cpp by taking the dev side wholesale: 2069 lines -> 259. That silently dropped seven of the file's nine converter registrations -- every encode direction this branch exists to provide: gpu -> compressed_host compressed_host -> compressed_disk gpu -> compressed_disk compressed_device -> compressed_host gpu -> compressed_device compressed_device -> compressed_disk compressed_disk -> gpu Only the two decompression registrations survived, which is why pinned-table compression kept working while spill compression became a silent no-op: the registry lookup in convertible_data_batch::try_convert_compressed found no encoder and every batch declined with "No converter registered for source type 'cucascade::gpu_table_representation' to target type 'sirius::compressed_host_representation'". It still compiled and still ran, so nothing failed loudly -- a full SF1000 A/B measured byte-identical spill traffic in both arms before the cause was found. Restored by redoing the merge properly (3-way from the real merge base ab3603c) rather than checking out the branch side, because dev did not simply delete this work: it replaced the branch's decode_equality_pushdown mechanism with the newer decompression_pushdown_scan from sirius-db#1474, and the branch's 2069-line version does not compile against the current headers. Six conflicts, resolved as branch-owns-encode / dev-owns-decode, includes unioned. decompress_device_to_gpu keeps dev's body but takes the deferred table(stream, mr) overload, since this converter now serves staged spill blobs as well as pinned chunks. try_release_table() is supplied by the paired cucascade commit; the branch called it at 930d823 (a wip commit) but it had never existed on any cucascade ref. Measured after restoring, TPC-H SF1000 on RTX PRO 6000, input pinned to GPU: q8 0.8933s -> 0.7694s (-13.9%), its downgrade time 368.8ms -> 193.7ms, and the 31 large spill batches that previously declined now compress. Under mixed pinning, where eviction actually reaches disk, q9's downgrade time falls 30,941ms -> 1,670ms and disk traffic 39.06GB -> 1.98GB, and q18 goes from gpu_oom to completing in 6.06s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… shape error decompress_with_pushdown folded "the probe returned null" into the same throw as "the probe returned the wrong shape", under the message "membership probe result shape mismatch". The two have opposite causes, and the message names only the second, so the failure reads as a row-count bug when it is not one. On TPC-H SF1000 that message fired 705 times in a single suite -- 100% of every filtered-decode attempt across q3/q5/q7/q8/q17/q20/q21, i.e. decompression pushdown was enabled and never once succeeded. Printing the actual shapes showed keys=109206639 rows against expected=109206639: the count was always correct and the probe was simply null. Every compute_mask implementation (in_list, small_in_list, Bloom) returns null for exactly two reasons -- the probe key dtype does not equal the filter's build-side key type, or this device has no replica -- so the null branch now says so and names both dtypes. That is what identifies the real conflict: pin-table compressed materialization narrows key columns (measured on q7: "pin column 1 narrowed: int64_t -> int32_t"), the decoded keys arrive as INT32, the dynamic filter's key type is the join key's INT64, and compute_mask declines. Compressed pinning currently disqualifies every membership probe from decompression pushdown, and enabling more table plans makes it strictly worse (q20's declines go 129 -> 147 when the four disabled plans are turned on). Diagnostics only, on the throw path; the fix for the dtype conflict itself is not attempted here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two configs differing only in compression.enable_spill_compression, both at memory.gpu.usage_limit_fraction 0.908 so the query pool is identical in each and the 4 GiB compression arena is a partition of the device rather than extra memory. The arena has to come from the config file: SiriusContext::initialize() is the only caller of init_compression_device_pool(), so `SET spill_compression = true` alone flips the flag while leaving the encoder allocating from the very pool whose exhaustion triggered the spill. run-suite.sh drives one query per process. In grouped mode a failed pin raises out of _execute_multi and takes the whole run down, and at SF1000 on a 96 GB card several queries have a pinned column set larger than the GPU tier; one process per query contains that to the query it affects. It also distinguishes pin_oom / gpu_oom / watchdog_killed / timeout, and appends when resuming a partial arm. memory-watchdog.sh is the backstop for the failure that took this machine down mid-run: a GPU OOM fell back to DuckDB CPU inside the same transaction, and a CPU q18 at SF1000 does not fit in the ~22 GB left after the host tier pins 40 GB of 62. The run scripts now set enable_duckdb_fallback=false to remove that path; the watchdog kills the benchmark rather than the machine for every other route to an OOM, SIGTERMing the harness first so the driver still records a status. spill-report.py attributes downgrade traffic to queries from the downgrade_executor "request done: ... to_host/to_disk" summaries (debug level) and diffs two arms. compare-results.py diffs per-query result.txt between arms -- GPU-to-GPU on purpose, since a DuckDB CPU reference at SF1000 is what OOM-killed the box. Host-tier notes recorded in the configs: q1's compressed lineitem pin is 71.2 GB on the GPU tier, larger than this box's entire 62 GB of RAM, so LINEITEM cannot be host-pinned here at any ceiling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… work Taking one side of a conflict is a legal resolution, so a merge can delete a feature wholesale without conflicting, failing to build, or failing any test that does not already cover the deleted code. eed19f0 did exactly that to compression_converters.cpp (9 converter registrations -> 2) and the loss was found only after a full SF1000 campaign measured both A/B arms identical. merge-guard snapshots a manifest of anchors -- grep patterns whose match count must not fall -- and re-checks after the merge. Verified against the incident: `merge-guard.sh compare eed19f0^1 eed19f0` reports the four lost anchors and exits 1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
25 commits, 9 conflicts. Notable resolutions: compression_converters.cpp / simpatico_codegen.cpp: took dev's filtered-decode body wholesale. Dev independently fixed the membership-probe bug this branch had only diagnosed: a null probe is now a legitimate DECLINE that fills its mask with all-ones (the AND identity) and lets the remaining sources proceed, throwing only when every probe declines. That supersedes the diagnostic-only change at 6816b17. Taking their file dropped two definitions only this branch had -- compress_columns() (both overloads) and reject_sliced_columns() -- caught by the linker, not by merge-guard, and restored here. merge-guard gained an anchor for them. cucascade: moved from ea9e4a14 to dev's 1b0e7b6c (13 commits) with try_release_table cherry-picked on top. The merge had kept this branch's older pointer, because touching the gitlink counts as a change on our side, which left dev's new code building against a cucascade missing sirius-db#184 "wait for asynchronous batch readers" -- two stream-lineage tests failed until the upgrade. The newer cucascade renames the host-space builder API, so src/ and 30 test files move from {use_host_per_numa, set_per_host_capacity, *_per_host} to {use_numa_id_as_host_id, set_per_numa_region_capacity, *_per_numa_region}, and use_host_per_gpu -> use_gpu_id_as_host_id. dev's setup_configurator is also a superset -- it handles a fractional host capacity this branch dropped. compressed_representation: kept dev's has_table()/table() contract exactly (has_table() == "a blob is attached", table() noexcept) and added this branch's deferred table(stream, scratch_mr) alongside it, which spill-staged blobs need because they reconstruct lazily. Making has_table() stricter was tried first and broke dev's cached-serving tests, which rely on reading a not-yet-built table. pipeline / task_scheduler: took dev's weak_ptr completion-handler ownership, which is the upstreamed form of this branch's 4b2fadd (dev 5737ad7 fixes the same hang). Kept the stall watchdog, which dev has no equivalent for. Dropped a write-only _no_pref_rr_counter reset that nothing read. sirius_extension.cpp: re-registered pin_table_compression_max_compressed_fraction as an alias for this branch's renamed compression_max_compressed_fraction, and the validator message now names both spellings contiguously. The rename was only half-applied on this branch -- its own tests still used the old name, so three tests were failing before this merge -- and dev keeps shipping tests for the old spelling. Tests: 3052 cases, 2 failing, both pre-existing and unrelated to the merge: - test_dense_count_join_detection: its child process loads the extension through the default pixi env's stock duckdb wheel and gets "Attempted to dereference unique_ptr that is NULL". Environmental; the same failure blocked the SF1000 harness until it was pointed at the duckdb-python env. - test_stream_lineage_item5:373: "Resource deadlock avoided", deterministic, still unexplained. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…pills-v2 13 commits carrying validity through the compressed late-materialization routes: a column's null count is now readable without decoding, compressed origins that can answer their own nullability are admitted instead of refused, and a filtered scan of a compressed pin composes the decode's survivors with the residual filter's. One conflict, in sirius_gpu_scan_operator.cpp, and additive on both sides: this branch's saturating_add/saturating_mul usings against the incoming composed_survivors helper. Unioned. That the rest merged clean is worth noting, because it lands on files the upstream/dev merge immediately before it (5b1d98b) had just rewritten -- simpatico_codegen.cpp and compressed_scan.cpp both took dev's side wholesale there. The two stages of survivor composition sit on top of dev's graceful probe-decline without contradicting it. Tests: 3052 cases, the same 2 failures as before this merge and no new ones -- test_dense_count_join_detection (its child loads the extension through the default pixi env's stock duckdb wheel) and test_stream_lineage_item5:373 ("Resource deadlock avoided", deterministic, still unexplained). merge-guard passes with every anchor intact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s-spills-v2
Brings in the frugal-caching / dynamic-IO work: readahead_scan_manager,
query_stage_manager, the reworked prefetching_cache and io_backend, and the
vendored ctrack instrumentation.
Conflict resolution notes, all in test/tpch_performance/performance_test.py,
where the branch had rewritten the CLI:
* drop_os_cache: the merge spliced two same-named implementations onto one
`else:` line. Kept both under distinct names and removed the orphan branch.
* The pin-compression SETs existed only in _build_nsys_temp_sql; restored them
in open_connection so --pin-compression is not silently inert.
* SIRIUS_PRE_SQL was no longer executed; restored in open_connection.
tools/merge-guard.sh check 8c6dfa7: PASS, all 9 anchors intact.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…RACK_REPORT Aggregating the ctrack tables walks every recorded sample across all call sites, and ds::device_read_async alone records ~17k calls while pinning a TPC-H SF1000 table. Running it at every query end cost a flat ~100 ms per query -- +138% on q6 (73 ms -> 174 ms) and +2% on q21 (1.315 s -> 1.337 s). The constant offset across queries of very different lengths is what identified it. The report is unchanged under SIRIUS_CTRACK_REPORT, and SIRIUS_IO_PROFILE still implies the aggregation it needs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… disk 82%
A batch containing a DECIMAL128 column could not be compressed for spill at
all, because neither carrier available to a column without a plan of its own
accepted the dtype:
* `bitpack` encodes a decimal column as its integer storage, but the codegen
dtype vocabulary stops at 64 bits, so a __int128_t storage type is not
fusable and the plan fails;
* `identity` leaves were reconstructed with cudf::make_numeric_column, which
rejects DECIMAL and chrono types with "Invalid, non-numeric type".
The batch therefore stored that column raw, which held the whole batch to
1.20x -- under the 1.33x gate (max_compressed_fraction 0.75). Batches that
declined three times had their edge written off (spill_error_tolerance), and
the write-off charges every column, so one unencodable column dragged the rest
of the table to disk uncompressed with it.
Three changes:
* default_plan_for gives DECIMAL128 `input -> ans`. `ans` is not a codegen op
and has no width limit; `simpatico explore` on a q18/SF3000 spill batch
measured 5.1x for it against 2.1x for bitpack on the int64 beside it.
* identity leaves are rebuilt with cudf::make_fixed_width_column, so the
passthrough carrier covers decimal, chrono and duration too.
* With every dtype carried, can_carry_without_plan is always true; it and its
three guards are removed, which also re-enables the early-release path for
batches that previously had to take the slower whole-table route.
Also hoists the plan-dir scan out of SiriusContext::initialize() into
load_compression_plan_dir() and calls it from the
`pin_table_input_compression_plan_dir` setter. The directory was otherwise read
only at initialize(), so setting it by SQL loaded nothing; pin_table masked
that by lazily loading a table's plan on first use, but the spill path reaches
plans through column lineage and never triggers that load.
Measured, q18 at SF3000 from S3 (3 iterations, unpinned):
baseline before after
bytes to disk 245.8 GB 149.2 GB 43.0 GB (-82%)
wall time 139.20 s 103.40 s 95.93 s (-31%)
below-gate declines - 483 3
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
set_current_device_resource_ref is gone; set_current_device_resource takes an any_resource and returns the previous one to restore. Without this the simpatico CLI target does not compile, so `simpatico explore` was unavailable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…arness
* run-baseline.sh / run-spill-compress.sh accept an s3:// --input (the local
directory check does not apply) and PIN=none, which leaves the per-table
tier vars unset and drops --pin-compression: pin_table globs local files,
so an s3:// input cannot be pinned, and compression happens at pin time.
* Both scripts drop --mode grouped, which the frugal-caching CLI rewrite
removed. Omitting --execution keeps the config as given.
* run-suite.sh compares the watchdog trip count either side of each query, so
a kill is attributed to the query it happened during rather than to every
later failure.
* The rtx6000 configs move to the `backend: sirius` spelling (uring for local
paths, REST for s3://) and omit the cache node, defaulting to mode "none".
* performance_test.py: the default (no --execution) path evicts the input
dataset with posix_fadvise(DONTNEED) instead of drop_os_cache(), which
needs passwordless sudo. Untangles the two implementations the merge had
spliced onto one `else:` line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…inality
STRING columns were always spilled raw: default_plan_for opened with a
non-fixed-width check that returned the passthrough DSL. On TPC-H q9/SF3000
that left the single largest column in the spilled batch -- 40.9% of its bytes
-- uncompressed, while `simpatico explore` found 17.5x on it with a dictionary
cascade.
Whether a dictionary pays depends entirely on cardinality. It stores the key
set once plus ceil(log2(D)) bits per row, so with D far below the row count the
index is much narrower than the string; as D approaches the row count the key
set converges on the whole column and the indices become pure overhead. The
offline SF1000 plans show both ends, having been explored per column:
dictionary for l_returnflag (19.3x), l_linestatus (37.4x) and l_shipinstruct
(61.8x), and rejected for the near-unique l_comment.
So estimate D and branch on it. cudf::approx_distinct_count is a HyperLogLog
sketch -- 16 KiB of registers at precision 12, ~1.6% standard error -- fed a
few cudf::slice windows rather than the whole column, so the probe allocates
nothing beyond the sketch and costs a bounded amount of work whatever the
column's size. That matters on a path that runs *because* memory ran out. It
samples several windows rather than one prefix because a spilled column is
often clustered by a partition key, and one window of a clustered column reads
as far less distinct than the whole.
Misjudging the estimate is not a correctness risk in either direction: an
over-optimistic dictionary still round-trips, and the batch's compression gate
declines it if it fails to pay. The probe is wrapped so a failure falls back to
the previous behaviour rather than failing the spill.
default_plan_for now takes the column rather than just its dtype, since the
decision needs the values.
Measured, q9 at SF3000 from S3 (3 iterations, unpinned), against the same build
with strings spilled raw:
spill compr. + this change
time 162.21 s 127.96 s (-21%)
bytes to disk 293.4 GB 206.7 GB (-30%)
achieved ratio 1.94x 2.66x
All 76 probes in that run estimated ~25 distinct and chose the dictionary,
matching the column's true global distinct count of 25 across 115.6M rows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DECIMAL128 was the one fixed-width dtype the codegen could not encode, so a
spilled batch containing one had to fall back to `ans` (the only non-codegen
op that handles it). SUM over any decimal promotes to DECIMAL(38,s), so that
covers every spilled partial aggregate in TPC-H.
NVRTC rejects __int128 by default -- "128-bit integer type is only supported in
Linux with the --device-int128 flag" -- and accepts it with that flag, so the
type needs no hi/lo decomposition. Passing it unconditionally costs nothing for
kernels that never name the type.
The rest is width plumbing plus the two places where 64 bits was load-bearing
rather than parameterised:
* simpatico_bitunpack_one stitches at most three uint32 words, which cannot
reach a 128-bit value: at an arbitrary bit offset that spans
ceil((128+31)/32) = 5 words. simpatico_bitunpack_one_128 is the 5-word
counterpart; simpatico_bp_at dispatches on sizeof(T).
* the packed buffer therefore needs five decode guard words, not three. They
are part of the stored payload (num_rows counts them), so this is a wire
change -- kVersion 12 -> 13.
* the chunk range and the pack accumulator were uint64_t and would silently
drop the high half; both now take the element's unsigned counterpart.
* simpatico_bit_width_u128 tests the high half first, since __clzll only
reaches 64 bits.
Two type tables needed DECIMAL128 named explicitly, because cudf has no 128-bit
integer type_id: unlike the narrower decimals and the chrono types, whose
storage column is a plain INT32/INT64, DECIMAL128's storage column stays
DECIMAL128 (scale 0) and is restored by apply_stored_dtype as usual.
unsigned_counterpart deserves a note: it defaulted every size except 8 to
uint32_t, so a 16-byte element would have been silently truncated rather than
failing to compile. It now names the 16-byte case first.
With this, default_plan_for's DECIMAL128 special case is gone -- it falls
through to the same bitpack default as every other fixed-width type.
Measured on a reconstruction of the q18 spill batch (SF1000 lineitem,
`select l_orderkey, sum(l_quantity) group by l_orderkey`, 40M rows, whose
DECIMAL(38,2) column reproduces the real batch's 5.117x under `ans` exactly):
ans (before) bitpack (after)
ratio 5.117x 8.493x (+66%)
compress 409.2 GB/s 556.0 GB/s (+36%)
decompress 808.3 GB/s 1138.3 GB/s (+41%)
`simpatico explore` on that column now selects bitpack over ans on its own, and
the benchmark's round-trip verify passes.
Full suite: 3213/3216, the same 3 pre-existing failures as the parent commit
(multi-GPU config on a single-GPU box, the duckdb-wheel dynamic load, and
test_stream_lineage_item5, which fails identically at 8c6dfa7).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The preceding commit made DECIMAL128 codegen-encodable and let it fall through
to the bitpack default, on the strength of an isolated benchmark: 8.49x against
ans's 5.12x on a DECIMAL(38,2) column, and faster in both directions.
That ranking inverts on real spill batches. Measured on TPC-H q18/SF3000, same
query and config, only the default differing:
encodes achieved ratio below-gate declines to disk
ans 1109 3.38x 3 @ 1.02x 43.5 GB
bitpack 403 2.42x 225 @ 1.30x 244.3 GB
bitpack's ratio is set by the per-chunk value range, and a spilled partial
aggregate's chunks are much wider than those of the fully aggregated column the
isolated benchmark used -- the benchmark column matched one dumped batch's ans
ratio, but not the population's (3.38x mean over 1109 batches). At 1.30x most
batches fall under the 1.33x gate, decline, and spill uncompressed, which is
how a query that runs in ~94s with ans was still on its first iteration after
20 minutes.
The codegen support stays: it is what makes DECIMAL128 eligible for cascades
and decompression pushdown, neither of which `ans` can do, and it resolved a
real inconsistency where compression_explorer offered bitpack for a dtype the
encoder rejected. It just should not be the blind default for spill.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tpack contract The simpatico test targets have not compiled since the RMM 26.08 update: set_current_device_resource_ref was removed in favour of set_current_device_resource, which takes an any_resource and returns the previous one. Same migration as 88a73a9 did for the CLI. `pixi run make` does not build these targets, so nothing surfaced it -- ctest reported all 19 as "Not Run" rather than failing. That mattered: with them building again, bitpack_layout_contract fails. It asserts the persisted packed column is the compact words plus the decode gather guard, and eaa0757 widened that guard from 3 words to 5 (the 128-bit gather spans ceil((128+31)/32) = 5 uint32 words) without updating the contract. The test is doing exactly its job -- the guard count is part of the persisted layout, which is why that commit also bumped kVersion. Updated to 5, with the reason recorded. Also adds DECIMAL128 coverage for both bitpack pack paths, which had none: the global atomicOr path takes the element's unsigned counterpart as its residual, while the shared-memory slab path takes a uint64_t. The slab path is safe today only because it is unreachable above 4 bytes -- RLE's `runs` child is always int32_t and its `values` child takes the slab only for int8/int16/int32 (see vals_is_bp_leaf_smem). The new case pins that invariant with values that straddle 2^64, so admitting a wider dtype to the slab without widening the residual fails here rather than silently truncating. The comment at that line now states the invariant instead of leaving the narrower type unexplained. simpatico ctest: 19/19 pass (previously 0 built). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
bitpack's ratio is set by the per-chunk value range, so it cannot exploit a
column whose values share a common divisor: a DECIMAL(38,2) holding whole
units is stored as mantissas that are all multiples of 100, and bitpack spends
~6.6 bits per element encoding zeroes.
bitpack is already an affine decoder -- it reconstructs chunk_min + unpack(i).
This makes it chunk_min + gcd * unpack(i), which is the same shape: both are
per-chunk constants applied uniformly to every element. The GCD is reduced in
the pass that already computes min/max, and the range is divided by it before
the bit width is taken.
Doing it here rather than as a separate operator matters, because a separate
node pays twice: an extra full read of the column for its own reduction, and a
division evaluated twice per element (it rewrites its child's read_expr, which
bitpack evaluates in both its min/max pass and its pack pass). Measured on a
real TPC-H q18/SF3000 spill batch, against a standalone factor -> bitpack:
column with no common divisor (the cost when this buys nothing)
plain bitpack 2.199x 361.4 GB/s enc 904.5 GB/s dec
factor -> bitpack 2.194x 292.0 GB/s 866.9 GB/s (-19.2% enc)
folded in 2.194x 347.2 GB/s 906.2 GB/s ( -3.9% enc)
DECIMAL(38,2) column (the case it is for)
factor -> bitpack 13.997x 465.6 GB/s 1221.4 GB/s
folded in 14.142x 567.8 GB/s 1212.0 GB/s
The encode penalty where it buys nothing drops from 19% to 4% and the decode
penalty disappears, which is what makes it defensible as always-on behaviour
rather than something the explorer has to select.
`chunk_divisors` is a TRAILING channel: it is appended after every pre-existing
one, so plan text naming only the original four channels stays valid and needs
no edit -- verified, such a plan still reaches 14.142x. A v13 payload has no
such buffer, so kVersion 13 -> 14 is what keeps an old payload from being bound
without it, rather than a runtime arity check.
Edge cases follow the standalone operator's: magnitudes are taken unsigned so
the type minimum is well defined, and an all-zero chunk, a GCD too wide for the
signed divisor slot, or (for DECIMAL128) a magnitude that does not fit the
64-bit GCD all fall back to the no-op divisor 1.
With this, the DECIMAL128 spill default goes back to bitpack, reverting
ec45e0d. That commit moved it to `ans` because bitpack left most q18 batches
at ~1.30x, just under the 1.33x gate, so they declined and spilled
uncompressed. Dividing the common divisor out first is what removes that: the
same column measures 14.14x here against `ans`'s 5.12x. (ec45e0d's stated
mechanism was also weaker than it claimed -- the two runs it compared were not
spilling the same batch shapes.)
The layout contract's final chunk was 0s plus one 65535, whose GCD is 65535 --
folding collapsed it from 16 bits to 1, correctly, but that stopped the chunk
exercising a wide bit width. Its filler is now 1 so the GCD is 1 and the intent
holds; the divisors channel gets its own assertion.
simpatico ctest 19/19; sirius [compression],[spill] 70/70.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fixed_size_host_memory_resource rounds every allocation up to a whole block (rmm::align_up(total_bytes, _block_size)) and charges the rounded size against the tier ceiling. At the 64 MB this config set, the host tier ran with no slack. The 64 MB was ours, from f2c773a, and had no justification beyond making the eager-pinning arithmetic in the comment land on a round number; cucascade's default is 1 MB. Measured on q18/SF3000, same binary, same data, config the only difference: host alloc failures task reschedules block_size 64 MB 78 254 block_size 1 MB 0 0 At 64 MB the tier hit its ceiling, GPU->host spills started failing, the GPU backed up and tasks retried against a downgrade that could not free anything -- the query never finished an iteration in 30 minutes. At 1 MB neither failure appears. This also explains a run-to-run difference that looked like a codec regression: a build whose only change was the DECIMAL128 spill carrier thrashed while its parent did not, on a config that was one allocation away from the ceiling in both. The block size was constant across that comparison, which is exactly why bisecting the code found a "culprit" commit that was merely the one that consumed the last of the headroom. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This reverts commit d0a8a37. That commit moved the host tier to 1 MB blocks because the newer code hit 78 host allocation failures at 64 MB. It fixed those failures and made things worse: q18/SF3000 went from a livelock to a 2.1x loss with `ans` (326 s against a 152 s uncompressed baseline) and a >20 min timeout with bitpack, because fixed_size_host_memory_resource stages a payload with one cudaMemcpyAsync per block -- a 191 MB payload is 3 copies at 64 MB and 182 at 1 MB. More to the point, 64 MB is the setting under which compression has produced its best measured result: q18/SF3000 in 81.2 s (best-of-3) at ba5c282, with 43.5 GB to disk, zero host allocation failures and zero reschedules. The block size is not the defect -- the known-good revision runs on it cleanly. Something in the four commits since consumes the margin it leaves, and that is what needs fixing, not the config that measured best. d0a8a37's reasoning was wrong on two further counts: it attributed the 64 MB to this campaign, when it was inherited from the GB300 reference config in bench/sf1000-repro (where a 471 GB tier gives 7362 blocks against our 40 GB tier's 625); and changing block_size alone silently cut eager pinning 64x, from 16.4 GB to 256 MB, because the three host knobs are a product. Choosing a block size for this hardware is a separate tuning question and should be measured on its own, not folded into a bug fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an experimental branch aiming to add support for compressing spills.
Some concepts already in place:
So far, (maybe partially due to the issues listed above) I haven't measured any performance impact yet. I'll try to get a relevant platform and workload combination (sf100 or larger on an RTX4500) and see what works there.
Then we'll figure out which modes we want to keep and start thinking about how the actual implementation should look like (e.g. for how to traverse the query graph to find good default plans).
Closes #949