WIP experimenting with compressing spills - #1343

Draft

joosthooz wants to merge 80 commits into

sirius-db:devfrom

joosthooz:compress-spills-v2

Commits on Jul 30, 2026

feat(compression): scaffold spill-compression infrastructure

joosthooz
and
claude
committed
48786c1
feat(compression): compress batches on the spill path

joosthooz
and
claude
committed
bd61a4f
docs(compression): record spill-compression progress and open items

joosthooz
and
claude
committed
f8d2463
test(compression): cover the spill-compression path

joosthooz
and
claude
committed
1053916
docs(compression): mark spill-compression work items complete

joosthooz
and
claude
committed
47d2bfa
fix(compression): make shared compression settings reach the spill path

joosthooz
and
claude
committed
0295260
feat(compression): memoize unviable spill edges and re-explore periodically

joosthooz
and
claude
committed
c723347
feat(compression): back off the spill replan interval when re-explores stop paying

joosthooz
and
claude
committed
0615f68
fix(compression): do not write off a spill edge on a transient error

joosthooz
and
claude
committed
239a188
feat(compression): track spill plans and verdicts per column

joosthooz
and
claude
committed
8b923be
feat(compression): only adopt a re-explored plan that performs differently

joosthooz
and
claude
committed
44397d4
bench(compression): measure spill compression at SF100; it is a 4.6x regression

joosthooz
and
claude
committed
2bc7f86
fix(compression): memoize failed explorations and bound explorer memory

joosthooz
and
claude
committed
bdde386
feat(planner): resolve base-table lineage for operator output columns

joosthooz
and
claude
committed
083f58d
feat(planner): name file-backed scans from their directory for lineage

joosthooz
and
claude
committed
07851af
fix(planner): resolve column lineage through scans and positional refs

joosthooz
and
claude
committed
6be7f3e
feat(compression): seed spill plans from column lineage

joosthooz
and
claude
committed
a333b84
bench(compression): full TPC-H sweep; regression is 1.15x and confined to q21

joosthooz
and
claude
committed
b92c877
bench(compression): 30% GPU budget sweep; overhead is per-edge, not per-second

joosthooz
and
claude
committed
21ac193
feat(compression): default new spill edges to bitpack instead of exploring

joosthooz
and
claude
committed
0d9a95f
fix(compression): clear per-query spill state at query end

joosthooz
and
claude
committed
03a01db
feat(compression): compress GPU-resident data, on task output and on downgrade

joosthooz
and
claude
committed
ef8058a
wip(compression): run the device tier for every downgrade request

joosthooz
and
claude
committed
9a3ce31
perf(compression): reconstruct the device compressed blob lazily

joosthooz
and
claude
committed
2a06147
fix(downgrade): bind the processing thread to its GPU

joosthooz
and
claude
committed
f29eccb
docs(compression): withdraw the "device tier is architecturally blocked" conclusion

joosthooz
and
claude
committed
948f7a2
fix(downgrade): draw in-place compression candidates from every in-flight query

joosthooz
and
claude
committed
e7ba387
fix(simpatico): key the JIT CUfunction cache by context, not by device

joosthooz
and
claude
committed
7021a64

Commits on Jul 31, 2026

perf(compression): compress a batch's columns in parallel, not one stream at a time

joosthooz
and
claude
committed
93e71e4

Commits on Aug 6, 2026

Merge upstream/dev into compress-spills-v2
joosthooz
committed
2f1b18d
refactor(compression): route the compress path through the shared stream pool

joosthooz
and
claude
committed
408eb3a

Commits on Aug 7, 2026

feat(compression): stop compressing spills while memory is under pressure

joosthooz
and
claude
committed
87dfd64

Commits on Aug 10, 2026

perf(compression): batch spill writes and stop staging whole payloads on the host

joosthooz
and
claude
committed
1f1d6d3
feat(compression): skip spill compression for batches too small to repay it

joosthooz
and
claude
committed
1571437

Commits on Aug 11, 2026

fix(spill): make spill compression reachable, and stop the query hanging when it is

joosthooz
and
claude
committed
4b2fadd
fix(spill): key spill compression by the batch's real source repository

joosthooz
and
claude
committed
d9a49f3

Commits on Aug 13, 2026

wip(compression): per-column disk artifacts, release-ordering fix, encode reservation

joosthooz
and
claude
committed
930d823
Merge upstream dev (c23da5aa) into compress-spills-v2

joosthooz
and
claude
committed
016f9b4
feat(memory): explain OOM trim decisions and make the threshold tunable

joosthooz
and
claude
committed
c8eef99
fix(compression): mark eagerly built blobs as having a table

joosthooz
and
claude
committed
d9d1084
feat(bench): validate every iteration, against a reusable CPU reference

joosthooz
and
claude
committed
71dd535
fix(build): commit the merge and reservation sources left in the working tree

joosthooz
and
claude
committed
ea7b3a2
feat(downgrade): sample tier occupancy so evictions can be interpreted

joosthooz
and
claude
committed
9b30145
perf(memory): rate-limit the OOM pool trim and its device sync

joosthooz
and
claude
committed
345ba40

Commits on Aug 18, 2026

feat(compression): key every spilled batch, so compression reaches the traffic

joosthooz
and
claude
committed
9b81ebc
Merge upstream dev (ab3603ca) into compress-spills-v2

joosthooz
and
claude
committed
0a2e1f9

Commits on Aug 24, 2026

Commits on Aug 25, 2026

(+LUMP)
aminaramoon
committed
f188ae7

Commits on Aug 26, 2026

feat(compression): support nullable columns transparently

joosthooz
and
claude
committed
0eb5fc1
feat(compression): read a column's null count without decoding it

joosthooz
and
claude
committed
7e3004b
feat(late-mat): carry validity through the compressed materialize routes

joosthooz
and
claude
committed
0182725
feat(late-mat): admit compressed origins that can answer their nullability

joosthooz
and
claude
committed
98b7df1
test(late-mat): cover the compressed materialize routes with nullable columns

joosthooz
and
claude
committed
9e1e11b
docs(late-mat): compressed origins are no longer refused for nullability

joosthooz
and
claude
committed
e03becc
fix(compression): refuse a nullable column in the fused scan filter up front

joosthooz
and
claude
committed
66c95d1
feat(compression): report the rows a filtered decode kept

joosthooz
and
claude
committed
811707e
feat(scan): compose the decode's survivors with the residual filter's

joosthooz
and
claude
committed
f8912ac
feat(late-mat): admit filtered scans of compressed pins

joosthooz
and
claude
committed
7446194
test(late-mat): cover a filtered scan of a compressed pin end to end
joosthooz
committed
8979d68
docs(late-mat): filtered compressed scans compose two stages of survivors
joosthooz
committed
f424002
docs(late-mat): record the measured effect of admitting filtered compressed scans
joosthooz
committed
e0669af

Commits on Aug 27, 2026

fix(compression): restore the spill-compression encode converters

joosthooz
and
claude
committed
a3b99c3
fix(compression): report why a membership probe declined, not a false shape error

joosthooz
and
claude
committed
6816b17
bench: SF1000 spill-compression A/B harness for RTX PRO 6000

joosthooz
and
claude
committed
f2c773a
tools: add merge-guard, an anchor check for merges that silently drop work

joosthooz
and
claude
committed
ff89ef7
Merge upstream/dev (3ac080d8) into compress-spills-v2

joosthooz
and
claude
committed
5b1d98b
Merge origin/feat/late-mat-nulls-compressed (e0669afd) into compress-spills-v2

joosthooz
and
claude
committed
8c6dfa7

Commits on Sep 7, 2026

Merge origin/feature_frugal_caching_dynamic_io (f188ae74) into compress-spills-v2

joosthooz
and
claude
committed
6500078
perf(profiling): gate the per-query ctrack aggregate behind SIRIUS_CTRACK_REPORT

joosthooz
and
claude
committed
87094c2
fix(compression): give DECIMAL128 a spill carrier, cutting q18/SF3000 disk 82%

joosthooz
and
claude
committed
68ad948
fix(simpatico): build the CLI against RMM 26.08

joosthooz
and
claude
committed
88a73a9
bench: support S3 inputs and unpinned arms in the spill-compression harness

joosthooz
and
claude
committed
ba5c282
feat(compression): pick a STRING spill carrier from an estimated cardinality

joosthooz
and
claude
committed
479c666

Commits on Sep 8, 2026

feat(simpatico): support DECIMAL128 in the codegen path

joosthooz
and
claude
committed
eaa0757
fix(compression): keep `ans` as the DECIMAL128 spill default

joosthooz
and
claude
committed
ec45e0d
fix(simpatico): build the tests against RMM 26.08, and restore the bitpack contract

joosthooz
and
claude
committed
8ecd8a7
feat(simpatico): fold a per-chunk common divisor into bitpack

joosthooz
and
claude
committed
f905e8a
fix(bench): host block_size back to cucascade's 1 MB default

joosthooz
and
claude
committed
d0a8a37
Revert "fix(bench): host block_size back to cucascade's 1 MB default"

joosthooz
and
claude
committed
d8bc9de