perf(compression): Add simpatico factor operator, improve ALP - #1729
Draft
joosthooz wants to merge 9 commits into
Draft
perf(compression): Add simpatico factor operator, improve ALP#1729joosthooz wants to merge 9 commits into
joosthooz wants to merge 9 commits into
Conversation
…imal columns Adds a codegen-fused `factor` operator: per 1024-row chunk it reduces the GCD of the values' magnitudes, stores it on the `divisors` channel, and hands `value / divisor` to its `quotients` child. Decode multiplies back. It is FOR's structural twin — a block reduction plus a closed-form lane rewrite, no shared slab and no change to the element count. Motivation: our TPC-H decimal columns all compress to plain `bitpack`, and a sweep of the other integer lightweight ops (for / delta / zigzag / rle / ans) confirmed bitpack genuinely wins — simpatico's bitpack already carries `chunk_min`, so it *is* per-chunk FOR+bitpack, and nothing else has anything left to exploit. The one exception is a column whose values share a common divisor, which no existing operator can exploit: TPC-H `l_quantity` is stored as mantissas 100..5000 that are all multiples of 100, so bitpack spends 13 bits on 50 distinct values. This is the integer/decimal analogue of ALP's factor step, but reached without ALP: a GCD is one block reduction rather than a search over (exponent, factor) combos, and it generalises ALP's power-of-ten factor to any common divisor. ALP itself is float-only and, being a decimal already, a DECIMAL64 column is in ALP's *output* form to begin with. Measured on TPC-H sf10 lineitem (60M rows, GB300), `factor -> bitpack` vs the current `bitpack`, all round-trip verified: l_quantity 4.885x -> 10.383x decode 2583 -> 2638 GB/s l_extendedprice 2.655x -> 2.649x (GCD 1: no-op, as expected) l_discount 15.604x -> 15.370x l_tax 15.604x -> 15.370x The ratio win comes with a faster decode, since there are fewer bits to unpack. On columns with no common divisor the per-chunk GCD collapses to 1 and the operator is an exact identity, costing only the divisors channel — so the explorer discards it on ratio. Edge cases handled: magnitudes are taken in uint64 via unsigned negation so INT*_MIN is well defined; an all-zero chunk (GCD 0) and a chunk whose GCD does not fit the signed divisor slot both fall back to the no-op divisor 1; the divisor divides every magnitude exactly, so the truncating signed division is exact for both signs and decode's multiply is lossless. Tests cover factor alone (Raw passthrough quotients), factor -> bitpack on int64 and int32, a non-factorable column exercising the divisor == 1 path, `divisors` routed to a separate non-fused leaf, and factor nested below delta (the slab decode path). Full simpatico ctest suite passes (19/19), including the registry-driven operator sweeps. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… to one parameter
Three defects in the ALP operator, all in how it applies the decimal scale.
**1. Scaling by a rounded reciprocal inflated the exception rate.** Encode and
decode both multiplied by `FRAC[k]` = the nearest double to 10^-k, which is
inexact. Since the encoder rejects any value that does not reproduce
bit-exactly, that one-ulp error turned ordinary decimal data into stored
exceptions: on TPC-H l_extendedprice as f64, 13.83% of rows. Scaling now goes
through the exact 10^k table only. (Worth noting: vortex's ALP has the
identical defect, so this is not a case of us lagging a reference.)
**2. `enc * FACT[f]` multiplied in int64 and could overflow.** With f near 18
the intermediate exceeded the encoded integer type -- signed overflow, UB. The
scale now applies in the float type, so it cannot overflow.
**3. Exception slots were filled with 0.** The cost model scores a candidate
using the min/max of its NON-exception encodings, but the buffer handed
downstream had 0 at every exception position, so a single exception could drag
a narrow chunk's bitpack range out to ~30 bits -- the emitted data disagreed
with the model that chose it. Exception slots now carry the vector's minimum
encoded value, which is exactly the frame of reference bitpack subtracts.
Fixing (1) exactly means dividing by 10^k, and an FP64 divide is punishing on
parts with cut FP64 throughput (measured 508 Gop/s vs 27.4 Top/s for FP32 on
GB300) -- it cost 4x on both encode and decode. Instead the reciprocal is kept
as an unevaluated sum: rhi = fl(10^-d), rlo = fl(10^-d - rhi), and the scale is
fma(v, rhi, v * rlo). Two multiply-class ops, and accuracy is a ratio concern
rather than a correctness one -- the encoder's round-trip check evaluates the
identical expression, so anything the approximation cannot reproduce simply
becomes a stored exception.
**The (e,f) collapse.** ALP as published encodes round(v * 10^e * 10^-f) and
decodes i * 10^f * 10^-e, but only the difference d = e - f affects the result,
and the published combo table constrains f <= e, so d covers exactly the same
ground. Sweeping d instead of (e,f) drops the f64 candidate set from 190 pairs
to 19 scales -- no loss of coverage, confirmed by the ratios below being equal
or better on every column -- and shrinks the metadata to a single value, which
also removes the per-vector combo tables, unpack_combo, and the pack/unpack of
(e << 8) | f. Encode's scale is now one exact multiply instead of two.
Measured on TPC-H sf10 lineitem (60M rows, GB300), all round-trip verified:
f64 alp -> bitpack before after
l_quantity 10.462x 10.462x
l_extendedprice 1.711x 2.654x (13.83% exceptions -> 0%)
l_discount 15.545x 15.545x
l_tax 15.545x 15.545x
table total 4.947x 6.655x
encode 2.48 GB/s 23.4 GB/s (9.4x)
decode 1032 GB/s 900 GB/s
l_extendedprice now matches what plain `bitpack` gets on the DECIMAL64 column
(2.655x), which is the right answer -- the column has no headroom beyond its
minimum bit width, and ALP no longer squanders 14% of it on phantom exceptions.
Encode is still far off bitpack; the remaining cost is the exhaustive sweep of
all 19 candidates over all 1024 values per vector, addressed separately.
The stored `metadata` channel changes meaning (scale exponent d, not the packed
pair) and the encoded integers change with it, so the .hpln format version goes
11 -> 12.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…mulators The encoder evaluated every candidate scale against every one of a vector's 1024 values -- ~19k round-trip encodes per vector, each several float multiplies -- and accumulated (exception count, min, max) with three shared-memory atomics per value per candidate. On GB300, where FP64 runs at 1/54 of FP32 (measured 508 Gop/s vs 27.4 Top/s), that search was the entire cost of ALP. Selection now scores 64 sampled values instead of 1024, and reduces each candidate across a warp by shuffle before a single atomic per warp -- 6 shared atomics per candidate per vector instead of ~3000. The emit pass still re-encodes all 1024 values exactly, so sampling can only ever cost ratio (a slightly worse scale), never correctness. The sample is 8 contiguous runs of 8 spread evenly across the vector rather than a fixed stride. A stride can land on a period of the data -- round-robin sensor readings, interleaved currencies -- and then observe only one phase of it, choosing a scale that suits a fraction of the rows. The published ALP algorithm samples for the same reason this does; vortex's plain-ALP path uses a stride, though its own ALP-RD path documents why runs are the safer choice. TPC-H sf10 lineitem (60M rows, GB300), f64 `alp -> bitpack`, round-trip verified: encode 23.4 GB/s -> 69.3 GB/s (28x against the 2.48 GB/s this started at) decode 900 GB/s -> 896 GB/s ratios 10.462x / 2.654x / 15.545x / 15.545x -- unchanged to the digit Ratios are identical here because these columns have a homogeneous decimal scale, which is the case sampling handles best; a column whose scale genuinely varies within a 1024-vector is where a sampled pick could differ from an exhaustive one, and the cost is bounded by the fallback to exceptions. f32 is unchanged at ~206 GB/s: its search was never FP-bound (FP32 runs at full rate), so the emit pass and memory traffic dominate there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ALP was float-only, at both the operator (it threw on anything else) and the explorer's dtype gate (fixed-point is classified with the integer op set), so a decimal column could never reach it -- which is why every TPC-H decimal column's plan is plain `bitpack`. A DECIMAL column's storage is already an integer mantissa, so ALP's scale search there needs no floating point at all: encoding at scale d is `m / 10^d` and the round-trip is exact iff 10^d divides m. That path shares the whole kernel -- sampling, warp-reduced accumulators, cost model, exception compaction -- and differs only in the per-value encode/decode, which now branches on whether the element type is integral. Exact power-of-ten tables replace the float constants. This is deliberately not redundant with the `factor` operator added earlier. `factor` divides by a GCD, so it needs a divisor common to EVERY value in a chunk and collapses to 1 the moment one value breaks the pattern; ALP can take a power of ten that MOST values share and bank the rest as exceptions. They cover different data, and the explorer can now weigh both. TPC-H sf10 lineitem, DECIMAL64, 60M rows, GB300, round-trip verified: plan l_quantity l_extendedprice l_discount table decode bitpack (today) 4.885x 2.655x 15.604x 5.638x 2724 GB/s factor -> bitpack 10.383x 2.649x 15.370x 6.622x 2617 GB/s alp -> bitpack 10.462x 2.654x 15.545x 6.655x 1206 GB/s ALP edges out `factor` on ratio and `factor` wins clearly on decode, so neither dominates -- which is the point of letting the explorer choose. DECIMAL128 stays out: its mantissa is __int128, with no power-of-ten table or atomic support here. The exceptions channel is now built from the source column's full data_type rather than a static type_id, so a fixed-point column's scale survives -- that column is also where `from_outputs` recovers the original type. compact_exceptions takes a data_type for the same reason (alp_rd's call site updated, no behaviour change). Tests cover DECIMAL64 with a clean common power of ten, with a fraction of values that do not divide (exercising exceptions on the decimal path), with no power of ten available at all (scale 10^0, an exact identity), and each composed with bitpack on the integers channel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…pln read `make_col` in the leaf reconstruction path used cudf::make_numeric_column, which rejects fixed-point and chrono types -- cudf classes both as non-numeric. Reading back any file with such a leaf buffer aborted with CUDF failure at column_factories.cpp:69: Invalid, non-numeric type. Nothing hit this before because no operator had ever stored a leaf buffer with one of those types: decimal and date columns reach the file through the codegen path, whose buffers are plain integers. ALP on a DECIMAL column is the first -- its `exceptions` channel is stored with the source column's own type, which is also where from_outputs recovers the original type. In-memory round-trips passed throughout; only write-then-read was broken. make_fixed_width_column covers numeric, chrono and fixed-point alike. The leaf tag yields scale 0 for a decimal, but nothing between here and the end of decompress reads the scale -- the buffer is the mantissa either way -- and apply_stored_dtype restores the real scale, which rides on the column record. Tests: .hpln round-trips for ALP on DECIMAL64 both with and without exceptions (a zero-exception column stores that channel empty, so both shapes matter), and for `factor -> bitpack`, whose `divisors` boundary channel has to survive describe/rebuild. The latter passed before this fix; it is here so the operator has file-path coverage rather than only in-memory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…lans
Re-explored every TPC-H sf1000 decimal column against the new operator set
(`--score pareto`, the policy this plan file documents: max ratio subject to a
decode floor). Exactly one column changes.
l_quantity 4.885x -> 10.383x decode 2918 -> 2938 GB/s
Its mantissas are every multiple of 100 in [100, 5000] -- 50 distinct values
that bitpack alone spends 13 bits on -- so dividing out the common factor first
halves the bit width, and with fewer bits to unpack decode comes out marginally
ahead. Compress drops 1367 -> 697 GB/s, a one-time cost against a 2.13x
standing size reduction.
Everything else is left alone deliberately:
* o_totalprice, l_extendedprice, l_discount, l_tax, ps_supplycost, c_acctbal
and s_acctbal all re-explore to exactly their committed plans. They are
already at their minimum bit width -- no common factor, nothing for `factor`
or ALP to exploit -- and a sweep of for/delta/zigzag/rle/ans loses to plain
bitpack on both ratio and speed on every one of them.
* ALP never wins, including on l_quantity, where it reaches a hair more ratio
(10.462x vs 10.383x) but decodes 2.1x slower. Enabling it for decimals was
still worth doing -- it is what let the explorer make that comparison rather
than us assert it -- but the frontier picks `factor`.
* The explorer's own l_quantity pick was a depth-4 variant adding two rle
levels on chunk_bits: 10.382x at decode 2595 GB/s. Taken at the knee
instead, matching the hand-picked-knee precedent already in this file.
* l_orderkey has a `delta -> ans` point at 20.7x, and p_retailprice a
`delta -> lz4` point at 90.3x. Both are large, both cost 3-4x decode, and
neither involves the new operators -- they were reachable before this work.
Out of scope here; worth revisiting on their own merits.
Measurements are sf1000 lineitem part.0 (100M rows) on GB300, round-trip
verified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…er drops it `factor` measures ~0.999x on ANY input when applied alone -- the quotients keep the source width and `divisors` adds bytes -- so it only pays off one level down, once bitpack sees the narrowed values. The BFS accommodates that with a blanket waiver: preprocessing ops are exempt from the "didn't shrink, discard" rule. But byte size then cannot distinguish a useful application from a useless one, so on every numeric column with no common divisor -- most of them -- `factor` consumed a GPU trial AND a top-ratio beam slot, only to be revealed worthless two levels later. It also meant `factor` reached the depth where it pays off purely by surviving that waiver, so on a column with many competing preprocessing candidates it could be pruned before its payoff was ever measured. The operator already computes the answer: its `divisors` channel is the per-chunk GCD, so an all-ones channel means every chunk divided by 1 and the transform is a bit-exact identity. `try_operator` now checks that and reports `no_benefit` on the trial; the BFS skips such a candidate, narrowing the waiver to ops that actually transformed something. Deliberately NOT done by failing the operator, which would have needed no explorer change at all: that would make `factor` a data-dependent hard failure, so a hand-written plan naming it would abort on any partition whose GCD happened to be 1. The sf1000 plans run across 60 lineitem parts; a plan validated on one must not die on another. `no_benefit` is advisory -- the operator still encodes correctly and an explicit plan is unaffected. The check is host-side (this TU is built by the host compiler, not nvcc) over a channel of one element per 1024-row chunk -- ~780 KB for a 100M-row column, far cheaper than the encode that just produced it. Note the trial itself still runs; what is saved is the beam slot and every expansion beneath it. Skipping the trial too would need a standalone GCD pre-pass that avoids allocating and writing the full-size quotients column -- a further win, not taken here. Verified on sf10 lineitem: l_quantity (GCD 100) still selects `factor -> bitpack` at 10.383x, l_extendedprice (GCD 1) now prunes `factor` and falls back to plain bitpack at 2.655x, matching its committed plan. Also fixes a latent hazard in test_operator_sweep exposed by adding a fixture: build_work() indexes fixtures positionally through kFixtureNames, which is a separate list from build_fixtures(). Adding to one and not the other ran every later fixture's chains against the wrong column while still reporting "passed", and dropped the last fixture off the work list entirely. Both lists now carry the new i64_scaled fixture, and their correspondence is asserted rather than left to a comment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ouse density The preceding commits over-commented: much of what landed was rationale that belongs in a commit message, not next to the code. Measured against each file's existing density, the additions ran 2-4x heavy -- alp_compressor.cu at 33% against a 15.7% baseline, compression_explorer.cpp at 32% against 12.5%, test_utils.hpp at 17% against 8.5%. Removed the narrative: motivation paragraphs, before/after measurements, and restatements of what the code plainly does. Kept the invariants a maintainer would otherwise have to rediscover -- never scale by a rounded reciprocal, the full-warp mask precondition on the shuffle reductions, why the fill value is the minimum encoded value, why `factor` reports rather than fails, and why make_fixed_width_column is required over make_numeric_column. The JIT renderers are left near their existing density on purpose: a banner per emit_* is the established style there (37% file-wide), so emit_factor keeps its banner -- only the paragraph motivating the operator came out, leaving the mechanism and channel wiring that emit_for's banner also documents. The sf1000 l_quantity plan entry gets the same treatment: 49 of the 54 blocks in that plan set carry exactly two comment lines and one carries three, so an eight-line justification was the sole outlier. Reduced to the existing "hand-picked knee" note form; the before/after numbers live in the commit that made the change. No behaviour change; 19/19 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hread Decode ran one element per thread and re-read `metadata[i / 1024]` for every element, so each thread issued a single 8-byte load and store with no other memory work in flight. Mapping a block to a vector instead makes the scale block-uniform (one read, not one per element) and gives each thread four elements, so four loads are outstanding at once. It also matches the encode kernel's block-per-vector mapping. Measured on TPC-H sf10 lineitem, GB300, idle GPU, three interleaved passes with variance under 0.5%: decompress f64 `alp` decimal `alp -> bitpack` baseline 941.7 GB/s 1212.0 GB/s metadata hoist only 943.2 (+0.2%) 1223.8 (+1.0%) this commit 950.9 (+1.0%) 1388.3 (+14.5%) The metadata hoist alone is noise -- the load was already an L1 broadcast. The win is the elements per thread, and decimal gains far more than f64 because its decode is a single integer multiply and so is memory-bound, where extra memory-level parallelism pays; f64's fma plus two constant-memory lookups is latency-bound and barely moves. From the vortex comparison, which decodes 32 elements per thread through shared memory. Not taken here: their shared-memory staging (which exists to let them apply patches in-place before a coalesced write-out) and their one-warp blocks, which would cap occupancy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a new operator to simpatico (technically a feature, but it improves performance), that detects a common divisor so it can be applied before bitpack. This allows bitpack to function much more effectively on l_quantity, which has a common divisor of 100:
In addition, this PR enables ALP to operate on decimal data and has various improvements to its implementation. However, it is still not chosen over bitpack for any of the input columns. There may be additional improvements possible.
Checklist
References