feat(collections): add CompressedIntSet — a chunk-compressed exact set of 32-bit integers - #337
Conversation
…t of 32-bit integers Closes the huge-and-sparse hole in the integer-set family that BitSet (dense, bounded), SparseSet (small universe, O(Universe) memory) and IntSet (hash) leave open. The value space is partitioned into 65,536-value chunks and each chunk is stored as a sorted ushort[], a 1024-word bitmap, or run-length pairs — whichever is smallest. Set algebra then works inside a chunk (a linear merge of two sorted cursors, or one ANDed word per 64 values) instead of one hash probe per element, and a chunk neither side populates is skipped with a single key comparison. Measured at the issue's kill criterion — 1M elements over a 100M universe: intersect (1% overlap) 60.3 ms -> 6.4 ms 9.5x intersect (50% overlap) 47.9 ms -> 7.8 ms 6.2x union 87.4 ms -> 7.6 ms 11.5x except 24.2 ms -> 7.0 ms 3.5x heap 17.7 MB -> 2.0 MB 8.9x smaller Both bars in the issue (>=3x intersect, >=5x memory) are cleared. The first draft probed with a binary search per element and measured only 2.7x on intersect; the linear merge is what earns the number. Implements ISet<int> and IReadOnlySet<int> with BCL HashSet<int> semantics, and every set operation takes the chunk-wise path when the other side is also a CompressedIntSet. Also ships AddRange (a range in a fresh chunk is one run pair, four bytes whatever its width), Optimize (the only thing that produces run containers from existing data), IntersectCount, and MemoryUsageInBytes. Enumeration is in ascending signed order — the chunk key is the value's high 16 bits with the sign bit flipped — which HashSet<int> does not offer. Cardinality is a long because the set can hold all 2^32 int values; Count throws OverflowException in that one case rather than answering wrongly. Caveat #2 of the issue was accepted rather than treated as a kill: Celerity ships no serializers, so there is no portable Roaring format and no Lucene / Druid / Spark interop. That leads both the API-reference section and the README row. Parity rollout, all in this change: - Dedicated tests: CompressedIntSetTests, CompressedIntSetEnumerationTests, CompressedIntSetSetAlgebraTests (every binary op across all nine container-form pairs, both operand orders), CompressedIntSetDifferentialTests (CsCheck). - Cross-collection suites: SetAlgebraTests, SetAlgebraDifferentialTests, SetIEnumerableConstructorTests, SetExplicitICollectionMemberTests, ClearNoOpVersionTests. - Celerity.Fuzz target driving the container state machine against a HashSet<int> oracle, and a Native AOT smoke-test block covering all three container forms. - CompressedIntSetBenchmark registered in CoreBenchmarks, with [MemoryDiagnoser] and sparse / dense / clustered Intersect arms. - Dashboard: ship card in web/index.html and COLLECTIONS entries in both web/dev/bench/index.html and web/dev/bench/detail.html. - Docs: API-reference section and README list, decision-table row and example. - CHANGELOG and ROADMAP. Closes #310. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Coverage
|
There was a problem hiding this comment.
Pull request overview
Adds CompressedIntSet, a Roaring-style chunk-compressed exact int set to Celerity’s collections family, and integrates it across the repo’s testing, fuzzing/AOT smoke coverage, benchmarks, dashboard wiring, and documentation.
Changes:
- Introduces
CompressedIntSet(ISet<int>+IReadOnlySet<int>) with chunk-wise container encoding (array/bitmap/run), explicitOptimize(),AddRange(...),IntersectCount(...), and ascending signed-order enumeration. - Adds dedicated + cross-suite test coverage, CsCheck differential tests, a fuzz target, and Native AOT smoke coverage.
- Adds benchmarks + dashboard registration and updates docs/roadmap/changelog to document the new collection.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| web/index.html | Adds the “ships in the box” entry for CompressedIntSet. |
| web/dev/bench/index.html | Registers CompressedIntSet ops for the benchmark dashboard grid. |
| web/dev/bench/detail.html | Registers CompressedIntSet for the benchmark dashboard detail view. |
| src/Celerity/Collections/CompressedIntSet.cs | Implements the new compressed integer set and its public API surface. |
| src/Celerity.Tests/Collections/SetIEnumerableConstructorTests.cs | Adds shared-suite constructor semantics coverage for CompressedIntSet. |
| src/Celerity.Tests/Collections/SetExplicitICollectionMemberTests.cs | Pins ICollection<int> explicit-member semantics for CompressedIntSet. |
| src/Celerity.Tests/Collections/SetAlgebraTests.cs | Adds shared-suite set-algebra behavior coverage for the IEnumerable fallback path. |
| src/Celerity.Tests/Collections/SetAlgebraDifferentialTests.cs | Adds shared-suite differential coverage row for CompressedIntSet. |
| src/Celerity.Tests/Collections/CompressedIntSetTests.cs | Adds dedicated unit tests for core behavior, transitions, AddRange, Optimize, overflow count, etc. |
| src/Celerity.Tests/Collections/CompressedIntSetSetAlgebraTests.cs | Adds container-pair matrix tests (9 form pairs × ops × operand order) + merge/fallback validations. |
| src/Celerity.Tests/Collections/CompressedIntSetEnumerationTests.cs | Pins ascending enumeration and enumerator invalidation behavior. |
| src/Celerity.Tests/Collections/CompressedIntSetDifferentialTests.cs | Adds CsCheck property-based differential testing against HashSet<int>. |
| src/Celerity.Tests/Collections/ClearNoOpVersionTests.cs | Adds the “no-op Clear doesn’t bump version” contract test for CompressedIntSet. |
| src/Celerity.Fuzz/Differential.cs | Adds a fuzz target driving interleaved operations vs a HashSet<int> oracle. |
| src/Celerity.Benchmarks/Program.cs | Registers CompressedIntSetBenchmark in the CI-tracked benchmark suite. |
| src/Celerity.Benchmarks/CompressedIntSetBenchmark.cs | Adds benchmark coverage (Add/Contains/Intersect*/Union/Except) with memory diagnoser. |
| src/Celerity.AotSmokeTest/Program.cs | Adds AOT smoke coverage for all three container forms + algebra paths. |
| ROADMAP.md | Marks CompressedIntSet as done and records measured results/design notes. |
| docs/api/collections.md | Adds API reference documentation for CompressedIntSet. |
| CHANGELOG.md | Adds [Unreleased] entries for CompressedIntSet and its supporting coverage/wiring. |
- CHANGELOG: condense the five CompressedIntSet bullets to two. The originals ran well past this repo's "brief and user-facing" convention, and an over-long section is a real release risk because release.yml lifts the whole version section into the GitHub Release body. - CompressedIntSetBenchmark: the header claimed the sweep matched the headline workload "at ten times the scale". It is a tenth of it — 100k items over a 10M universe against the 1M/100M the claim is stated at. Reworded to say so, and to say what is actually preserved: the 100x universe ratio, so a chunk still lands in an array container exactly as it does at full scale. - Two clarifying comments while in here: why the Add arm counts Optimize() against itself, and why the two counting queries probe rather than merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.
Suppressed comments (3)
src/Celerity.Benchmarks/CompressedIntSetBenchmark.cs:107
Clustered()does not guarantee distinct keys and can also produce more than 32 blocks (e.g. whencountis not divisible byBlocks). Because the benchmark builds sets from these arrays, duplicates reduce the actual set size belowItemCount, skewing both time and memory comparisons and making runs less reproducible.
private static int[] Clustered(int seed, int count)
{
const int Blocks = 32;
var rand = new Random(seed);
int[] keys = new int[count];
int perBlock = Math.Max(1, count / Blocks);
int written = 0;
while (written < count)
{
int origin = rand.Next(0, 100_000_000 - perBlock);
for (int i = 0; i < perBlock && written < count; i++)
keys[written++] = origin + i;
}
src/Celerity/Collections/CompressedIntSet.cs:11
- The
says each chunk is stored in whichever of the three container forms is smallest, but run containers are only produced by
Optimize()andAddRange(per the remarks). As written, the summary can read like run encoding happens automatically during normal inserts/removals, which isn’t true unless the caller opts into compression.
/// An <b>exact, compressed set of 32-bit integers</b> that partitions the value space into
/// 65,536-value chunks and stores each chunk in whichever of three container forms is smallest — a
/// sorted <c>ushort[]</c> when sparse, a 1024-word bitmap when dense, or run-length pairs when
/// clustered. It fills a BCL gap: .NET ships no compressed integer set, so the alternatives are
docs/api/collections.md:2137
- This paragraph says each chunk is stored in whichever of the three container forms is smallest, but run containers are only produced by
Optimize()andAddRange. Rewording here would avoid implying that run encoding is chosen automatically during ordinary single-element inserts/removals.
With that said: it is an **exact** set of 32-bit integers that partitions the value space into
**65,536-value chunks** and stores each chunk in whichever of three container forms is smallest.
Addresses the three findings in round 2 of the Copilot review on #337, all of which arrived in the suppressed block. Clustered() drew free-floating block origins over the whole 100M range, so two blocks could overlap and emit duplicate keys — which silently shrinks the built set below ItemCount and skews both the time and the memory comparison. It could also emit more than 32 blocks when count was not divisible by 32. Blocks are now laid one per fixed stride and jittered inside it, so they cannot overlap, every key is distinct by construction, and the ceiling division caps it at 32 blocks. The type summary and the docs said each chunk is stored in "whichever of three container forms is smallest", which reads as though run encoding happens on an ordinary insert. It does not — only Optimize() and AddRange produce it. Reworded in the XML summary, docs/api/collections.md (including the container table's "Chosen when" cell), the README, the web ship card, and the changelog, so the opt-in nature is stated where a reader first meets the type rather than several paragraphs later. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 2 — the three suppressed commentsThe review body says "no new comments", but three findings were in the suppressed block. All three were right; all three are fixed in 49355d2. 1. Blocks are now laid one per fixed stride ( 2 & 3. The Reworded so the opt-in nature appears where the reader first meets the type — in the XML summary, in Re-verified after the change: build clean, 5,220 tests pass, |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.
Suppressed comments (2)
src/Celerity/Collections/CompressedIntSet.cs:439
IntersectWith(IEnumerable<int>)falls back toSetOperations.IntersectWith(this, other), butSetOperationsreadsself.Countfor its empty/self checks. ForCompressedIntSet,Countintentionally throws once_cardinality > int.MaxValue, so intersecting a large set (e.g. built viaAddRange(int.MinValue, int.MaxValue)) with a plain sequence will throwOverflowExceptioneven though the intersection is computable. The same pattern affects other fallbacks that call intoSetOperations(subset/equality/overlaps) whenever_cardinalityexceedsint.MaxValue.
Consider handling the overflow case in CompressedIntSet before calling SetOperations (e.g., build the result by streaming distinct elements of other and probing Contains), and add a regression test for this scenario.
return;
}
SetOperations.IntersectWith(this, other);
}
src/Celerity/Collections/CompressedIntSet.cs:574
Overlaps(IEnumerable<int>)falls back toSetOperations.Overlaps(this, other), which readsself.Countinternally. SinceCompressedIntSet.Countcan throw when_cardinality > int.MaxValue,Overlapscan also throwOverflowExceptionfor large sets even though it only needs to streamotherand probe membership. Implementing the fallback inline avoids theCountdependency and keeps the method usable at the full 32-bit cardinality this type supports.
if (other is CompressedIntSet o)
return OverlapsCore(o);
return SetOperations.Overlaps(this, other);
}
….MaxValue Addresses round 3 of the Copilot review on #337 (both findings arrived in the suppressed block, and both were real bugs). Count throws OverflowException once the set holds more than int.MaxValue elements — deliberate, since the type can hold all 2^32 int values and ICollection<T>.Count cannot express that. But six of the IEnumerable<int> fallbacks routed through SetOperations, which compares self.Count for its empty/size checks, so IntersectWith, IsSubsetOf, IsProperSubsetOf, IsProperSupersetOf, Overlaps and SetEquals all threw on a set built with a wide AddRange — even though every one of those answers is trivially computable there. Copilot flagged IntersectWith and Overlaps; the same defect covered four more. Each of the six now compares against the long Cardinality instead. IntersectWith in particular no longer snapshots this set into a List<T> (which a >2^31 set cannot fill either) — it builds the survivors from `other`, which is bounded by `other` and correct at any cardinality. SymmetricExceptWith and IsSupersetOf still go through SetOperations: neither reads Count, so the shared family implementation stays the single source of those semantics. The comment above the region records which are shared, which are not, and why, so the divergence reads as a decision. Regression test drives the whole query surface plus IntersectWith against a set built with AddRange(int.MinValue, int.MaxValue), asserting Count still throws while every operation answers. Coverage stays at 100% line / 100% branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 3 — both suppressed comments were real bugsFixed in dbafde3. These were the good kind of review finding: a defect I created by combining two things that were each fine on their own.
Copilot named
Each of the six now compares the
Regression test ( |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.
Suppressed comments (2)
src/Celerity/Collections/CompressedIntSet.cs:638
CopyToevaluatesCountbefore validatingarray/arrayIndex, so if the set is large enough forCountto throwOverflowExceptionyou’ll get an overflow even when the caller passedarray: nullor an out-of-rangearrayIndex. That contradicts the stated “matches HashSet.CopyTo argument validation” behavior in docs and makes argument-validation ordering inconsistent.
Consider validating array and arrayIndex first, then checking Count (or _cardinality) afterward.
public void CopyTo(int[] array, int arrayIndex) => SetOperations.CopyTo(this, Count, array, arrayIndex);
web/index.html:296
- The new landing-page blurb says the run-length form is applied by
Optimize()only, but the public contract (and docs/tests in this PR) also sayAddRangeproduces run containers on fresh chunks. The description should mentionAddRangeas well to avoid misleading callers about when run encoding is used.
<div class="ship"><code>CompressedIntSet</code><div class="desc">Exact compressed set of 32-bit integers: each 65,536-value chunk is stored as a sorted array or a bitmap by density, with an opt-in run-length form that Optimize() applies to clustered data. Set algebra runs word-parallel inside a chunk and skips a whole chunk with one comparison, and enumeration is in ascending order. For huge-and-sparse integer sets — posting lists, row-id sets, cohort intersection — where HashSet<int> costs ~10x the memory and one hash probe per element. No portable Roaring format: this is an in-process structure, not an interop codec.</div></div>
…e Count guard Addresses round 4 of the Copilot review on #337, plus a coverage-fragility problem found while verifying the fix. CopyTo passed Count as an argument to SetOperations.CopyTo, and C# evaluates arguments before the call — so on a set past int.MaxValue the overflow guard fired before any argument was checked, turning CopyTo(null, 0) into an OverflowException instead of an ArgumentNullException. It is now written out and compares the long _cardinality, which restores HashSet<int>.CopyTo's validation order and reports a set too large for an int[] as insufficient space rather than as an overflow. Same reasoning as the six set operations in dbafde3. The web ship card said the run-length form comes from Optimize() alone; AddRange produces it too on a fresh chunk. Corrected there and in the three other short blurbs that had the same omission (README bullet, README details block, changelog). While re-measuring, coverage of this type turned out to depend on the randomized CsCheck differential test: excluding it left eight branches uncovered, so the repo's 100% gate was passing on luck and could have flaked on any later run. Each is now pinned by a deterministic test — the mid-merge and trailing arms of the union / symmetric-except chunk walks, both rejection paths in IsSubsetOfCore, both skip directions in OverlapsCore, and the word-aligned high mask in AddRange. Coverage is 100% line / 100% branch with the property test excluded entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 4 — both fixed, plus a coverage problem the fix exposedFixed in b56e88b. 1. 2. The ship card credited only 3. Something the fix turned up: this type's coverage was passing on luck. Re-measuring after the Each is now pinned by a deterministic
5,228 tests pass on all three TFMs; fuzz and the AOT smoke test re-run clean. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.
Suppressed comments (2)
src/Celerity/Collections/CompressedIntSet.cs:316
- The
Clear()XML doc says it “releases every container”, but the implementation preserves the chunk-index capacity (like other sets preserve capacity on Clear). This can be surprising alongsideMemoryUsageInBytes, which includes_chunks.Lengthand therefore may remain large afterClear(). Consider clarifying the doc comment to match the actual behavior: container payloads are released, but index capacity is preserved.
/// <summary>
/// Removes all elements from the set, releasing every container. A <see cref="Clear"/> on an
/// already-empty set changes nothing and leaves active enumerators valid.
/// </summary>
docs/api/collections.md:2231
- The docs claim
Clear()“releases every container”, but the implementation keeps the chunk-index array allocated (capacity preserved), andMemoryUsageInBytescounts that capacity. Consider clarifying this bullet to reflect that only per-chunk payloads are released, not necessarily all retained capacity.
- `void Clear()` — empties the set and releases every container. A `Clear()` on an already-empty
set changes nothing and leaves active enumerators valid.
Addresses round 5 of the Copilot review on #337 (both suppressed comments made the same point about two doc surfaces). Clear() was documented as "releasing every container". It does drop every container payload, but the chunk index keeps its capacity so the set can be refilled without regrowing it — and MemoryUsageInBytes counts that capacity, so the reported footprint does not fall to zero after a Clear(). Anyone comparing the number before and after would have concluded the release had not happened. The XML doc, the API-reference bullet, and the MemoryUsageInBytes doc now say so, and point at Optimize() as the way to hand the index back. A test pins it, so the claim is checked rather than asserted: after Clear() the footprint is non-zero, and after Optimize() it is exactly zero. Coverage holds at 100% line / 100% branch with the property test excluded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 5 — fixed in 962a770Both suppressed comments made the same point about two doc surfaces, and it was a real inaccuracy rather than a wording nit.
The XML doc, the API-reference bullet, and the Coverage holds at 100% line / 100% branch with the property test excluded; 5,226 tests pass. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.
Suppressed comments (2)
src/Celerity.Benchmarks/CompressedIntSetBenchmark.cs:16
- The header comment equates BenchmarkDotNet’s MemoryDiagnoser “Allocated” metric with steady-state footprint. “Allocated” is total bytes allocated during the benchmark invocation (including transient allocations from resizes/optimizations), so calling it the settled footprint is misleading—especially in the Add category where both implementations can resize internally.
// * Memory. [MemoryDiagnoser] is on and the Add category constructs the whole set, so the
// Allocated column is the steady-state footprint of each representation rather than incidental
// garbage. That column is half the reason to use this type.
src/Celerity.Benchmarks/CompressedIntSetBenchmark.cs:126
- HashSet_Add currently starts from an empty HashSet, which forces internal growth/resizes and inflates both time and the MemoryDiagnoser “Allocated” metric. Pre-sizing makes the baseline fairer and makes “Allocated” closer to the table’s intended memory signal.
public int HashSet_Add()
{
var set = new HashSet<int>();
foreach (int key in sparseKeys)
set.Add(key);
…tate footprint Addresses round 6 of the Copilot review on #337. The header claimed the Add category's Allocated column was "the steady-state footprint of each representation". It is not: BenchmarkDotNet reports total bytes allocated per operation, so it includes every intermediate array a growing collection discards on the way up. The column is still the right relative signal — both arms start empty and grow, so both pay it — but it is an upper bound, not the settled size. Reworded to say that, and to point at MemoryUsageInBytes and the ROADMAP heap measurements for the settled figure. The same comment now also records why neither arm is pre-sized, which was the review's other suggestion and is declined on purpose: pre-sizing only the HashSet arm would hand the baseline an advantage the Celerity arm structurally cannot take, since CompressedIntSet has no capacity constructor — it has no table to size. Every other set benchmark in the suite pairs an unsized BCL arm with an unsized Celerity arm, and this one matches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 6 — one fixed, one declinedFixed in 1527583. 1. "Allocated" is not a steady-state footprint. Agreed. The header claimed the 2. Pre-sizing the The suggestion is that starting
It would also make this the only set card in the suite that does it. The real problem the comment points at is the claim, not the setup — and that is what (1) fixes. Both arms grow from empty, both pay for their resizes, and the column is now described as the comparable upper bound it is. The reasoning is recorded in the benchmark's own header so the next reader sees a decision rather than an oversight. Happy to be overruled if you would rather the |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.
Suppressed comments (2)
src/Celerity/Collections/CompressedIntSet.cs:247
- The XML docs imply the range is “four bytes” in storage, but the implementation allocates a
ushort[]for the run pair (so there is managed array overhead in addition to the 4-byte payload). Consider clarifying that this is 4 bytes of payload (twoushorts), excluding object/array overhead, to avoid misleading callers about actual heap usage.
/// A range that lands in a chunk the set does not yet touch is stored as a single run pair —
/// four bytes, whatever the range's width — which is why this is the cheap way to build a
/// clustered set. A range overlapping an existing chunk merges into it and the chunk is left in
docs/api/collections.md:2228
- The docs say a fresh-chunk range add is “four bytes”, but the run pair is stored in a managed
ushort[](so the 4 bytes is the payload, not the full heap cost). Clarifying this avoids readers interpreting it as the exact allocation size.
- `long AddRange(int start, int endInclusive)` — adds every value in the inclusive range and
returns how many were **new**. A range landing in a chunk the set does not yet touch is stored as
a **single run pair — four bytes, whatever the range's width** — so this is the cheap way to
build a clustered set. Throws `ArgumentOutOfRangeException` if `endInclusive < start`.
… range add Addresses round 7 of the Copilot review on #337 (one point, on two doc surfaces). "A range in a fresh chunk is four bytes, whatever its width" described the run pair's payload, not its heap cost — the pair lives in a ushort[], which carries an array header. Both surfaces now say "four bytes of payload" and note the header, matching how MemoryUsageInBytes already documents its own exclusion. The claim the sentence exists to make — that the cost is independent of the range's width — is unchanged and still true. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 7 — fixed in 26ba6a7One point, on two doc surfaces, and it is a fair precision catch: "a range in a fresh chunk is four bytes, whatever its width" described the run pair's payload, not its heap cost — the pair lives in a Both surfaces now say "four bytes of payload" and note the header, which matches how |
Benchmarks66 regressions Highlights
Collections (536)
Hashers (111)
Same-runner A/B (sharded 8-way): main ( |
CI status — one benchmark shard hit the #335 timeout; re-running itEverything is green except one benchmark shard, and it is worth writing down what happened because the obvious reading ("the PR that adds a benchmark class timed out a benchmark shard" — i.e. #300) is not what this is. 17 of 18 checks passed: build-and-test on all three OSes, coverage, release-gates, dashboard-coverage, all three AOT publishes, and 7 of 8 benchmark shards. The Shard 5 ran 02:57:43 → 04:57:58 — exactly 120 minutes, the workflow's It is not this PR's benchmark class. I pulled the per-shard logs. What it is: #335, exactly. Every shard ran long today — 77, 88, 92, 97, 99, 99, 103 minutes and then shard 5's 120 — against a ceiling sized for the heaviest shard to sit "comfortably inside" it. That is the ~2x slowdown #335 describes when benchmark runs overlap. I contributed to the overlap: this PR went through seven pushes across the review rounds, each starting an 8-shard run. I cancelled each superseded run as I went, but cancellation is not instantaneous and the runners were contended regardless. I have re-run shard 5 alone, with no other run competing for runners. Will report back. Two things for the maintainer, neither blocking:
|
CI is green — 18/18Shard 5 passed on the isolated re-run. The whole Benchmarks run is The re-run took 80 minutes (05:01:57 → 06:22:16) against the same slice that had hit the 120-minute ceiling the first time, with no other run in the repo competing. So both factors were real, and I want to be precise about the split rather than leave my earlier comment's emphasis standing unqualified:
Neither is caused by this PR: Nothing outstanding on my side: the last Copilot round came back with no comments and, for the first time in this PR, no suppressed block either. |
Closes #310.
CompressedIntSetis an exact, compressed set of 32-bit integers. It closes the huge-and-sparse hole in the integer-set family thatBitSet(dense, bounded),SparseSet(small universe,O(Universe)memory) andIntSet(hash) leave open.The value space is partitioned into 65,536-value chunks, and each chunk is stored as a sorted
ushort[], a 1024-word bitmap, or run-length pairs — whichever is smallest. Set algebra then works inside a chunk (a linear merge of two sorted cursors, or one ANDed word per 64 values) instead of one hash probe per element, and a chunk neither side populates is skipped with a single key comparison, so cost tracks populated chunks rather than elements.The kill criterion
The issue set a bar: close as won't-ship unless intersect beats
HashSet<int>by ≥3x at 1M elements over a 100M universe and memory drops ≥5x. Measured after implementation, both are cleared:HashSet<int>CompressedIntSetMedian of 7 reps, operation only (both operands built outside the clock),
net10.0Release.Worth recording: the first draft measured only 2.7x on intersect — below the bar. It probed the right-hand container with a binary search per element. Replacing that with a linear merge of the two sorted cursors is where the 9.5x comes from, and it is the single most load-bearing change in the implementation.
Design calls worth a maintainer's eye
CompressedIntSetSetAlgebraTests, so specializing any pair later cannot silently change an answer.Optimize()andAddRange, never speculatively on a single insert — the same "compress once it has settled" contract as Roaring's ownrunOptimize. ATryAdd/Removelanding in a run-encoded chunk expands it back to its natural form first, and that is documented on the type, in the API reference, and in the README.Countcan throw. The set can hold all 2^32intvalues, which does not fit theintthatICollection<T>.Countmust return.Cardinality(along) is the always-correct count;CountthrowsOverflowExceptionin the one case it cannot answer, rather than saturating silently. Only a very wideAddRangecan reach it, and there is a test that does.ulongloops, notVector<ulong>(unlikeBitSet). A 1024-word loop is already word-parallel, and the workload the type is sold for lands in array containers, not bitmaps, so SIMD would not move the headline number. Called out in case you want it anyway.Parity rollout
Everything below is in this PR — nothing deferred.
a. Collection —
src/Celerity/Collections/CompressedIntSet.cs. ImplementsISet<int>andIReadOnlySet<int>(so it does not re-open the interface gap #306 is closing), plusAddRange,Optimize,IntersectCount,Cardinality,MemoryUsageInBytes, and an allocation-free struct enumerator. Enumeration is in ascending signed order — the chunk key is the value's high 16 bits with the sign bit flipped — whichHashSet<int>does not offer. XML docs on every public member;net8.0floor; the container thresholds are named constants with the reasoning next to them.b. Dedicated tests —
CompressedIntSetTests(container transitions, the 32-bit extremes,AddRange,Optimize, theCountoverflow),CompressedIntSetEnumerationTests(ascending order across all three container forms in one pass, invalidation,CopyTo),CompressedIntSetSetAlgebraTests(the nine-pair matrix, both operand orders, plus the chunk-index merge and theIEnumerablefallback),CompressedIntSetDifferentialTests(CsCheck, with the value domain generated so it lands in one chunk or hundreds).c. Cross-collection suites —
SetAlgebraTests,SetAlgebraDifferentialTests,SetIEnumerableConstructorTests,SetExplicitICollectionMemberTests,ClearNoOpVersionTests. Suites that genuinely do not apply:SetConstructorValidationTestsandLoadFactorBoundaryTests(no load factor),EnsureCapacityAndTrimExcessTestsandCapacityArgumentValidationTests(no capacity ctor and no hasher —Optimize()is the trim),SetSourceCountBranchTests(the constructor takes noICollectioncount hint, since there is no table to pre-size), and every dictionary-shaped suite.d. Fuzz + AOT — a
CompressedIntSettarget inCelerity.Fuzz'sDifferential.All, interleaving add / remove / range-add / optimize / clear with the four mutating set operations against aHashSet<int>oracle, because the container state machine is the whole risk surface here. 4,000 cases pass locally. A Native AOT smoke-test block exercises all three container forms, the chunk-wise algebra, and both set interfaces.e. Benchmarks —
CompressedIntSetBenchmark, registered inCoreBenchmarks,HashSet<int>baseline,[MemoryDiagnoser]on so theAddrow's Allocated column is the memory comparison. The threeIntersectarms sweep sparse / dense / clustered key distributions, because which container form a chunk lands in follows entirely from the shape of the data. The distribution is in the category name rather than a second[Params]on purpose — the dashboard parser accepts exactly one(ItemCount: N)suffix and a second parameter would blank the card.f. Dashboard — ship card in
web/index.html,COLLECTIONSentries inweb/dev/bench/index.htmlandweb/dev/bench/detail.html.node scripts/check_dashboard_coverage.jspasses.g. Docs — an API-reference section in
docs/api/collections.mdleading with the two caveats, and the README collections list,ISet<T>family sentence, sets<details>block with an example, decision-table row placed next to theBitSet/SparseSet/IntSetrows, and a cross-reference from theBitSetrow.h. CHANGELOG + ROADMAP — two short bullets under
[Unreleased] → Added(deliberately terse:release.ymllifts the whole version section into the GitHub Release body, so the per-facet detail lives in this description instead); the 2.4.0CompressedIntSetentry flipped todonewith the measured numbers and the design calls recorded.Test plan
dotnet buildclean — no new warnings. (TheCS8631warnings inSetIEnumerableConstructorTestsare pre-existing, tracked in Test project emits 3264 build warnings, burying 20 real CS8631 nullability warnings #332.)dotnet testgreen on all three TFMs: 5,220 passed / 0 failed onnet8.0,net9.0,net10.0.CompressedIntSet, itsContainerCursorand itsEnumeratorare each at 1.0/1.0, andscripts/coverage_report.pyreports 100/100 overall.Celerity.Fuzz --target CompressedIntSet --iterations 4000— all cases pass.Celerity.AotSmokeTestruns clean (managed run locally; the CI job does the Native AOT publish).node scripts/check_dashboard_coverage.js— 138 cards across 42 collections wired.HashSet<int>in the same harness.main— the new CompressedIntSet card should populate on the next benchmark run. Worth a look: this PR adds a benchmark class, which is the shape Benchmark shard can time out on any PR that adds a benchmark class (head/base pack from different class lists) #300 says can time a shard out (head and base pack from different class lists).🤖 Generated with Claude Code