feat(collections): add CompressedIntSet — a chunk-compressed exact set of 32-bit integers

CHANGELOG.md

@@ -6,6 +6,8 @@ All notable changes to Celerity are documented here. This project follows [Keep
 
 ### Added
 
+- **`CompressedIntSet`** in `Celerity.Collections` — an exact, compressed set of 32-bit integers for the huge-and-sparse shape `BitSet`, `SparseSet` and `IntSet` do not serve. Each 65,536-value chunk is stored as a sorted array, a bitmap, or run-length pairs, whichever is smallest, so set algebra works chunk-at-a-time instead of one hash probe per element: at 1M values over a 100M universe it intersects ~9x faster and unions ~11x faster than `HashSet<int>`, in ~9x less memory. Implements `ISet<int>` and `IReadOnlySet<int>`, plus `AddRange`, `Optimize`, `IntersectCount`, `Cardinality` and `MemoryUsageInBytes`; enumeration is in ascending order. There is **no portable Roaring format** — Celerity ships no serializers — so this is an in-process structure, not Lucene / Druid / Spark interop. Closes [#310](https://github.com/marius-bughiu/Celerity/issues/310).
+- `CompressedIntSetBenchmark` in the CI-tracked suite and the matching **CompressedIntSet** dashboard card, plus API-reference and README docs, dedicated and cross-collection tests, a `Celerity.Fuzz` target, and Native AOT smoke coverage. Closes [#310](https://github.com/marius-bughiu/Celerity/issues/310).
 - **`RankSelectBitVector`** in `Celerity.Collections` — an immutable succinct index over a dense bit vector that answers `Rank(i)` (set bits below a position) in `O(1)` and `Select(k)` (position of the `k`-th set bit) in `O(log n)`, filling a BCL gap: .NET ships no rank or select anywhere, so the alternative is a hand-rolled `O(i/64)` popcount loop. Builds from a `BitSet`, packed `ulong[]`, or a list of set positions; `Rank0`, `TrySelect`, `IndexSizeInBytes`, and `ToBitSet` round out the surface. The index costs 25% over the bits and is **build-once** — any mutation requires an `O(n/64)` rebuild, so a vector that keeps changing should stay a `BitSet`. Closes [#312](https://github.com/marius-bughiu/Celerity/issues/312).
 - **`RankSelectBitVectorTests` and `RankSelectBitVectorDifferentialTests`** — dedicated coverage of the constructors, bounds, and the block / superblock boundaries, plus a CsCheck property test and a `Celerity.Fuzz` target reconciling every rank position and select ordinal against the naive `bool[]` oracle. Closes [#312](https://github.com/marius-bughiu/Celerity/issues/312).
 - **`RankSelectBitVectorBenchmark`** (registered in the CI-tracked suite) and its **RankSelectBitVector** dashboard card — the baseline arm is the hand-rolled `ulong[]` popcount loop, and the three `Rank` arms sweep the query position so the `O(index / 64)` gap is visible. A `Build` row keeps the index-construction cost honest. Closes [#312](https://github.com/marius-bughiu/Celerity/issues/312).

ROADMAP.md

@@ -220,7 +220,7 @@ The 2026-Q3 review surveyed the shipped surface against the current .NET 8/9/10
 **The ordered / compressed integer-data lane.** Opened by the sorted-container hole — 38 collections and not one sorted map or set, with `Trie` the only ordered type, and that one string-keyed. The B-trees below close that half; the compressed-integer half is still open.
 
 - `BTreeDictionary<TKey, TValue, TComparer>` / `BTreeSet<T, TComparer>` — cache-friendly sorted containers against `SortedDictionary<,>` / `SortedSet<T>`, which are red-black trees with a pointer chase and an allocation per node. Status: `done`. Tracked in [#305](https://github.com/marius-bughiu/Celerity/issues/305).
-- `CompressedIntSet` — a Roaring-style compressed set of 32-bit integers, covering the huge-and-sparse shape that neither `BitSet` (dense, bounded) nor `SparseSet` (small universe, `O(Universe)` memory) nor `IntSet` (hash) serves. Status: `planned`.
+- `CompressedIntSet` — a Roaring-style compressed set of 32-bit integers, covering the huge-and-sparse shape that neither `BitSet` (dense, bounded) nor `SparseSet` (small universe, `O(Universe)` memory) nor `IntSet` (hash) serves. Status: `done` — the value space is partitioned into 65,536-value chunks, each stored as a sorted `ushort[]`, a 1024-word bitmap, or run-length pairs, whichever is smallest. The issue's kill criterion (≥3x faster intersect and ≥5x less memory at 1M elements over a 100M universe) was measured after implementation and cleared: **9.5x** on intersect at 1% overlap, **6.2x** at 50%, **11.5x** on union, **3.5x** on except, and **8.9x** less heap (17.7 MB → 2.0 MB). Two design calls are worth recording. First, the issue's "per-container-pair dispatch" was implemented as *two* paths per operator rather than nine — a word-parallel one for the dense bitmap⊕bitmap case and a cursor-driven one for everything else — because a run container that is only *read* must not be decompressed, and a single sorted-cursor abstraction gets that for free where nine hand-written pairs would each have had to re-derive it; the observable contract is still pinned for all nine pairs, in both operand orders, by `CompressedIntSetSetAlgebraTests`. The first draft of the cursor path probed with a binary search per element and measured only 2.7x on intersect — below the kill criterion — and was replaced with a linear merge of the two sorted cursors, which is where the 9.5x comes from; that is the single most load-bearing line of the implementation. Second, run containers are produced only by `Optimize()` and `AddRange`, never speculatively on a single insert, matching Roaring's own `runOptimize` contract. The type can hold all 2^32 `int` values, which does not fit the `int` that `ICollection<T>.Count` must return, so `Cardinality` (a `long`) is the always-correct count and `Count` throws `OverflowException` in the one case it cannot answer. Caveat #2 of the issue — no portable Roaring format, so no Lucene / Druid / Spark interop — was accepted rather than treated as a kill: the in-process memory and set-algebra win stands on its own, and it now leads both the API reference section and the README row. Tracked in [#310](https://github.com/marius-bughiu/Celerity/issues/310).
 - `RankSelectBitVector` — succinct `Rank` / `Select` over a dense bit vector, the primitive the above compose on. Status: `done` — an immutable snapshot of a `BitSet` (or packed `ulong[]`, or a list of set positions) carrying a two-level popcount index: an `int` per 256-bit superblock and a `byte` per 64-bit word, so `Rank` is two index loads and one masked `POPCNT` and `Select` is a binary search over the superblocks. The issue's estimated 3% space overhead did not survive contact with the layout it specified — a byte-wide per-word counter caps the superblock at 256 bits, which puts the index at **25%** of the vector, the same price as the classic rank9 layout; `IndexSizeInBytes` reports it per instance and the docs lead with it. The build-once contract is stated first in every doc surface, and the benchmark ships the hand-rolled popcount loop as its baseline with the query position swept early / mid / late. Tracked in [#312](https://github.com/marius-bughiu/Celerity/issues/312).
 - Sorted-span set algebra in `Celerity.Primitives` — merge-based `Intersect` / `Union` / `Except` / `IntersectCount` over already-sorted spans, where the BCL answer is LINQ or a `HashSet` round-trip. Status: `planned`.
 

docs/api/collections.md

@@ -2113,6 +2113,182 @@ for (int start = 0; start < nodeCount; start++)
 
 ---
 
+## CompressedIntSet
+
+```csharp
+public sealed class CompressedIntSet : ISet<int>, IReadOnlySet<int>
+```
+
+**Two caveats first, because they decide whether this type is for you.**
+
+1. **There is no portable serialization format.** Celerity does not ship serializers, so
+   `CompressedIntSet` cannot read or write the portable **Roaring** format. The Lucene / Druid /
+   Spark posting-list interop that is the most common reason to reach for a Roaring bitmap is
+   **not available here**. This is an in-process data structure, not an interop codec — which is
+   why it is named for what it does rather than for Roaring.
+2. **This is not the only compressed integer set in .NET.** A maintained pure-C# Roaring
+   implementation exists (`Equativ.RoaringBitmaps`). What this type offers instead of novelty is
+   integration: full mutability, verified Native AOT and trim compatibility on `net8.0` /
+   `net9.0` / `net10.0`, published benchmarks against `HashSet<int>` on the project dashboard, and
+   membership of the same [`BitSet`](#bitset) / [`SparseSet`](#sparseset) / [`IntSet`](#intset)
+   family and shared test suites.
+
+With that said: it is an **exact** set of 32-bit integers that partitions the value space into
+**65,536-value chunks** and stores each chunk in whichever of three container forms is smallest.
+
+| Container | Layout | Chosen when |
+|---|---|---|
+| **Sorted array** | `ushort[]` of offsets | the chunk is sparse (≤ 4096 values) |
+| **Bitmap** | 1024 × 64-bit words (8 KB) | the chunk is dense (> 4096 values) |
+| **Run-length** | `(start, length)` pairs | the chunk is clustered and runs cost less than either of the above |
+
+4096 is where a sorted `ushort[]` and a 1024-word bitmap both cost 8 KB, so above it the bitmap is
+never larger and answers `Contains` in `O(1)` instead of `O(log n)`.
+
+It closes a real hole in the integer-set family:
+
+| Shape | Type |
+|---|---|
+| dense and bounded | [`BitSet`](#bitset) |
+| small and bounded, cleared often | [`SparseSet`](#sparseset) |
+| unbounded, hash-probed | [`IntSet`](#intset) / `HashSet<int>` |
+| **huge and sparse, set-algebra-heavy** | **`CompressedIntSet`** |
+
+### The documented BCL-beating workload
+
+**Set algebra over large, sparse integer sets** — intersecting or unioning ~1M values drawn from a
+~100M-value space: inverted-index posting lists, bitmap analytics, column-store row-id sets, cohort
+intersection. Two mechanisms:
+
+- Inside a chunk the work is a sorted merge or a **whole-word bitmap operation** (64 values per
+  ANDed word), not one hash probe and one random memory access per element.
+- The chunk index is sorted, so an entire 65,536-value range is **skipped with a single
+  comparison** whenever one side has nothing there. Cost tracks the number of *populated chunks*,
+  not the number of elements.
+
+Memory drops roughly **10x** against `HashSet<int>` for the sparse case, and far more for dense or
+clustered data: a dense region collapses to one bit per value, a clustered one to four bytes per
+run. `MemoryUsageInBytes` reports the current footprint.
+
+`Contains` is where `HashSet<int>` still wins — a hash probe beats a binary search inside a chunk.
+If point lookups are the whole workload and memory is not a concern, stay with `IntSet` /
+`HashSet<int>`.
+
+### Enumeration order
+
+Chunk keys are the value's high 16 bits **with the sign bit flipped**, so the chunk index is sorted
+by *signed* value and the set enumerates **in ascending order** from `int.MinValue` to
+`int.MaxValue` — a guarantee `HashSet<int>` does not make. The full 32-bit range is storable,
+negatives included.
+
+### Compression is explicit
+
+Run containers are produced by `Optimize()` and by `AddRange`, never speculatively on a single
+`TryAdd`: deciding on every insert would cost more than it saves. This is the same
+"compress once it has settled" contract as Roaring's own `runOptimize`.
+
+- A single-element `TryAdd` / `Remove` landing in a run-encoded chunk **expands that chunk** back
+  to its natural form first. Call `Optimize()` again after a burst of mutation.
+- `Remove` never demotes a representation — a chunk that grew into a bitmap stays one until
+  `Optimize()` (or a bulk set operation that rewrites it) says otherwise.
+- Reading a chunk never changes it, so passing an optimized set as the *right-hand* operand of any
+  set operation leaves its representation intact.
+
+### Constructors
+
+```csharp
+CompressedIntSet()
+CompressedIntSet(IEnumerable<int> source)
+```
+
+- The default constructor allocates no chunk storage until the first value is added.
+- The `source` constructor silently deduplicates (matching BCL `HashSet<int>(IEnumerable<int>)`)
+  and throws `ArgumentNullException` when `source` is `null`. There is **no capacity and no
+  `loadFactor`** parameter — the structure has no table to pre-size.
+
+### Properties
+
+- `long Cardinality { get; }` — the number of elements. Always correct.
+- `int Count { get; }` — the same number as an `int`. **Throws `OverflowException`** when the set
+  holds more than `int.MaxValue` elements, which only a very wide `AddRange` can produce (the set
+  can hold all 2^32 `int` values). Use `Cardinality` if that is reachable for your data.
+- `long MemoryUsageInBytes { get; }` — the chunk index plus every container payload, excluding
+  object headers. A measure of how well the data compressed; watch it across `Optimize()`.
+
+### Methods
+
+- `void Add(int item)` — throws `ArgumentException` on a duplicate.
+- `bool TryAdd(int item)` — `true` if added, `false` if already present.
+- `long AddRange(int start, int endInclusive)` — adds every value in the inclusive range and
+  returns how many were **new**. A range landing in a chunk the set does not yet touch is stored as
+  a **single run pair — four bytes, whatever the range's width** — so this is the cheap way to
+  build a clustered set. Throws `ArgumentOutOfRangeException` if `endInclusive < start`.
+- `bool Contains(int item)` — `O(1)` in a bitmap chunk, `O(log n)` in an array or run chunk, and a
+  single comparison against the chunk index when nothing covers the value.
+- `bool Remove(int item)` — `true` if removed. A chunk emptied by its last removal is dropped.
+- `void Clear()` — empties the set and releases every container. A `Clear()` on an already-empty
+  set changes nothing and leaves active enumerators valid.
+- `void Optimize()` — re-encodes every chunk in its smallest form (the only thing that produces
+  run containers from existing data) and trims the chunk index and array containers to exact size.
+  Purely a representation change: no element is added or removed, and active enumerators stay
+  valid.
+- `long IntersectCount(CompressedIntSet other)` — the size of the intersection, without building
+  it. Allocation-free, and it skips a whole chunk with one key comparison wherever one side is
+  empty. Throws `ArgumentNullException` for a `null` argument.
+- `Enumerator GetEnumerator()` — allocation-free struct enumerator, ascending signed order.
+- `void CopyTo(int[] array, int arrayIndex)` — matches `HashSet<int>.CopyTo` argument validation.
+
+### Set operations (`ISet<int>` and `IReadOnlySet<int>`)
+
+The full BCL `HashSet<int>` set-algebra surface, with `HashSet<int>` semantics exactly
+(duplicate-tolerant `other`, self-aliasing `other == this`):
+
+- **Mutating:** `UnionWith`, `IntersectWith`, `ExceptWith`, `SymmetricExceptWith`.
+- **Query:** `IsSubsetOf`, `IsProperSubsetOf`, `IsSupersetOf`, `IsProperSupersetOf`, `Overlaps`, `SetEquals`.
+
+Each throws `ArgumentNullException` when `other` is `null`. **Every one of them takes the chunk-wise
+fast path when `other` is also a `CompressedIntSet`** — that is the workload the type exists for —
+and otherwise falls back to the same element-at-a-time implementation the rest of the set family
+uses, which is correct but forfeits the whole-chunk skipping. If you are intersecting two of these
+sets, keep both as `CompressedIntSet`; do not project one through LINQ first.
+
+As with the other sets, `ISet<int>.Add(int)` returns `bool` (equivalent to `TryAdd`), the concrete
+`public void Add(int)` keeps its throw-on-duplicate behaviour, and `ICollection<int>.Add(int)`
+ignores duplicates.
+
+The type is single-threaded, and any structural mutation invalidates active enumerators.
+
+### Usage example
+
+```csharp
+using Celerity.Collections;
+
+// Two inverted-index posting lists: document ids drawn from a ~100M-document corpus.
+var termA = new CompressedIntSet(PostingsFor("celerity"));
+var termB = new CompressedIntSet(PostingsFor("collections"));
+
+// The data has settled — re-encode each chunk in its smallest form.
+termA.Optimize();
+termB.Optimize();
+
+// How many documents match both? No intersection is materialized.
+long both = termA.IntersectCount(termB);
+
+// Materialize the conjunction. Chunks only one side populates are skipped by key comparison.
+termA.IntersectWith(termB);
+
+foreach (int documentId in termA) // ascending order, no allocation
+    Render(documentId);
+
+// Ranges are the cheap case: a contiguous block in a fresh chunk is one run pair.
+var recentlyIngested = new CompressedIntSet();
+recentlyIngested.AddRange(90_000_000, 99_999_999); // 10M ids
+recentlyIngested.Optimize();
+Console.WriteLine(recentlyIngested.MemoryUsageInBytes); // hundreds of bytes, not tens of megabytes
+```
+
+---
+
 ## EnumMap&lt;TEnum, TValue&gt;
 
 ```csharp

src/Celerity.AotSmokeTest/Program.cs

@@ -568,6 +568,42 @@ void Check(bool condition, string message)
     Check(reached.Count == 4 && reached.Contains(2), "SparseSet ISet<int> union within universe");
 }
 
+// CompressedIntSet — chunk-compressed 32-bit integer set. Drive every one of the three
+// container forms (sorted array, bitmap, run-length) through the AOT compiler, plus the
+// chunk-wise set algebra, the range add that produces runs, and Optimize.
+{
+    var cis = new CompressedIntSet(new[] { 5, 5, -3, 900_000, int.MinValue, int.MaxValue });
+    Check(cis.Count == 5 && cis.Cardinality == 5, "CompressedIntSet source ctor dedupe");
+    Check(cis.Contains(int.MinValue) && cis.Contains(int.MaxValue) && !cis.Contains(0),
+        "CompressedIntSet spans the whole int range");
+
+    var order = new List<int>();
+    foreach (int x in cis) order.Add(x);
+    Check(order.Count == 5 && order[0] == int.MinValue && order[4] == int.MaxValue,
+        "CompressedIntSet enumerates in ascending signed order");
+
+    // Past the array→bitmap crossover, then back down via Optimize.
+    var dense = new CompressedIntSet();
+    for (int i = 0; i < 5000; i++) dense.TryAdd(i * 2);
+    Check(dense.Count == 5000 && dense.MemoryUsageInBytes >= 8192, "CompressedIntSet bitmap promotion");
+
+    // A range add on a fresh chunk is stored as a single run pair.
+    var runs = new CompressedIntSet();
+    Check(runs.AddRange(1_000_000, 1_100_000) == 100_001, "CompressedIntSet AddRange");
+    runs.Optimize();
+    Check(runs.Count == 100_001 && runs.MemoryUsageInBytes < 1024, "CompressedIntSet run encoding");
+    Check(runs.Contains(1_050_000) && !runs.Contains(1_100_001), "CompressedIntSet run probe");
+
+    var left = new CompressedIntSet(new[] { 1, 2, 3, 900_000 });
+    var right = new CompressedIntSet(new[] { 2, 3, 4 });
+    Check(left.IntersectCount(right) == 2, "CompressedIntSet IntersectCount");
+    left.IntersectWith(right);
+    Check(left.Count == 2 && left.Contains(2) && left.Contains(3), "CompressedIntSet chunk-wise intersect");
+    ((ISet<int>)left).UnionWith(new[] { -1, 3 });
+    Check(left.Count == 3 && left.Contains(-1), "CompressedIntSet ISet<int> union");
+    Check(((IReadOnlySet<int>)left).IsSubsetOf(new[] { -1, 2, 3, 7 }), "CompressedIntSet IReadOnlySet<int>");
+}
+
 // FenwickTree — Binary Indexed Tree over a numeric sequence. This is the one collection
 // built on generic math (INumber<T>), so the static abstract interface members resolve
 // through constrained calls the AOT compiler must specialize per T — worth pinning here