Skip to content

Commit c1bc826

Browse files
marius-bughiuclaude
andcommitted
docs: address Copilot review on #337
- CHANGELOG: condense the five CompressedIntSet bullets to two. The originals ran well past this repo's "brief and user-facing" convention, and an over-long section is a real release risk because release.yml lifts the whole version section into the GitHub Release body. - CompressedIntSetBenchmark: the header claimed the sweep matched the headline workload "at ten times the scale". It is a tenth of it — 100k items over a 10M universe against the 1M/100M the claim is stated at. Reworded to say so, and to say what is actually preserved: the 100x universe ratio, so a chunk still lands in an array container exactly as it does at full scale. - Two clarifying comments while in here: why the Add arm counts Optimize() against itself, and why the two counting queries probe rather than merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 8ce217b commit c1bc826

3 files changed

Lines changed: 14 additions & 8 deletions

File tree

CHANGELOG.md

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,8 @@ All notable changes to Celerity are documented here. This project follows [Keep
66

77
### Added
88

9-
- **`CompressedIntSet`** in `Celerity.Collections` — an exact, compressed set of 32-bit integers, filling the huge-and-sparse hole that `BitSet` (dense, bounded), `SparseSet` (small universe) and `IntSet` (hash) leave open. Each 65,536-value chunk is stored as a sorted `ushort[]`, a 1024-word bitmap, or run-length pairs, whichever is smallest; set algebra runs inside a chunk and skips a chunk neither side populates with a single comparison, so cost tracks populated chunks rather than elements. Memory is roughly 10x below `HashSet<int>` when sparse and far lower when dense or clustered, and enumeration is in ascending signed order. Implements `ISet<int>` and `IReadOnlySet<int>`, plus `AddRange`, `Optimize`, `IntersectCount`, `Cardinality` and `MemoryUsageInBytes`. Two caveats stated up front in the docs: point `Contains` is still faster in a hash table, and there is **no portable Roaring format** (Celerity ships no serializers), so this is an in-process structure, not Lucene / Druid / Spark interop. Closes [#310](https://github.com/marius-bughiu/Celerity/issues/310).
10-
- **`CompressedIntSetTests`, `CompressedIntSetEnumerationTests`, `CompressedIntSetSetAlgebraTests` and `CompressedIntSetDifferentialTests`** — dedicated coverage of the container transitions, the 32-bit extremes, ascending enumeration and `Count`'s `OverflowException`, every binary operation across all nine container-form pairs in both operand orders, and a CsCheck property test plus a `Celerity.Fuzz` target driving the container state machine against a `HashSet<int>` oracle. Closes [#310](https://github.com/marius-bughiu/Celerity/issues/310).
11-
- `CompressedIntSet` joined the cross-collection suites — `SetAlgebraTests`, `SetAlgebraDifferentialTests`, `SetIEnumerableConstructorTests`, `SetExplicitICollectionMemberTests` and `ClearNoOpVersionTests` — and the Native AOT smoke test. Closes [#310](https://github.com/marius-bughiu/Celerity/issues/310).
12-
- **`CompressedIntSetBenchmark`** (registered in the CI-tracked suite) and its **CompressedIntSet** dashboard card, against a `HashSet<int>` baseline with `[MemoryDiagnoser]` on so the `Add` row shows the footprint difference. The three `Intersect` arms sweep sparse / dense / clustered key distributions, because which container form a chunk lands in follows entirely from the shape of the data. Closes [#310](https://github.com/marius-bughiu/Celerity/issues/310).
13-
- Docs for `CompressedIntSet`: an API-reference section leading with the two caveats, plus the README collections list, decision-table row and usage example. Closes [#310](https://github.com/marius-bughiu/Celerity/issues/310).
9+
- **`CompressedIntSet`** in `Celerity.Collections` — an exact, compressed set of 32-bit integers for the huge-and-sparse shape `BitSet`, `SparseSet` and `IntSet` do not serve. Each 65,536-value chunk is stored as a sorted array, a bitmap, or run-length pairs, whichever is smallest, so set algebra works chunk-at-a-time instead of one hash probe per element: at 1M values over a 100M universe it intersects ~9x faster and unions ~11x faster than `HashSet<int>`, in ~9x less memory. Implements `ISet<int>` and `IReadOnlySet<int>`, plus `AddRange`, `Optimize`, `IntersectCount`, `Cardinality` and `MemoryUsageInBytes`; enumeration is in ascending order. There is **no portable Roaring format** — Celerity ships no serializers — so this is an in-process structure, not Lucene / Druid / Spark interop. Closes [#310](https://github.com/marius-bughiu/Celerity/issues/310).
10+
- `CompressedIntSetBenchmark` in the CI-tracked suite and the matching **CompressedIntSet** dashboard card, plus API-reference and README docs, dedicated and cross-collection tests, a `Celerity.Fuzz` target, and Native AOT smoke coverage. Closes [#310](https://github.com/marius-bughiu/Celerity/issues/310).
1411
- **`RankSelectBitVector`** in `Celerity.Collections` — an immutable succinct index over a dense bit vector that answers `Rank(i)` (set bits below a position) in `O(1)` and `Select(k)` (position of the `k`-th set bit) in `O(log n)`, filling a BCL gap: .NET ships no rank or select anywhere, so the alternative is a hand-rolled `O(i/64)` popcount loop. Builds from a `BitSet`, packed `ulong[]`, or a list of set positions; `Rank0`, `TrySelect`, `IndexSizeInBytes`, and `ToBitSet` round out the surface. The index costs 25% over the bits and is **build-once** — any mutation requires an `O(n/64)` rebuild, so a vector that keeps changing should stay a `BitSet`. Closes [#312](https://github.com/marius-bughiu/Celerity/issues/312).
1512
- **`RankSelectBitVectorTests` and `RankSelectBitVectorDifferentialTests`** — dedicated coverage of the constructors, bounds, and the block / superblock boundaries, plus a CsCheck property test and a `Celerity.Fuzz` target reconciling every rank position and select ordinal against the naive `bool[]` oracle. Closes [#312](https://github.com/marius-bughiu/Celerity/issues/312).
1613
- **`RankSelectBitVectorBenchmark`** (registered in the CI-tracked suite) and its **RankSelectBitVector** dashboard card — the baseline arm is the hand-rolled `ulong[]` popcount loop, and the three `Rank` arms sweep the query position so the `O(index / 64)` gap is visible. A `Build` row keeps the index-construction cost honest. Closes [#312](https://github.com/marius-bughiu/Celerity/issues/312).

src/Celerity.Benchmarks/CompressedIntSetBenchmark.cs

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,10 @@
1616
// garbage. That column is half the reason to use this type.
1717
//
1818
// The sweep stays at the family's 1,000 / 100,000 item counts so the card sits alongside the other
19-
// set cards on the dashboard and the CI A/B run stays affordable. The type's headline claim is
20-
// stated for ~1M elements over a ~100M universe, which is the Sparse shape here at ten times the
21-
// scale — the mechanism, and the ratio, are the same.
19+
// set cards on the dashboard and the CI A/B run stays affordable. That is a *tenth* of the scale
20+
// the type's headline claim is stated at (~1M elements over a ~100M universe) — but the same shape,
21+
// because the Sparse arms keep the 100x universe ratio, so a chunk still holds a few hundred values
22+
// and still lands in an array container. The full-scale measurements are recorded in ROADMAP.md.
2223
//
2324
// The distribution is encoded in the category name rather than in a second [Params] on purpose: the
2425
// dashboard's benchmark-name parser accepts exactly one `(ItemCount: N)` suffix, and a second
@@ -118,6 +119,9 @@ public int HashSet_Add()
118119
return set.Count;
119120
}
120121

122+
// The Optimize() call is deliberate and is counted against this arm: it is what a caller does
123+
// after a bulk load, and it is what makes the Allocated column report the settled footprint
124+
// rather than an un-compacted one. It costs this arm time to make the memory number honest.
121125
[Benchmark]
122126
[BenchmarkCategory("Add")]
123127
public int CompressedIntSet_Add()

src/Celerity/Collections/CompressedIntSet.cs

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1309,6 +1309,11 @@ private static int ContainerIntersectCount(in Chunk a, in Chunk b)
13091309
return count;
13101310
}
13111311

1312+
// The two counting queries below probe the larger side rather than merging the two sorted
1313+
// cursors the way the mutating operators do. That is deliberate: probing lets Overlaps bail
1314+
// on the first hit (a merge would have to keep stepping the smaller side), and neither query
1315+
// writes a result, so the merge's one advantage — producing the output in order for free —
1316+
// buys nothing here.
13121317
int total = 0;
13131318
if (a.Cardinality <= b.Cardinality)
13141319
{

0 commit comments

Comments
 (0)