feat(collections): add RankSelectBitVector — O(1) Rank and O(log n) Select over a dense bit vector - #329
Merged
Conversation
…elect Adds the succinct rank/select layer over a dense bit vector, the primitive the compressed-integer lane composes on. The BCL has no counterpart at any level: BitArray exposes neither rank nor select, BitOperations.PopCount is per-word, and .NET 8/9/10 ship no succinct data-structure support — so the baseline is the O(index / 64) popcount loop a caller writes by hand. The index is two arrays: an int per 256-bit superblock plus a byte per 64-bit word holding the within-superblock prefix. Rank is two index loads and one masked POPCNT; Select binary-searches the superblocks, walks the at most four words inside one, and resolves within the word by popcount narrowing. The issue estimated 3% space overhead, which the layout it specified cannot deliver: a byte-wide per-word counter caps the superblock at 256 bits (the within-superblock prefix must stay under 256), which puts the index at 25% of the vector — the same price as the classic rank9 layout. IndexSizeInBytes reports it per instance and every doc surface leads with the real number. The type is immutable by construction and the build-once contract opens the XML docs, the API reference, and the README entry: a mutating vector should stay a BitSet and be snapshotted only once the bits have settled. Full parity rollout in this commit: dedicated + CsCheck differential tests (100% line and branch on the new file), a Celerity.Fuzz target against the bool[] oracle, an AOT smoke-test block, a benchmark registered in the CI-tracked suite with the hand-rolled loop as its baseline and the query position swept early/mid/late, dashboard cards in all three HTML surfaces, API reference and README sections, CHANGELOG entries, and the ROADMAP status. The cross-collection shared suites do not apply: they pin the hash-table family contract (Add/TryAdd, load factor, indexer return type, IEnumerable constructors), and none of them cover BitSet, FenwickTree, or SparseSet either. Closes #312 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Coverage
|
Also names the superblock-count expression, mirroring the existing WordCount helper, so the two index-array sizings read the same way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Benchmarks4 regressions Highlights
Collections (508)
Hashers (111)
Same-runner A/B (sharded 8-way): main ( |
Owner
Author
|
Note on the benchmark comment above: the 71 flagged regressions are runner noise, not this PR. The tell is that a large share of them are the BCL baseline arms — types this PR does not touch and cannot affect:
A This is the mechanism #300 describes: a PR that adds a benchmark class makes head and base pack into different shards, so the two sides of every A/B row are measured on differently-loaded runners. This PR adds The new rows themselves are worth reading, because they land the claim cleanly at 1M bits:
|
…elect-bit-vector Only CHANGELOG.md conflicted, and both sides were pure additions to [Unreleased] > Added — this branch's three RankSelectBitVector entries and main's six #311 span-lookup entries. Kept both. Everything else auto-merged: the dashboard COLLECTIONS arrays, the benchmark registry, the fuzz target list and the AOT smoke test each took one new entry from either side. main's new scripts/check_dashboard_coverage.js confirms the result — 131 cards across 41 collections, RankSelectBitVector's card included.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #312.
Adds
RankSelectBitVector— the succinct rank/select layer over a dense bit vector, and the primitive the compressed-integer lane (CompressedIntSet, #310) composes on.Rank(i)— set bits belowi, inO(1).Select(k)— position of thek-th set bit, inO(log n).There is no BCL counterpart at any level:
BitArrayexposes neither,BitOperations.PopCountis per-word, and .NET 8/9/10 ship no succinct data-structure support. The honest baseline is theO(index / 64)popcount loop a caller writes by hand, and that loop is what the benchmark measures against.Design
The index is two arrays: an
intper 256-bit superblock holding the set bits before it, plus abyteper 64-bit word holding the set bits before that word within its superblock.Rankis two index loads and one maskedPOPCNT;Selectbinary-searches the superblock array, walks the at most four words inside the chosen superblock, and resolves within that word by a six-step popcount narrowing.The type is immutable by construction — no mutating member — and the build-once contract opens the XML docs, the API reference section, and the README entry, per the issue's instruction to lead with it. A vector that keeps changing should stay a
BitSetand be snapshotted only once the bits have settled;ToBitSet()is the documented way back.One deliberate deviation from the issue
The issue specifies "a
uintper 512-bit superblock plus abyteper 64-bit block … at roughly 3% space overhead". That layout is not self-consistent: within a 512-bit superblock the per-word prefix reaches 448, which does not fit in a byte. Keeping the byte-wide per-word counter caps the superblock at 256 bits, and the resulting index is 25% of the vector — the same price as the classic rank9 layout, and the standard cost of an exact rank that touches a single word. I kept the access pattern the issue asked for (two loads, one masked popcount) and corrected the number everywhere it appears:IndexSizeInBytesreports it per instance, and the XML docs, API reference, README, dashboard card, and ROADMAP entry all state 25% rather than 3%.Bounds behaviour, per the issue's "pick one and document it":
Rank(Length) == Count;SelectthrowsArgumentOutOfRangeExceptionoutside[0, Count), withTrySelect(rank, out position)as the non-throwing form returning-1.Parity rollout
Everything below is in this PR — nothing deferred.
src/Celerity/Collections/RankSelectBitVector.cs; twointernalmembers added toBitSet(a word-array snapshot and a take-ownership ctor) so the two types convert without a per-bit round tripRankSelectBitVectorTests.cs— all three constructors and their validation,Rank/Rank0/Select/TrySelect/Get/ToBitSet/IndexSizeInBytes, the 63/64/65 and 255/256/257 and 511/512/513 boundaries, single bit at position 0 and atLength - 1, and the empty / all-zero / all-ones vectorsRankSelectBitVectorDifferentialTests.cs— CsCheck over random length × density, reconciling every rank position and every select ordinal against a naivebool[]oracle, plus theSelect(Rank(i))round-trip identity and constructor agreementRankSelectBitVectortarget registered inDifferential.All, samebool[]oracleCelerity.AotSmokeTest/Program.cscovering all three ctors, both rank forms, both select forms, andToBitSetRankSelectBitVectorBenchmark.cs, registered inProgram.cs'sCoreBenchmarks. Baseline is the hand-rolledulong[]popcount loop (Array_*,Baseline = true), swept over query position (RankEarly/RankMid/RankLate) plusSelectandBuild;[Params]is namedItemCountso the dashboard parser picks it upCOLLECTIONSentry inweb/dev/bench/index.htmlandweb/dev/bench/detail.html(keys anditemsagree, asscripts/check_dashboard_coverage.json #327 will require), plus a ship card inweb/index.htmldocs/api/collections.md— full section afterBitSetwith a "when not to use this" block first;README.md— collections list, the bit-level details section with a runnable example, and a new "Choosing a collection" row adjacent to and cross-referencingBitSet[Unreleased] → Addedbullets: the collection, the test surface, the benchmark + dashboarddone, recording the 25%-not-3% correctionShared cross-collection suites do not apply.
AddAndTryAddTests,SetConstructorValidationTests,LoadFactorBoundaryTests,IndexerReturnTypeTests,IEnumerableConstructorTestsand the rest pin the hash-table family contract —Add/TryAddsemantics, load factor, probe counts, indexer return type.grepconfirms none of them coverBitSet,FenwickTree, orSparseSeteither. An immutable, read-only positional index has no rows to add there.Test plan
dotnet buildclean acrossnet8.0;net9.0;net10.0, 0 errors, no new warningsdotnet test— 4971 passed, 0 failed onnet8.0locallyCelerity.Fuzz --target RankSelectBitVector --iterations 400— all cases passCelerity.AotSmokeTestruns green (managed run locally; CI does the real Native AOT publish on linux-x64 × net8/9/10)Rankis ~16× faster at the midpoint and ~25× at the tail while staying flat with position, andBuildcosts ~3.3× a raw word-array clone — the tradeoff theBuildrow exists to exposeaot-publishon net8/9/10, and coveragemain— the new RankSelectBitVector card should populate from the first post-merge benchmark run rather than render blank🤖 Generated with Claude Code