Merge pull request #168 from marius-bughiu/fix/issue-27-bulk-ctor-resize

perf(collections): size bulk constructors for load factor so they never resize (#27)

Author: marius-bughiu

CHANGELOG.md

@@ -113,6 +113,12 @@ All notable changes to Celerity are documented here. This project follows [Keep
 
 - All remaining `xUnit2013` warnings in `Celerity.Tests` are gone: `Assert.Equal(0, x.Count)` / `Assert.Equal(1, x.Count)` against a Celerity collection now use the xUnit-idiomatic `Assert.Empty(x)` / `Assert.Single(x)`, which produce better failure messages (`"The collection was expected to contain a single element, but it was empty"` instead of `"Expected: 1 / Actual: 0"`) and are also a strictly stronger assertion — `Assert.Single` enumerates and counts, so a collection whose `Count` property and enumerator disagree would now fail at this site instead of silently passing. The replacement is mechanical and spans 10 files: `RemoveOutValueTests.cs` (44 sites), `AddAndTryAddTests.cs` (28), `LongDictionaryTests.cs` / `IntDictionaryTests.cs` / `CelerityDictionaryTests.cs` / `CelerityDictionaryCollisionTests.cs` (8 each), `IEnumerableConstructorTests.cs` (6), `ReadOnlyDictionaryInterfaceTests.cs` (4), and `LongDictionaryCollisionTests.cs` / `IntDictionaryCollisionTests.cs` (2 each) — exactly the file list and call counts the weekly automated code-review sweep flagged. A clean `dotnet build --no-incremental` now produces zero `xUnit2013` warnings, so genuinely new warnings on a touched test file aren't drowned in 64 pre-existing ones. No test was added, removed, or weakened, and the full `Celerity.Tests` suite (818 tests) still passes locally on `net8.0`. Closes #123.
 
+### Fixed
+
+- Bulk `IEnumerable<…>`-source constructors now size the backing table to hold the whole source **without resizing**, honouring the documented "the source's `Count` is used to size the backing storage so inserts do not resize" contract that was previously not met (issue #27 — *Optimize resize operations* / *Reduce allocations in critical paths*). The resize threshold is `size × loadFactor`, so a table sized to the raw source count still tripped a full rehash-and-copy on the last inserts of the bulk fill (e.g. a 100-pair source at the default `0.75` load factor sized to `128`, threshold `96`, and rehashed into `256` near the end). The fix scales the requested capacity up by `1 / loadFactor` in each collection's `InitialCapacityForSource` helper, so a known-count source fits below the threshold and the bulk build is a single allocation with one hash per entry and zero rehashes. Applied uniformly to all seven hash-table collections — `IntDictionary`, `LongDictionary`, `CelerityDictionary`, `IntSet`, `LongSet`, `CeleritySet`, and `CelerityMultiMap` (distinct-key fills; duplicate-heavy sources simply leave slack, never resize). The plain `(capacity, loadFactor)` constructor is unchanged — `capacity` still rounds to the next power of two as documented — and `SmallDictionary` is exempt (no load factor, no hasher: its capacity-verbatim sizing already fills without resizing). The null-source-before-loadFactor-validation ordering (issue #94) is preserved: the new sizing math runs only after the `ArgumentNullException.ThrowIfNull(source)` check and is skipped for an out-of-range load factor, which the primary constructor still rejects with `ArgumentOutOfRangeException`. A non-collection source (unknown count) falls through to the plain capacity exactly as before.
+- `BulkConstructorNoResizeTests` — a new cross-collection shared regression suite pinning the fix the same way `TryAddProbeCountTests` pins the single-probe contract: by counting `IHashProvider<T>.Hash` calls during construction. Building a table of `N = 100` distinct keys with no resize costs exactly `N` hash calls (one probe-chain walk per insert); a resize re-hashes every entry already placed, so a mid-build resize would push the count above `N`. Each of the seven hash-table collections gets a from-collection no-resize fact, plus a load-factor-scaling theory (`0.25` / `0.5` / `0.75` / `0.95`) asserting the headroom tracks a non-default load factor, and a non-collection-enumerable fallback case asserting correctness when the count is unknown. `SmallDictionary` is intentionally excluded (no hasher to count). The assertion fails on the pre-fix sizing and passes after it. Full `Celerity.Tests` suite is green at 1978 tests on `net8.0` (10 new).
+- `MemoryAllocationBenchmark` extended with a `FromCollection` category — `Dictionary<int, int>`, `IntDictionary<int>`, and `CelerityDictionary<int, int, Int32WangNaiveHasher>` each built from a known-count `KeyValuePair<int, int>[]` source via the `IEnumerable` constructor, with the full `MemoryDiagnoser` columns. It sits alongside the existing `Grow` / `Presized` cases and makes the saved rehash-and-copy allocation visible. Part of the extended (weekly / on-demand) suite published to the `dev/bench-extended` dashboard; no `Program.cs` change needed (`MemoryAllocationBenchmark` is already registered).
+
 ## [1.4.0] - 2026-05-31
 
 ### Added

docs/performance.md

@@ -66,10 +66,10 @@ for (int i = 0; i < 1000; i++) d[i] = Work(i);
 
 Account for the load factor when sizing: at the default `0.75`, a `capacity` of 1024 resizes once you pass 768 entries. To hold 1,000 entries without any resize, you need `capacity` ≥ ~1365 (which rounds to 2048), or a higher load factor.
 
-**Building from an existing collection sizes itself.** The `IEnumerable<KeyValuePair<TKey, TValue>>` constructor reads `ICollection<T>.Count` when the source exposes it and pre-sizes the backing storage, so bulk-filling from a `List<>`, array, or BCL `Dictionary<,>` avoids resize work for free:
+**Building from an existing collection sizes itself.** The `IEnumerable<KeyValuePair<TKey, TValue>>` constructor reads `ICollection<T>.Count` when the source exposes it and pre-sizes the backing storage — **including the load-factor headroom** — so bulk-filling from a `List<>`, array, or BCL `Dictionary<,>` avoids resize work for free. Unlike the plain `capacity` constructor (where you account for the load factor yourself, per the note above), the source constructor scales the count up by `1 / loadFactor` for you, so the whole source lands below the resize threshold in a single allocation:
 
 ```csharp
-var fast = new IntDictionary<string>(existingList);   // pre-sized from Count
+var fast = new IntDictionary<string>(existingList);   // pre-sized from Count, no resize
 ```
 
 For a non-`ICollection` enumerable (a LINQ query, a generator), `Count` is unknown — pass an explicit `capacity` if you can estimate the size.

src/Celerity.Benchmarks/MemoryAllocationBenchmark.cs

@@ -19,19 +19,35 @@
 /// </list>
 /// The gap between the two rows is exactly what a caller saves by passing a
 /// capacity to the constructor.
+///
+/// A third <c>FromCollection</c> case builds each dictionary from a known-count
+/// <see cref="ICollection{T}"/> source via the <c>IEnumerable</c> constructor.
+/// That constructor sizes the backing store from the source's <c>Count</c>; the
+/// fix for issue #27 adds the load-factor headroom so the whole source fits below
+/// the resize threshold, eliminating the one rehash-and-copy a count-sized table
+/// would otherwise pay near the end of the bulk fill.
 /// </remarks>
 [MemoryDiagnoser]
 [CategoriesColumn]
 [GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]
 public class MemoryAllocationBenchmark
 {
     private int[] keys = null!;
+    private KeyValuePair<int, int>[] pairs = null!;
 
     [Params(100_000)]
     public int ItemCount;
 
     [GlobalSetup]
-    public void Setup() => keys = KeyDistributions.Int32(Distribution.Uniform, ItemCount);
+    public void Setup()
+    {
+        keys = KeyDistributions.Int32(Distribution.Uniform, ItemCount);
+        // Distinct sequential keys: the IEnumerable constructor uses Add, which
+        // rejects duplicates, so the source must be collision-free.
+        pairs = new KeyValuePair<int, int>[ItemCount];
+        for (int i = 0; i < ItemCount; i++)
+            pairs[i] = new KeyValuePair<int, int>(i, i);
+    }
 
     // ── Grow from default capacity (resize churn included) ──────────────────────
 
@@ -108,4 +124,19 @@ public CelerityDictionary<int, int, Int32WangNaiveHasher> CelerityDictionary_Pre
         }
         return map;
     }
+
+    // ── Bulk-built from a known-count collection (issue #27 sizing fix) ──────────
+
+    [Benchmark]
+    [BenchmarkCategory("FromCollection")]
+    public Dictionary<int, int> Dictionary_FromCollection() => new Dictionary<int, int>(pairs);
+
+    [Benchmark]
+    [BenchmarkCategory("FromCollection")]
+    public IntDictionary<int> IntDictionary_FromCollection() => new IntDictionary<int>(pairs);
+
+    [Benchmark]
+    [BenchmarkCategory("FromCollection")]
+    public CelerityDictionary<int, int, Int32WangNaiveHasher> CelerityDictionary_FromCollection()
+        => new CelerityDictionary<int, int, Int32WangNaiveHasher>(pairs);
 }

src/Celerity.Tests/Collections/BulkConstructorNoResizeTests.cs

@@ -0,0 +1,222 @@
+using Celerity.Collections;
+using Celerity.Hashing;
+
+namespace Celerity.Tests.Collections;
+
+/// <summary>
+/// Regression tests for the bulk-constructor sizing fix (issue #27). The
+/// <c>IEnumerable&lt;…&gt;</c> source constructors of every hash-table collection
+/// document that the source's <c>Count</c> is used to size the backing storage
+/// "so inserts do not resize" (or "so the initial fill avoids resize work").
+/// Before the fix the table was sized to the raw count, but the resize threshold
+/// is <c>size × loadFactor</c>, so the last few inserts of the bulk fill still
+/// tripped a full rehash. The fix scales the requested capacity up by
+/// <c>1 / loadFactor</c> so the whole source fits below the threshold.
+///
+/// These tests pin the contract the same way <see cref="TryAddProbeCountTests"/>
+/// does: by counting <see cref="IHashProvider{T}.Hash"/> calls. Building a table
+/// of <c>N</c> distinct keys with no resize costs exactly <c>N</c> hash calls
+/// (one probe-chain walk per insert). A resize re-hashes every entry already in
+/// the table, so a resize during construction would push the count above <c>N</c>.
+/// Asserting <c>== N</c> therefore fails on the pre-fix sizing and passes after it.
+///
+/// <c>SmallDictionary</c> is intentionally excluded: it has no load factor and no
+/// hasher (it linear-scans), so its capacity-verbatim sizing already fills without
+/// resizing and there is nothing to count.
+/// </summary>
+public class BulkConstructorNoResizeTests
+{
+    private const int N = 100;
+
+    private static int _hashCallCount;
+
+    private struct CountingIntHasher : IHashProvider<int>
+    {
+        public int Hash(int key)
+        {
+            _hashCallCount++;
+            unchecked
+            {
+                uint x = (uint)key;
+                x = ((x >> 16) ^ x) * 0x45d9f3b;
+                x = ((x >> 16) ^ x) * 0x45d9f3b;
+                x = (x >> 16) ^ x;
+                return (int)x;
+            }
+        }
+    }
+
+    private struct CountingLongHasher : IHashProvider<long>
+    {
+        public int Hash(long key)
+        {
+            _hashCallCount++;
+            unchecked
+            {
+                ulong x = (ulong)key;
+                x = (x ^ (x >> 30)) * 0xbf58476d1ce4e5b9UL;
+                x = (x ^ (x >> 27)) * 0x94d049bb133111ebUL;
+                x = x ^ (x >> 31);
+                return (int)x;
+            }
+        }
+    }
+
+    private struct CountingStringHasher : IHashProvider<string>
+    {
+        public int Hash(string key)
+        {
+            _hashCallCount++;
+            return key.GetHashCode();
+        }
+    }
+
+    private static KeyValuePair<int, int>[] IntPairs() =>
+        Enumerable.Range(1, N).Select(i => new KeyValuePair<int, int>(i, i * 10)).ToArray();
+
+    private static KeyValuePair<long, int>[] LongPairs() =>
+        Enumerable.Range(1, N).Select(i => new KeyValuePair<long, int>(i, i * 10)).ToArray();
+
+    private static KeyValuePair<string, int>[] StringPairs() =>
+        Enumerable.Range(1, N).Select(i => new KeyValuePair<string, int>($"k{i}", i)).ToArray();
+
+    // ── Dictionaries ────────────────────────────────────────────────────────────
+
+    [Fact]
+    public void IntDictionary_BulkConstruct_FromKnownCount_DoesNotResize()
+    {
+        KeyValuePair<int, int>[] src = IntPairs();
+        _hashCallCount = 0;
+
+        var map = new IntDictionary<int, CountingIntHasher>(src);
+
+        Assert.Equal(N, _hashCallCount);
+        Assert.Equal(N, map.Count);
+        for (int i = 1; i <= N; i++)
+            Assert.Equal(i * 10, map[i]);
+    }
+
+    [Fact]
+    public void LongDictionary_BulkConstruct_FromKnownCount_DoesNotResize()
+    {
+        KeyValuePair<long, int>[] src = LongPairs();
+        _hashCallCount = 0;
+
+        var map = new LongDictionary<int, CountingLongHasher>(src);
+
+        Assert.Equal(N, _hashCallCount);
+        Assert.Equal(N, map.Count);
+        for (int i = 1; i <= N; i++)
+            Assert.Equal(i * 10, map[i]);
+    }
+
+    [Fact]
+    public void CelerityDictionary_BulkConstruct_FromKnownCount_DoesNotResize()
+    {
+        KeyValuePair<string, int>[] src = StringPairs();
+        _hashCallCount = 0;
+
+        var map = new CelerityDictionary<string, int, CountingStringHasher>(src);
+
+        Assert.Equal(N, _hashCallCount);
+        Assert.Equal(N, map.Count);
+        for (int i = 1; i <= N; i++)
+            Assert.Equal(i, map[$"k{i}"]);
+    }
+
+    // ── Sets ──────────────────────────────────────────────────────────────────────
+
+    [Fact]
+    public void IntSet_BulkConstruct_FromKnownCount_DoesNotResize()
+    {
+        int[] src = Enumerable.Range(1, N).ToArray();
+        _hashCallCount = 0;
+
+        var set = new IntSet<CountingIntHasher>(src);
+
+        Assert.Equal(N, _hashCallCount);
+        Assert.Equal(N, set.Count);
+        for (int i = 1; i <= N; i++)
+            Assert.True(set.Contains(i));
+    }
+
+    [Fact]
+    public void LongSet_BulkConstruct_FromKnownCount_DoesNotResize()
+    {
+        long[] src = Enumerable.Range(1, N).Select(i => (long)i).ToArray();
+        _hashCallCount = 0;
+
+        var set = new LongSet<CountingLongHasher>(src);
+
+        Assert.Equal(N, _hashCallCount);
+        Assert.Equal(N, set.Count);
+        for (int i = 1; i <= N; i++)
+            Assert.True(set.Contains(i));
+    }
+
+    [Fact]
+    public void CeleritySet_BulkConstruct_FromKnownCount_DoesNotResize()
+    {
+        string[] src = Enumerable.Range(1, N).Select(i => $"v{i}").ToArray();
+        _hashCallCount = 0;
+
+        var set = new CeleritySet<string, CountingStringHasher>(src);
+
+        Assert.Equal(N, _hashCallCount);
+        Assert.Equal(N, set.Count);
+        for (int i = 1; i <= N; i++)
+            Assert.True(set.Contains($"v{i}"));
+    }
+
+    // ── MultiMap ────────────────────────────────────────────────────────────────
+
+    [Fact]
+    public void CelerityMultiMap_BulkConstruct_FromKnownDistinctCount_DoesNotResize()
+    {
+        KeyValuePair<int, int>[] src = IntPairs();
+        _hashCallCount = 0;
+
+        var map = new CelerityMultiMap<int, int, CountingIntHasher>(src);
+
+        Assert.Equal(N, _hashCallCount);
+        Assert.Equal(N, map.Count);
+        for (int i = 1; i <= N; i++)
+            Assert.Equal(new[] { i * 10 }, map[i].ToArray());
+    }
+
+    // ── Load-factor scaling: the headroom must track a non-default load factor ────
+
+    [Theory]
+    [InlineData(0.25f)]
+    [InlineData(0.5f)]
+    [InlineData(0.75f)]
+    [InlineData(0.95f)]
+    public void IntDictionary_BulkConstruct_HoldsSourceWithoutResize_AcrossLoadFactors(float loadFactor)
+    {
+        KeyValuePair<int, int>[] src = IntPairs();
+        _hashCallCount = 0;
+
+        var map = new IntDictionary<int, CountingIntHasher>(src, loadFactor: loadFactor);
+
+        Assert.Equal(N, _hashCallCount);
+        Assert.Equal(N, map.Count);
+        for (int i = 1; i <= N; i++)
+            Assert.Equal(i * 10, map[i]);
+    }
+
+    // ── Fallback: a non-collection source (unknown count) still builds correctly ──
+    // (No resize guarantee is possible without a count; only correctness is pinned.)
+
+    [Fact]
+    public void IntDictionary_BulkConstruct_FromNonCollectionEnumerable_IsCorrect()
+    {
+        IEnumerable<KeyValuePair<int, int>> src =
+            Enumerable.Range(1, N).Select(i => new KeyValuePair<int, int>(i, i * 10));
+
+        var map = new IntDictionary<int, CountingIntHasher>(src);
+
+        Assert.Equal(N, map.Count);
+        for (int i = 1; i <= N; i++)
+            Assert.Equal(i * 10, map[i]);
+    }
+}

src/Celerity/Collections/CelerityDictionary.cs

@@ -86,8 +86,9 @@ public CelerityDictionary(
     /// <c>Count</c> is used to size the backing storage so inserts do not resize.
     /// </param>
     /// <param name="capacity">
-    /// The minimum initial capacity. The final capacity is the larger of this
-    /// value and the source's count, rounded up to the next power of two.
+    /// The minimum initial capacity, rounded up to the next power of two. When
+    /// the source's count is larger, the backing store is sized — including
+    /// load-factor headroom — to hold the whole source without resizing.
     /// </param>
     /// <param name="loadFactor">
     /// The fraction of the dictionary's size that can be filled before resizing.
@@ -102,7 +103,7 @@ public CelerityDictionary(
         IEnumerable<KeyValuePair<TKey, TValue>> source,
         int capacity = DEFAULT_CAPACITY,
         float loadFactor = DEFAULT_LOAD_FACTOR)
-        : this(InitialCapacityForSource(source, capacity), loadFactor)
+        : this(InitialCapacityForSource(source, capacity, loadFactor), loadFactor)
     {
         foreach (KeyValuePair<TKey, TValue> kvp in source)
         {
@@ -116,10 +117,28 @@ public CelerityDictionary(
     // when the user also passed an invalid loadFactor.
     private static int InitialCapacityForSource(
         IEnumerable<KeyValuePair<TKey, TValue>> source,
-        int capacity)
+        int capacity,
+        float loadFactor)
     {
         ArgumentNullException.ThrowIfNull(source);
-        return Math.Max(capacity, (source as ICollection<KeyValuePair<TKey, TValue>>)?.Count ?? 0);
+        int count = (source as ICollection<KeyValuePair<TKey, TValue>>)?.Count ?? 0;
+
+        // Size for the source count *including* load-factor headroom: the resize
+        // threshold is size*loadFactor, so a table sized to the raw count would
+        // still rehash on the last inserts of the bulk fill. Scaling the count up
+        // by 1/loadFactor makes the "Count is used to size the backing storage so
+        // inserts do not resize" contract actually hold (issue #27). A
+        // non-collection source (count 0) or an out-of-range loadFactor — left for
+        // the primary ctor to reject, so null-source-beats-bad-loadFactor ordering
+        // is preserved — falls through to the plain capacity.
+        if (count > 0 && loadFactor > 0f && loadFactor < 1f)
+        {
+            int withHeadroom = (int)Math.Ceiling(count / (double)loadFactor);
+            if (withHeadroom > count)
+                count = withHeadroom;
+        }
+
+        return Math.Max(capacity, count);
     }
 
     /// <summary>