diff --git a/CHANGELOG.md b/CHANGELOG.md index e5f99343..8884356e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,12 @@ All notable changes to Celerity are documented here. This project follows [Keep ### Added +- **`ISpanHashProvider`** in `Celerity.Hashing` — a `Hash(ReadOnlySpan)` contract returning exactly what `Hash(string)` returns for the same characters. All 23 built-in `String*Hasher` types implement it. Closes [#311](https://github.com/marius-bughiu/Celerity/issues/311). +- **Span-keyed lookups** — `TryGetValue` / `ContainsKey` / `Contains` now take a `ReadOnlySpan` on `FrozenCelerityDictionary`, `FrozenCeleritySet`, `CelerityDictionary`, `CeleritySet` and `Trie`, so a caller holding a slice of a buffer no longer allocates a `string` per probe. Results match the `string` overloads; an empty span means `""`, never the `null` key. Additive on all three target frameworks. Closes [#311](https://github.com/marius-bughiu/Celerity/issues/311). +- **`StringInternTable` / `StringInternTable`** in `Celerity.Collections` — canonicalizes tokens probed as spans: `GetOrAdd` returns the shared `string` and allocates only on a miss, so a parse over few distinct tokens allocates per token rather than per occurrence. Implements `IReadOnlyCollection`; not thread-safe. Closes [#311](https://github.com/marius-bughiu/Celerity/issues/311). +- Test coverage for the above: `SpanHashParityTests` (the `Hash(s) == Hash(s.AsSpan())` contract, per hasher), `SpanLookupTests`, `StringInternTableTests`, `StringInternTableDifferentialTests`, a `Celerity.Fuzz` target, and Native AOT smoke-test coverage. Closes [#311](https://github.com/marius-bughiu/Celerity/issues/311). +- `StringInternTableBenchmark` in the CI-tracked suite, plus `SpanLookup` rows on the frozen-dictionary and trie benchmarks, with the matching dashboard cards. Closes [#311](https://github.com/marius-bughiu/Celerity/issues/311). +- Docs for the new surface: `ISpanHashProvider`, span-keyed lookups and `StringInternTable` in the API reference, and matching README entries. Closes [#311](https://github.com/marius-bughiu/Celerity/issues/311). - **`StringKeyProbeBenchmark`** — the tracked suite's first `string`-keyed dictionary and set rows (lookup hit, lookup miss, set `Contains`), shown as the **String-keyed probe** card on the benchmark dashboard. Closes [#308](https://github.com/marius-bughiu/Celerity/issues/308). - **`ReferenceKeyProbeTests`** — cross-collection coverage for vacant-slot detection on the open-addressed collections with reference-type keys. Closes [#308](https://github.com/marius-bughiu/Celerity/issues/308). - **`BTreeDictionary` and `BTreeSet`** (with `BTreeDictionary` / `BTreeSet` aliases and the `DefaultComparer` struct comparer) in `Celerity.Collections` — the library's first sorted map and set, and the B-tree the BCL lacks. Up to 31 keys per node keep a lookup `log₃₂(n)` node visits deep instead of chasing the `log₂(n)` pointers a red-black tree costs, and both add the ordered surface a hash table cannot answer: `Min` / `Max`, lower / upper bound, `EnumerateRange` in `O(log n + k)`, and in-order enumeration. They win on the interleaved insert + lookup + range-scan workload and on memory, and lose slightly on a delete-dominated one. Not thread-safe. Closes [#305](https://github.com/marius-bughiu/Celerity/issues/305). diff --git a/README.md b/README.md index 5082543a..a5736a6f 100644 --- a/README.md +++ b/README.md @@ -86,6 +86,11 @@ The mutable sets (`CeleritySet`, `SwissSet`, `RobinHoodSet`, `HashCachingSet`, ` - `Trie` — ordered **prefix tree** mapping string keys to values. `GetByPrefix` lists every entry whose key starts with a prefix in `O(prefix + matches)`, and `TryGetLongestPrefix` finds the longest stored key that is a prefix of a query in `O(query)`. The trie the BCL lacks — autocomplete, longest-prefix routing, and ordered (ascending-ordinal) iteration, where a `Dictionary` has no prefix index and must scan every key and run `StartsWith`. Exact `Add` / `TryGetValue` favour a `Dictionary` (one hash vs a character walk); the trie earns its place on the prefix operations. Implements `IReadOnlyDictionary`. +**Span-keyed string lookups** + +- `StringInternTable` / `StringInternTable` — a **canonicalizing token table** probed with a `ReadOnlySpan`: `GetOrAdd` returns the one shared `string` for those characters and allocates **only on a miss**. A 10M-cell parse over 100 distinct tokens creates 100 strings instead of 10,000,000. The collection you cannot build on the pre-.NET-9 BCL — `HashSet.TryGetValue` makes you allocate the string *before* you can discover you already had it, and `string.Intern` is process-wide, never collected, and still needs a `string`. Implements `IReadOnlyCollection`. +- The same span-keyed probes ship on `FrozenCelerityDictionary`, `FrozenCeleritySet`, `CelerityDictionary`, `CeleritySet`, and `Trie` — so a tokenizer, CSV/log reader, or route dispatcher holding a slice of its input buffer never has to call `new string(span)` per lookup. Works on all three target frameworks, including the `net8.0` floor where the BCL has no equivalent (.NET 9 added `Dictionary.GetAlternateLookup`). See [span-keyed lookups](docs/api/collections.md#span-keyed-lookups). + **Sorted (ordered) collections** - `BTreeDictionary` / `BTreeDictionary` — a **sorted map backed by a B-tree**: up to **31 keys per node** in flat arrays, so a lookup visits `log₃₂(n)` nodes instead of chasing `log₂(n)` pointers — roughly **4 cache misses instead of ~20 at `n = 1M`**. Adds the ordered surface a hash table cannot answer: `Min`, `Max`, `TryGetLowerBound` / `TryGetUpperBound`, `EnumerateRange` in `O(log n + k)`, and in-order enumeration. The B-tree the BCL lacks — `SortedDictionary<,>` is a red-black tree with one heap object per entry, and `SortedList<,>` memmoves the tail on every insert. Implements `IDictionary` and `IReadOnlyDictionary`. @@ -580,6 +585,8 @@ Each type buys a different tradeoff. Find your workload below; if it isn't here, | **Incremental connectivity / connected components** — union equivalence classes and ask whether two elements are in the same group (Kruskal MST, clustering, image segmentation, undirected cycle detection, "are these accounts linked?") | `DisjointSet` | Union-find with **union by size** + **path halving**: near-`O(1)` amortized `Union` / `Find` / `Connected`, `O(α(n)) ≤ 4`. Runs a stream of merges + connectivity queries in near-linear total time, where the BCL substitutes are super-linear — a `Dictionary>` set-merge is `O(n²)` to coalesce `n` singletons, and a per-query BFS/DFS is `O(V+E)` every query. Grows only by merging (no un-union); it is not an `ISet` — for element membership with add/remove/set-algebra use `CeleritySet` or `HashSet`. | | **Priority queue whose priorities change** — a best-so-far frontier you relax (Dijkstra / Prim / A\*), or an event scheduler that reschedules / cancels pending items | `IndexedPriorityQueue` | Addressable binary min-heap with an element→slot index: `Update` (decrease-/increase-key) and `Remove` an arbitrary element in `O(log n)`, `Contains` / `TryGetPriority` in `O(1)`. The BCL `PriorityQueue<,>` can do none of these — its only substitute is lazy deletion, which grows the heap by one entry per update. Each element is a key (appears once); custom `IComparer` for a max-heap. For plain enqueue/dequeue with duplicate elements, the BCL `PriorityQueue<,>` is simpler. | | **Prefix / autocomplete / longest-prefix** over string keys — list everything under a prefix, find the most specific stored key that prefixes a query, or iterate keys in order (typeahead, route/dispatch tables, tokenizer / dictionary matching, namespace listing) | `Trie` | Ordered prefix tree: `GetByPrefix` yields every entry under a prefix in `O(prefix + matches)` and in ascending key order, `TryGetLongestPrefix` finds the longest stored prefix of a query in `O(query)`, and enumeration is sorted for free — none of which a `Dictionary` can do without an `O(n)` scan + `StartsWith`. For **pure exact-key** `Add` / `TryGetValue` / `Remove` a `Dictionary` (one hash vs a per-character walk) is faster; the trie earns its place only when you use the prefix operations. Implements `IReadOnlyDictionary`; not thread-safe. | +| **Many occurrences of few distinct strings**, produced as slices of a buffer you already hold — CSV / log / JSON parsing, tokenizers, column stores, header dispatch: you want one canonical `string` per distinct token, not one per occurrence | `StringInternTable` | Probed with a `ReadOnlySpan`: `GetOrAdd` returns the shared instance and **allocates only on a miss**, so a 10M-cell parse over 100 distinct tokens creates 100 strings, not 10,000,000. `HashSet` cannot express this before .NET 9 — its `TryGetValue` takes a `string`, so you must allocate the string *before* you can discover you already had it. `string.Intern` is process-wide, never collected, and still needs a `string`; this table's lifetime is yours and `Clear` releases it. On .NET 9+ `Dictionary.GetAlternateLookup` is comparable; this works on `net8.0` too. Not thread-safe. | +| **Look a string key up from a `ReadOnlySpan`** you already hold (route dispatch, header lookup, parse-then-map) without allocating a `string` per probe | span overloads on `FrozenCelerityDictionary` / `FrozenCeleritySet` / `CelerityDictionary` / `CeleritySet` / `Trie` | `TryGetValue(ReadOnlySpan, …)` / `ContainsKey` / `Contains` probe the table directly, deleting the `new string(span)` allocation and copy per lookup. Available whenever the hasher implements `ISpanHashProvider` — every built-in `String*Hasher` does. Same results as the `string` overloads (ordinal comparison); an empty span means `""`, never the `null` key. See [span-keyed lookups](docs/api/collections.md#span-keyed-lookups). | | **Sorted keys** — you need the entries in comparer order, or the ordered questions a hash table cannot answer: smallest / largest key, "first key at or after *x*", "every key in `[a, b)`" (time-series by timestamp, order books, LSM-style memtables, sweep-line events, interval endpoints) | `BTreeDictionary` / `BTreeSet` | B-tree with up to 31 keys per node in flat arrays: a lookup visits `log₃₂(n)` nodes instead of chasing `log₂(n)` pointers (~4 cache misses instead of ~20 at `n = 1M`), an in-order walk streams contiguous arrays rather than successor pointers, and allocation is one node per 31 entries instead of one object per entry. The BCL has no B-tree: `SortedDictionary<,>` / `SortedSet<>` are red-black trees, `SortedList<,>` is `O(n)` per middle insert, and `OrderedDictionary<,>` (.NET 9) is *insertion*-ordered, not sorted. Wins on the **interleaved insert + lookup + range-scan** load; for a few dozen entries a `SortedList<,>` is hard to beat, and if you never need order a hash table answers in `O(1)`. | | **Prefix / range sums over a sequence you keep mutating** — running aggregates, rank / order-statistics counters (inversions, "how many ≤ x seen"), cumulative-frequency tables | `FenwickTree` | Binary Indexed Tree (`T : INumber`): **point update** and **prefix / range sum** both `O(log n)`, in one array with no per-node overhead. The BCL has no prefix-sum structure; a plain array forces `O(n)` per query (recompute the slice) *or* `O(n)` per update (fix the suffix). Wins precisely when updates and partial-sum queries interleave. If the data is immutable after build, a one-shot precomputed prefix-sum array answers in `O(1)` with less code; if you only update and never query a partial sum, a raw array is simpler. | | Need a stable iteration order or multi-threaded access | `BTreeDictionary<,>` / `BTreeSet<>` for sorted order, `Trie` for ordered string keys; BCL `ConcurrentDictionary<,>` for concurrency | Celerity is single-threaded, and the **hash-based** collections leave iteration order unspecified. The ordered collections do promise order by contract: the B-trees iterate in comparer order, `Trie` in ascending ordinal key order. | @@ -605,6 +612,8 @@ The value of a struct hasher is **distribution quality (avalanche), determinism, > **Fixed-seed hashers are not a HashDoS defence.** `string.GetHashCode()` is already a purpose-built **Marvin32** with per-process random seeding; a hardcoded-seed Murmur3 / FNV / xxHash is *not* more flood-resistant — usually **less**, because an attacker who knows the fixed algorithm and seed can precompute colliding keys offline. What stops hash-flooding is a **keyed** PRF with a *secret, per-process-random* key, not merely picking a "stronger" fixed hash. For untrusted `string` keys, the BCL `string.GetHashCode()` (`DefaultHasher`) is the safe default; reach for the keyed SipHash / HighwayHash hashers only when you also supply a secret seed. The fixed-seed hashers' real strength is **reproducibility** (same code across processes and runtimes), which `GetHashCode()` deliberately does not give you. +> **Probing from a `ReadOnlySpan`? Any `String*Hasher` will do.** All 23 implement **`ISpanHashProvider`** — a `Hash(ReadOnlySpan)` overload that returns exactly what `Hash(string)` returns for the same characters — which is what unlocks the [span-keyed lookups](docs/api/collections.md#span-keyed-lookups) and `StringInternTable`. The two overloads share one body, so they cannot drift; a custom string hasher only needs to implement the interface to work the same way. See [`ISpanHashProvider`](docs/api/hashing.md#ispanhashprovider). + The hashing library also ships classic / compatibility hashes (djb2, sdbm, ELF/PJW, CRC-32, Adler-32, FNV-1, MurmurHash2, CityHash, MetroHash, xxHash32/64) for matching an external system's key distribution — see [`docs/api/hashing.md`](docs/api/hashing.md) for the complete list, costs, and avalanche notes, and use `HashQualityEvaluator` (below) to compare candidates on your own keys. ## Benchmarks @@ -731,7 +740,7 @@ Celerity is **Native AOT and trimming compatible** — no reflection, runtime co ## API at a glance -The dictionaries mirror the parts of `Dictionary` most callers reach for: indexer get/set, `ContainsKey`, `TryGetValue`, `Add`, `TryAdd`, `Remove` (both overloads), `Clear`, `EnsureCapacity` / `TrimExcess`, `Count`, `Keys`, `Values`, `GetEnumerator()`. They implement `IReadOnlyDictionary` and accept an `IEnumerable>` at construction. The sets expose `Add`, `TryAdd`, `Contains`, `Remove`, `Clear`, `EnsureCapacity` / `TrimExcess`, `Count`, and a struct enumerator. `EnsureCapacity(n)` pre-grows the table once for a known-size bulk insert (no incremental rehashes); `TrimExcess()` rehashes back down to fit `Count`. The zero / `default(TKey)` key (or element) is stored out-of-band so it never collides with the empty-slot sentinel. +The dictionaries mirror the parts of `Dictionary` most callers reach for: indexer get/set, `ContainsKey`, `TryGetValue`, `Add`, `TryAdd`, `Remove` (both overloads), `Clear`, `EnsureCapacity` / `TrimExcess`, `Count`, `Keys`, `Values`, `GetEnumerator()`. The string-keyed types additionally take a `ReadOnlySpan` on `TryGetValue` / `ContainsKey` / `Contains`, so a caller holding a slice of a buffer never allocates a `string` to probe. They implement `IReadOnlyDictionary` and accept an `IEnumerable>` at construction. The sets expose `Add`, `TryAdd`, `Contains`, `Remove`, `Clear`, `EnsureCapacity` / `TrimExcess`, `Count`, and a struct enumerator. `EnsureCapacity(n)` pre-grows the table once for a known-size bulk insert (no incremental rehashes); `TrimExcess()` rehashes back down to fit `Count`. The zero / `default(TKey)` key (or element) is stored out-of-band so it never collides with the empty-slot sentinel. Full constructors, signatures, exceptions, and per-type examples: **[API reference](docs/README.md)**. diff --git a/ROADMAP.md b/ROADMAP.md index 118af84b..8de80eb6 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -214,7 +214,7 @@ The 2026-Q3 review surveyed the shipped surface against the current .NET 8/9/10 - Fix `HyperLogLog`'s hash-entropy floor. `Hash64` widened a 32-bit `IHashProvider` result, so the reachable hash space was 2^32 — while the type's own docs asserted a 64-bit space and skipped the classical large-range correction on that basis. The bias exceeded the advertised 0.81% standard error from ~1e8 distinct elements, in exactly the regime the type is sold for. Status: `done` — `IHashProvider64` (`ulong Hash64(T key)`) ships as a standalone sibling interface in `Celerity.Hashing`, deliberately *not* deriving from `IHashProvider` so the two contracts stay independent and a 64-bit hasher is never forced to publish a lossy 32-bit fold. Fourteen built-in hashers implement it — `Int64WangHasher`, `Int64Murmur3Hasher`, `UInt64WangHasher`, `UInt64Hasher`, `GuidHasher`, and the nine 64-bit `string` hashers — each of which already computed 64 bits internally and folded them away, so `Hash64` is the same mixer minus the narrowing. The 32-bit-only hashers (`Int32*` / `UInt32*`, the naive folds, `DefaultHasher`) deliberately do not, since a key type narrower than 64 bits has no entropy to publish; a roster test pins that judgement. All five sketches route through it when the hasher provides it, via a compile-time type test the JIT folds away (so neither path allocates or branches) and with existing constructors and type parameters unchanged; on a 32-bit hasher `HyperLogLog` now applies the classical Flajolet large-range correction it previously skipped. `HashQualityEvaluator.Evaluate64` reports distribution over the 64-bit surface. Tracked in [#304](https://github.com/marius-bughiu/Celerity/issues/304). - Implement `IReadOnlySet` on the mutable sets and `IDictionary` on the dictionaries. The sets implement `ISet` and the dictionaries `IReadOnlyDictionary<,>`, but `ISet` does not derive from `IReadOnlySet` — so an ordinary BCL-shaped API taking either interface is a compile error against a Celerity type today. This is the same Guiding Principle #3 gap the 2.2.0 set-algebra work closed, one level up. Status: `planned`. - Delete the per-probe virtual call. The probe loops test for an empty slot with `EqualityComparer.Default.Equals(slot, default(TKey))`, which the JIT devirtualizes for value-type keys but not under `__Canon`-shared reference-type instantiations — one `callvirt` per probe iteration to perform what is a null check. Guiding Principle #2 exists to remove exactly this. Status: `done` — the twelve open-addressed collections now route every vacant-slot test through an internal `EmptySlot.Is` helper whose `typeof(T).IsValueType` guard the JIT folds, so a reference-type instantiation compiles to a plain null test and a value-type one keeps the existing intrinsic comparison unchanged. Behaviour is identical by construction and the whole existing suite passes untouched; `ReferenceKeyProbeTests` pins the substitution against a key type whose `Equals` claims equality with `null`, and the new `StringKeyProbeBenchmark` gives the dashboard its first reference-type-key rows. The follow-up `IEqualityProvider` idea was **not** opened: a `HashCachingDictionary` control arm showed the residual reference-type-key deficit is dominated by re-hashing the key on every probe, not by the remaining equality dispatch — the actionable guidance is to use the hash-caching variants, now documented in [`docs/performance.md`](docs/performance.md#reference-type-keys-cache-the-hash). Tracked in [#308](https://github.com/marius-bughiu/Celerity/issues/308). -- Span-keyed lookups on the string-keyed collections. .NET 9's `GetAlternateLookup>` lets the BCL `Dictionary` probe with a span key and no allocation; Celerity's string-keyed types require a materialized `string`, so the BCL is now *ahead* on the axis this library has invested most in. Status: `planned`. +- Span-keyed lookups on the string-keyed collections. .NET 9's `GetAlternateLookup>` lets the BCL `Dictionary` probe with a span key and no allocation; Celerity's string-keyed types require a materialized `string`, so the BCL is now *ahead* on the axis this library has invested most in. Status: `done` — `ISpanHashProvider` (`int Hash(ReadOnlySpan key)`) ships as a standalone sibling interface in `Celerity.Hashing`, deliberately *not* deriving from `IHashProvider`: that interface is generic in its key type, and a `ref struct` could not be a generic type argument before `allows ref struct` (C# 13 / .NET 9) while `net8.0` remains the floor — expressing the span overload as a non-generic sibling sidesteps that, because the span is a method parameter rather than a type argument. All 23 built-in `String*Hasher` types implement it, each sharing one body between the two overloads so they cannot drift; `SpanHashParityTests` pins `Hash(s) == Hash(s.AsSpan())` per hasher across every length class and as a slice of a larger buffer, since a divergence would silently report a stored key as absent rather than merely being slow. `FrozenCelerityDictionary`, `FrozenCeleritySet`, `CelerityDictionary`, `CeleritySet` and `Trie` gained span `TryGetValue` / `ContainsKey` / `Contains`; on the four hashed types they are extension methods carrying the extra `ISpanHashProvider` constraint on the *method*, so no shipped type's own constraints changed (which would have broken every existing instantiation) and the JIT still devirtualizes through the struct type parameter. `StringInternTable` ships alongside them as the type the pattern makes possible: `GetOrAdd(ReadOnlySpan)` allocates only on a miss, so a 10M-cell parse over 100 distinct tokens creates 100 strings — the one collection the pre-.NET-9 BCL cannot express, since `HashSet.TryGetValue` makes you allocate the string before you can discover you already had it. The optional `ReadOnlySpan` UTF-8 axis and the `#if NET9_0_OR_GREATER` `IAlternateEqualityComparer` plumbing were both left out as the issue's own scoping allowed — neither is needed for the workload win, and each would widen a brand-new public abstraction before it is load-bearing. Tracked in [#311](https://github.com/marius-bughiu/Celerity/issues/311). **The ordered / compressed integer-data lane.** Opened by the sorted-container hole — 38 collections and not one sorted map or set, with `Trie` the only ordered type, and that one string-keyed. The B-trees below close that half; the compressed-integer half is still open. diff --git a/docs/api/collections.md b/docs/api/collections.md index 772be339..066321cd 100644 --- a/docs/api/collections.md +++ b/docs/api/collections.md @@ -144,6 +144,10 @@ public Enumerator GetEnumerator() Returns a struct enumerator that yields `KeyValuePair`. The out-of-band default-key entry is yielded first if present. *Structurally* mutating the dictionary during enumeration — adding a new key, removing a key, or `Clear` — throws `InvalidOperationException` from the next `MoveNext` / `Reset` call, matching BCL `Dictionary<,>` semantics. Overwriting the value of an existing key via the indexer (`dict[existingKey] = newValue`) is *not* a structural change and does not invalidate an active enumerator, so the common "iterate and update values in place" pattern is legal. Iteration order is unspecified and may change between versions. +### Span-keyed lookups (string keys) + +When `TKey` is `string` and the hasher also implements `ISpanHashProvider` (every built-in `String*Hasher` does), `TryGetValue(ReadOnlySpan, out TValue?)` and `ContainsKey(ReadOnlySpan)` probe directly from a slice of a caller-held buffer — no `new string(span)` per lookup. See [span-keyed lookups](#span-keyed-lookups). + ### IReadOnlyDictionary<TKey, TValue?> `CelerityDictionary` implements `IReadOnlyDictionary` via thin explicit interface forwarders on top of the existing struct `KeyCollection` / `ValueCollection` / `Enumerator` types. The zero-allocation `foreach` fast path is preserved; the interface path boxes the enumerator exactly once per `GetEnumerator()` call, matching BCL `Dictionary<,>` behaviour. The out-of-band default-key entry is surfaced through every interface member. @@ -657,6 +661,10 @@ Each throws `ArgumentNullException` when `other` is `null`. The subset / equalit > **`Add` note.** `ISet.Add(T)` returns `bool` (the non-throwing add, equivalent to `TryAdd`). The concrete `public void Add(T)` keeps its throw-on-duplicate behaviour — cast to `ISet`, or use `TryAdd`, when you want the boolean result. `ICollection.Add(T)` ignores duplicates (never throws). +### Span-keyed lookups (string elements) + +When `T` is `string` and the hasher also implements `ISpanHashProvider` (every built-in `String*Hasher` does), `Contains(ReadOnlySpan)` probes directly from a slice of a caller-held buffer — no `new string(span)` per lookup. See [span-keyed lookups](#span-keyed-lookups). + ### Default-element handling `default(T)` is stored out-of-band via a `_hasDefaultValue` flag and never collides with the empty-slot sentinel. Mutating the set during enumeration — including via a mutating set operation such as `UnionWith` — throws `InvalidOperationException` on the next `MoveNext` / `Reset`, matching BCL `HashSet`. @@ -1274,6 +1282,8 @@ var byName = new FrozenCelerityDictionary(new[] Console.WriteLine(byName["alice"]); // 1 ``` +Both the convenience type and this one also carry `TryGetValue(ReadOnlySpan, out TValue?)` and `ContainsKey(ReadOnlySpan)` — see [span-keyed lookups](#span-keyed-lookups). + ## FrozenCeleritySet ```csharp @@ -1391,6 +1401,8 @@ var tags = new FrozenCeleritySet(new[] { "alice", "bob" }); Console.WriteLine(tags.Contains("alice")); // True ``` +Both the convenience type and this one also carry `Contains(ReadOnlySpan)` — see [span-keyed lookups](#span-keyed-lookups). + ## CelerityMultiMap<TKey, TValue, THasher> ```csharp @@ -3662,11 +3674,14 @@ The getter throws `KeyNotFoundException` if `key` is absent (an interior prefix | `void Add(string key, TValue value)` | Adds a key. Throws `ArgumentException` if it already exists. | | `bool TryAdd(string key, TValue value)` | Adds a key, leaving an existing entry unchanged. Returns `false` if already present. | | `bool ContainsKey(string key)` | Whether `key` is a stored key (an interior-only prefix returns `false`). | +| `bool ContainsKey(ReadOnlySpan key)` | The same, from a character span — no `string` is materialized. See [span-keyed lookups](#span-keyed-lookups). | | `bool TryGetValue(string key, out TValue? value)` | Non-throwing exact lookup. | +| `bool TryGetValue(ReadOnlySpan key, out TValue? value)` | The same, from a character span. | | `bool Remove(string key)` | Removes a key, pruning any newly-dead nodes. Returns `false` if absent. | | `bool Remove(string key, out TValue? value)` | `Remove` returning the removed value (`default` when the key was absent). | | `void Clear()` | Removes all keys. | | `bool ContainsPrefix(string prefix)` | Whether any stored key starts with `prefix` (a key equal to the prefix counts). The empty prefix matches iff the trie is non-empty. | +| `bool ContainsPrefix(ReadOnlySpan prefix)` | The same, from a character span. | | `IEnumerable> GetByPrefix(string prefix)` | Every entry whose key starts with `prefix`, in ascending key order (lazy). | | `IEnumerable GetKeysWithPrefix(string prefix)` | The keys of `GetByPrefix`, in ascending order (lazy). | | `bool TryGetLongestPrefix(string query, out string? key, out TValue? value)` | The longest stored key that is a prefix of `query` (an exact match qualifies and is longest). On a miss (`false`), `key` is `null` and `value` is `default`. | @@ -3703,6 +3718,129 @@ if (routes.TryGetLongestPrefix("/api/v1/users/42", out string? route, out string Console.WriteLine($"matched {route} -> {handler}"); // matched /api/v1/users -> users-v1 ``` +## Span-keyed lookups + +Every string-keyed Celerity collection can be probed with a `ReadOnlySpan` — a slice of a buffer the caller already holds — without first materializing a `string`. + +| Type | Span members | +|------|--------------| +| `FrozenCelerityDictionary` | `TryGetValue(ReadOnlySpan, out TValue?)`, `ContainsKey(ReadOnlySpan)` | +| `FrozenCeleritySet` | `Contains(ReadOnlySpan)` | +| `CelerityDictionary` | `TryGetValue(ReadOnlySpan, out TValue?)`, `ContainsKey(ReadOnlySpan)` | +| `CeleritySet` | `Contains(ReadOnlySpan)` | +| `Trie` | `TryGetValue(ReadOnlySpan, out TValue?)`, `ContainsKey(ReadOnlySpan)`, `ContainsPrefix(ReadOnlySpan)` | + +### Why it matters + +Without them, a tokenizer, CSV / log reader, or route dispatcher holding a `ReadOnlySpan` must call `new string(span)` to probe the collection: **one allocation plus a copy per lookup**, on the hot path of exactly the workloads these types exist for. The span overloads delete both. Nothing else changes — stored keys are compared ordinally against the span, which is what `EqualityComparer.Default` does, so a span lookup and the equivalent string lookup always agree. + +> **.NET 9+.** `Dictionary.GetAlternateLookup>()` gives the BCL the same capability on .NET 9 and later. These overloads work on **all three** of Celerity's target frameworks, including the `net8.0` floor where the BCL has no answer. + +### How they are exposed + +On the four hashed collections the span overloads are **extension methods** in `Celerity.Collections` (`SpanLookupExtensions`), because the span probe needs the hasher to implement [`ISpanHashProvider`](hashing.md#ispanhashprovider) as well as `IHashProvider` — and adding that to the class's own constraint would break every existing instantiation. The extra constraint therefore lives on the methods: + +```csharp +where THasher : struct, IHashProvider, ISpanHashProvider +``` + +They bind only when the hasher supplies both, resolve statically (no boxing — the JIT still devirtualizes the hash call through the struct type parameter), and read like instance methods at the call site as long as `Celerity.Collections` is in scope. Every built-in `String*Hasher` implements both. `Trie` takes no hasher, so its span overloads are ordinary instance methods. + +### The empty span + +A span has no `null` state, so an **empty span means the empty string `""`** — an ordinary key — and never the out-of-band `null` key. Look the `null` key up through the `string` overload. + +### Usage example + +```csharp +using Celerity.Collections; +using Celerity.Hashing; + +var routes = new FrozenCelerityDictionary( +[ + new("/api/v1/users", 1), + new("/api/v1/orders", 2), +]); + +ReadOnlySpan line = "GET /api/v1/users HTTP/1.1".AsSpan(); +ReadOnlySpan path = line.Slice(4, 14); + +if (routes.TryGetValue(path, out int handler)) // no string allocated + Dispatch(handler); +``` + +## StringInternTable + +A canonicalizing table of strings: probe it with a `ReadOnlySpan` and it returns the one shared `string` for those characters, allocating **only on a miss**. + +```csharp +public sealed class StringInternTable : StringInternTable + +public class StringInternTable : IReadOnlyCollection + where THasher : struct, IHashProvider, ISpanHashProvider +``` + +### The documented BCL-beating workload + +**A 10M-cell CSV or log parse over ~100 distinct tokens.** The intern table allocates **100 strings instead of 10,000,000** — only the miss path materializes one. Downstream reference equality then works, and the GC never sees the other 9,999,900 copies. + +This is the one collection you cannot build on the pre-.NET-9 BCL: `HashSet.TryGetValue` takes a `string`, so you must **allocate the string before you can discover you already had it** — the very allocation you were trying to avoid. `string.Intern` is not a substitute either: it is process-wide, never collected for the life of the process, and still requires a `string` to hand it. A `StringInternTable` is an ordinary object — its scope is yours, `Clear` releases everything it holds, and dropping the table drops the interned strings with it. + +> **.NET 9+.** `Dictionary.GetAlternateLookup>()` can express the same pattern on .NET 9 and later. This type works on all three of Celerity's target frameworks, including the `net8.0` floor. + +### Default hasher + +The non-generic `StringInternTable` uses `StringFnV1AFullHasher` — cheap, and **full Unicode width**, so tokens that differ only in a character's high byte do not collide. (This deliberately differs from the frozen collections' `StringFnV1AHasher` default, which folds only the low byte: an intern table's inputs are arbitrary parsed text rather than curated identifiers.) A collision is never a correctness problem in either case — the ordinal span comparison resolves it — but it costs probes. Supply any `String*Hasher` through `StringInternTable` for a different speed/quality point. + +### Constructors + +```csharp +public StringInternTable(int capacity = 16, float loadFactor = 0.75f) +``` + +**Throws:** + +- `ArgumentOutOfRangeException` if `capacity` is negative, or `loadFactor` is not in the open interval (0, 1). + +### Methods and properties + +| Member | Description | +|--------|-------------| +| `int Count` | Number of distinct strings interned. | +| `string GetOrAdd(ReadOnlySpan key)` | The canonical instance for those characters, allocating one only on a miss. | +| `string GetOrAdd(string key)` | The canonical instance; on a miss the supplied instance itself becomes canonical, so this never allocates. Throws `ArgumentNullException` on `null`. | +| `bool TryGet(ReadOnlySpan key, out string? value)` | Pure lookup — never interns. `value` is `null` on a miss. | +| `bool Contains(ReadOnlySpan key)` | Whether those characters are already interned. | +| `bool Contains(string key)` | The same. Throws `ArgumentNullException` on `null`. | +| `void Clear()` | Drops every interned string; the backing capacity is preserved. | +| `Enumerator GetEnumerator()` | An allocation-free struct enumerator over the interned strings. Order is unspecified. | + +`GetOrAdd` (when it interns) and `Clear` are structural changes that invalidate an in-flight enumerator; `TryGet` and `Contains` never mutate. + +### Empty-string and `null` handling + +The empty string is an ordinary entry, and an empty span means `""`. `null` is not storable — the `string` overloads reject it. + +### Choosing it + +Reach for `StringInternTable` when you are producing **many occurrences of few distinct strings** from a buffer you already hold: parsers, tokenizers, log ingestion, column stores, header dispatch. If you need to associate a *value* with each token, use a `CelerityDictionary` with the [span lookups](#span-keyed-lookups) instead. It is not thread-safe. + +### Usage example + +```csharp +using Celerity.Collections; + +var interned = new StringInternTable(); + +foreach (ReadOnlySpan cell in SplitCells(line)) +{ + string token = interned.GetOrAdd(cell); // allocates only the first time each token is seen + Consume(token); +} + +Console.WriteLine(interned.Count); // distinct tokens, == strings allocated +``` + ## FenwickTree<T> ```csharp diff --git a/docs/api/hashing.md b/docs/api/hashing.md index 4af991bc..acfb4a28 100644 --- a/docs/api/hashing.md +++ b/docs/api/hashing.md @@ -138,6 +138,65 @@ public struct MyLongHasher : IHashProvider, IHashProvider64 --- +## ISpanHashProvider + +```csharp +public interface ISpanHashProvider +{ + int Hash(ReadOnlySpan key); +} +``` + +The span sibling of `IHashProvider`, so a caller holding a slice of a buffer can hash it — and probe a string-keyed collection with it — without first materializing a `string`. + +**Every built-in `String*Hasher` implements it.** Each already walks the characters, so the span overload *is* the body and the `string` overload delegates to it. The integer and `Guid` hashers do not: a character span is not their key shape. + +### The contract + +For every `string s`, an implementation must satisfy: + +```csharp +hasher.Hash(s) == hasher.Hash(s.AsSpan()) +``` + +This is load-bearing, not cosmetic. The [span lookups](collections.md#span-keyed-lookups) hash a span and then compare the result against keys that were *placed* using the `string` overload. A divergence would not merely be slow — it would report a stored key as absent. Share one body between the two overloads rather than maintaining two copies; `SpanHashParityTests` pins the contract for every built-in hasher across every length class, including as a slice of a larger buffer. + +A span has no `null` state: `default(ReadOnlySpan)` is empty, and empty means the empty string `""`. The `string` overloads keep throwing `ArgumentNullException` on `null` exactly as before. + +### Why it is a separate interface + +`IHashProvider.Hash(T)` is generic in its key type, and a `ref struct` such as `ReadOnlySpan` could not be used as a generic type argument before `allows ref struct` (C# 13 / .NET 9) — while `net8.0` is Celerity's floor. Expressing the span overload as a separate **non-generic** interface sidesteps that entirely, because the span is a method parameter rather than a type argument. The same reasoning gave `IHashProvider64` its independent shape. + +### Implementing it on a custom hasher + +Put the algorithm in the span overload and delegate: + +```csharp +public struct MyStringHasher : IHashProvider, ISpanHashProvider +{ + public int Hash(string key) + { + ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); // one body, so the two cannot drift + } + + public int Hash(ReadOnlySpan key) + { + uint hash = 2166136261u; + foreach (char c in key) + { + hash ^= c; + hash *= 16777619u; + } + return unchecked((int)hash); + } +} +``` + +Implementations must be structs, for the same devirtualization reason as `IHashProvider`. The collections' span overloads are constrained on `where THasher : struct, IHashProvider, ISpanHashProvider`, so a hasher that implements only one of the two simply will not bind there. + +--- + ## Built-in Hashers ### Int32IdentityHasher diff --git a/src/Celerity.AotSmokeTest/Program.cs b/src/Celerity.AotSmokeTest/Program.cs index aaec7ef4..7b04b5d0 100644 --- a/src/Celerity.AotSmokeTest/Program.cs +++ b/src/Celerity.AotSmokeTest/Program.cs @@ -761,6 +761,71 @@ void Check(bool condition, string message) Check(wide.Count == 0 && !wide.ContainsPrefix("key"), "Trie.Clear"); } +// Span-keyed lookups + StringInternTable. These are the paths where the JIT's devirtualization +// of a struct hasher through a *method*-level generic constraint (ISpanHashProvider on the +// SpanLookupExtensions methods, not on the collection's own type parameter) has to survive AOT +// compilation — every instantiation below is one the ILC has to see and generate. +{ + // The key sits inside a larger buffer, exactly as a parser would hold it: nothing here is + // ever turned into a string before the probe. + char[] buffer = "..alpha..beta..".ToCharArray(); + ReadOnlySpan alpha = buffer.AsSpan(2, 5); + ReadOnlySpan beta = buffer.AsSpan(9, 4); + ReadOnlySpan missing = "gamma".AsSpan(); + + var pairs = new[] + { + new KeyValuePair("alpha", 1), + new KeyValuePair("beta", 2), + }; + + var frozenDict = new FrozenCelerityDictionary(pairs); + Check(frozenDict.TryGetValue(alpha, out int fdA) && fdA == 1, "FrozenCelerityDictionary span TryGetValue"); + Check(frozenDict.ContainsKey(beta) && !frozenDict.ContainsKey(missing), "FrozenCelerityDictionary span ContainsKey"); + + var frozenSet = new FrozenCeleritySet(new[] { "alpha", "beta" }); + Check(frozenSet.Contains(alpha) && !frozenSet.Contains(missing), "FrozenCeleritySet span Contains"); + + var dict = new CelerityDictionary(); + dict.Add("alpha", 1); + dict.Add("beta", 2); + Check(dict.TryGetValue(beta, out int dB) && dB == 2, "CelerityDictionary span TryGetValue"); + Check(dict.ContainsKey(alpha) && !dict.ContainsKey(missing), "CelerityDictionary span ContainsKey"); + + var set = new CeleritySet(); + set.Add("alpha"); + Check(set.Contains(alpha) && !set.Contains(missing), "CeleritySet span Contains"); + + var spanTrie = new Trie(); + spanTrie["alpha"] = 1; + Check(spanTrie.TryGetValue(alpha, out int tA) && tA == 1, "Trie span TryGetValue"); + Check(spanTrie.ContainsKey(alpha) && spanTrie.ContainsPrefix("alp".AsSpan()), "Trie span ContainsKey/ContainsPrefix"); + + // Every String*Hasher must answer Hash(s) == Hash(s.AsSpan()) after AOT compilation too — + // the contract the span probes above are built on. + var spanHasher = new StringXxHash3Hasher(); + Check(spanHasher.Hash("alpha") == spanHasher.Hash(alpha), "ISpanHashProvider string/span parity"); + + // StringInternTable: the miss path allocates, every repeat returns the same reference. + var interned = new StringInternTable(); + string first = interned.GetOrAdd(alpha); + string second = interned.GetOrAdd("xxalphaxx".AsSpan(2, 5)); + Check(first == "alpha" && ReferenceEquals(first, second), "StringInternTable canonicalizes by contents"); + Check(interned.Count == 1 && interned.Contains(alpha) && !interned.Contains(missing), "StringInternTable Count/Contains"); + Check(interned.TryGet(alpha, out string? got) && ReferenceEquals(got, first), "StringInternTable.TryGet"); + interned.GetOrAdd(beta); + int internedSeen = 0; + foreach (string _ in interned) internedSeen++; + Check(internedSeen == 2, "StringInternTable struct enumerator"); + interned.Clear(); + Check(interned.Count == 0, "StringInternTable.Clear"); + + // A second hasher instantiation, so the ILC generates more than one closed generic. + var internedStrong = new StringInternTable(); + Check(ReferenceEquals(internedStrong.GetOrAdd(alpha), internedStrong.GetOrAdd(alpha)), + "StringInternTable canonicalizes"); +} + // EnumMap — dense array-backed dictionary for enum keys (the .NET EnumMap). Exercise // the indexer, TryAdd/Add, TryGetValue, Remove, the parallel occupancy vector // (default value distinct from absent), and the ascending-order struct enumerator. diff --git a/src/Celerity.Benchmarks/FrozenCelerityDictionaryBenchmark.cs b/src/Celerity.Benchmarks/FrozenCelerityDictionaryBenchmark.cs index 8ebe6b5e..0074ebae 100644 --- a/src/Celerity.Benchmarks/FrozenCelerityDictionaryBenchmark.cs +++ b/src/Celerity.Benchmarks/FrozenCelerityDictionaryBenchmark.cs @@ -11,6 +11,11 @@ public class FrozenCelerityDictionaryBenchmark private string[] keys = null!; private KeyValuePair[] pairs = null!; + // The same keys laid end to end in one buffer, with the slice bounds of each — the shape a parser or + // router actually holds its keys in, and the input to the SpanLookup category below. + private char[] buffer = null!; + private (int Offset, int Length)[] slices = null!; + private FrozenDictionary frozenDictionary = null!; private FrozenCelerityDictionary frozenCelerity = null!; @@ -22,13 +27,19 @@ public void Setup() { keys = new string[ItemCount]; pairs = new KeyValuePair[ItemCount]; + slices = new (int, int)[ItemCount]; + var text = new System.Text.StringBuilder(); for (int i = 0; i < ItemCount; i++) { // Identifier-shaped, guaranteed-distinct keys. keys[i] = "celerity/key/" + i + "/" + (i * 2654435761u); pairs[i] = new KeyValuePair(keys[i], i); + slices[i] = (text.Length, keys[i].Length); + text.Append(keys[i]).Append(' '); } + buffer = text.ToString().ToCharArray(); + frozenDictionary = pairs.ToFrozenDictionary(p => p.Key, p => p.Value); frozenCelerity = new FrozenCelerityDictionary(pairs); } @@ -70,4 +81,38 @@ public int FrozenCelerityDictionary_Lookup() acc += frozenCelerity[key]; return acc; } + + // ── SpanLookup (the caller holds spans, not strings) ────────────────────── + // The baseline is what a net8.0 caller must do to probe any string-keyed collection: allocate the + // string first. The Celerity arm probes the span directly, so the whole allocation and copy vanish — + // which is the point, and why this category is the one with allocation numbers worth reading. + // (.NET 9's Dictionary.GetAlternateLookup closes this gap on that runtime; this project + // targets net8.0, the floor where the BCL has no answer.) + + [Benchmark(Baseline = true)] + [BenchmarkCategory("SpanLookup")] + public int FrozenDictionary_SpanLookup() + { + int acc = 0; + for (int i = 0; i < slices.Length; i++) + { + (int offset, int length) = slices[i]; + acc += frozenDictionary[new string(buffer, offset, length)]; + } + return acc; + } + + [Benchmark] + [BenchmarkCategory("SpanLookup")] + public int FrozenCelerityDictionary_SpanLookup() + { + int acc = 0; + for (int i = 0; i < slices.Length; i++) + { + (int offset, int length) = slices[i]; + frozenCelerity.TryGetValue(buffer.AsSpan(offset, length), out int value); + acc += value; + } + return acc; + } } diff --git a/src/Celerity.Benchmarks/Program.cs b/src/Celerity.Benchmarks/Program.cs index 72aa01c9..0585bb7a 100644 --- a/src/Celerity.Benchmarks/Program.cs +++ b/src/Celerity.Benchmarks/Program.cs @@ -47,6 +47,7 @@ internal class Program typeof(DisjointSetBenchmark), typeof(IndexedPriorityQueueBenchmark), typeof(TrieBenchmark), + typeof(StringInternTableBenchmark), typeof(FenwickTreeBenchmark), typeof(BTreeDictionaryBenchmark), typeof(BTreeSetBenchmark), diff --git a/src/Celerity.Benchmarks/StringInternTableBenchmark.cs b/src/Celerity.Benchmarks/StringInternTableBenchmark.cs new file mode 100644 index 00000000..f280a5f1 --- /dev/null +++ b/src/Celerity.Benchmarks/StringInternTableBenchmark.cs @@ -0,0 +1,146 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using Celerity.Collections; + +/// +/// against the BCL shapes a parser would otherwise reach for, over the +/// workload the type exists to serve: a token stream held as spans of one input buffer, with far fewer +/// distinct tokens than occurrences. +/// +/// +/// +/// The headline is the Allocated column, not the nanoseconds — which is why +/// [MemoryDiagnoser] reports bytes here. Every BCL arm must call new string(span) before it +/// can even ask whether it already had that token, so it allocates once per occurrence; +/// is probed with the span and allocates once per distinct token. +/// At the sweep's 64-token universe that is a 15×–1500× difference in strings created, and the gap widens +/// with the stream length. +/// +/// +/// The Dedupe category is the honest end-to-end comparison: all three arms end up holding one +/// canonical instance per distinct token, so they differ only in how much garbage they made getting there. +/// Lookup isolates the probe on an already-warm table, where the intern table's win is just the +/// deleted allocation and copy. +/// +/// +/// The .NET 9+ caveat. .NET 9 shipped +/// Dictionary<string,V>.GetAlternateLookup<ReadOnlySpan<char>>(), which closes +/// this gap for a plain dictionary on that runtime. It is not benchmarked here because this project +/// targets net8.0 only — the floor where the BCL has no answer at all, and the reason the type +/// exists. On .NET 9+ the two approaches are comparable; the library's value there is that the same code +/// works across all three of Celerity's target frameworks. +/// +/// +[MemoryDiagnoser] +[CategoriesColumn] +[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] +public class StringInternTableBenchmark +{ + // Distinct tokens in the stream. Deliberately far smaller than ItemCount: the point of interning is + // that a long stream draws from a small vocabulary (column values, log levels, header names, …). + private const int DistinctTokens = 64; + + // One contiguous buffer holding the whole token stream, exactly as a parser would have it. The + // benchmarks slice it — they never hold the tokens as strings. + private char[] buffer = null!; + private (int Offset, int Length)[] tokens = null!; + + // A warm table / set for the Lookup category, built once in [GlobalSetup]. + private StringInternTable internedFull = null!; + private HashSet hashSetFull = null!; + + [Params(1000, 100_000)] + public int ItemCount; + + [GlobalSetup] + public void Setup() + { + var vocabulary = new string[DistinctTokens]; + for (int i = 0; i < DistinctTokens; i++) + vocabulary[i] = $"token_{i:D4}_value"; + + tokens = new (int, int)[ItemCount]; + var text = new System.Text.StringBuilder(ItemCount * 17); + for (int i = 0; i < ItemCount; i++) + { + string token = vocabulary[i % DistinctTokens]; + tokens[i] = (text.Length, token.Length); + text.Append(token).Append(','); // the separator a real parser would slice around + } + + buffer = text.ToString().ToCharArray(); + + internedFull = new StringInternTable(DistinctTokens); + hashSetFull = new HashSet(DistinctTokens, StringComparer.Ordinal); + foreach (string token in vocabulary) + { + internedFull.GetOrAdd(token); + hashSetFull.Add(token); + } + } + + // ---- Dedupe: walk the stream and end up holding one instance per distinct token ------------------- + + // The pre-.NET-9 BCL answer, and the true functional analogue: a dictionary keyed by the token + // hands the canonical instance back, exactly as GetOrAdd does. It still allocates one string per + // *occurrence*, because the key has to exist before it can be looked up — which is the whole gap. + // (A HashSet fill is the same shape minus the canonical instance, so it is not benchmarked + // separately here; the Lookup category below uses HashSet as its baseline.) + [Benchmark(Baseline = true)] + [BenchmarkCategory("Dedupe")] + public int Dictionary_Dedupe() + { + var map = new Dictionary(StringComparer.Ordinal); + for (int i = 0; i < tokens.Length; i++) + { + (int offset, int length) = tokens[i]; + string materialized = new string(buffer, offset, length); + if (!map.TryGetValue(materialized, out _)) + map[materialized] = materialized; + } + return map.Count; + } + + [Benchmark] + [BenchmarkCategory("Dedupe")] + public int StringInternTable_Dedupe() + { + var table = new StringInternTable(); + for (int i = 0; i < tokens.Length; i++) + { + (int offset, int length) = tokens[i]; + table.GetOrAdd(buffer.AsSpan(offset, length)); + } + return table.Count; + } + + // ---- Lookup: probe an already-warm table once per occurrence -------------------------------------- + + [Benchmark(Baseline = true)] + [BenchmarkCategory("Lookup")] + public int HashSet_Lookup() + { + int hits = 0; + for (int i = 0; i < tokens.Length; i++) + { + (int offset, int length) = tokens[i]; + if (hashSetFull.Contains(new string(buffer, offset, length))) + hits++; + } + return hits; + } + + [Benchmark] + [BenchmarkCategory("Lookup")] + public int StringInternTable_Lookup() + { + int hits = 0; + for (int i = 0; i < tokens.Length; i++) + { + (int offset, int length) = tokens[i]; + if (internedFull.Contains(buffer.AsSpan(offset, length))) + hits++; + } + return hits; + } +} diff --git a/src/Celerity.Benchmarks/TrieBenchmark.cs b/src/Celerity.Benchmarks/TrieBenchmark.cs index 19801099..f7d5f255 100644 --- a/src/Celerity.Benchmarks/TrieBenchmark.cs +++ b/src/Celerity.Benchmarks/TrieBenchmark.cs @@ -19,6 +19,11 @@ public class TrieBenchmark private string[] keys = null!; private string[] prefixes = null!; + // The same keys laid end to end in one buffer, with the slice bounds of each — the shape a tokenizer + // actually holds its keys in, and the input to the SpanLookup category below. + private char[] buffer = null!; + private (int Offset, int Length)[] slices = null!; + // Rebuilt per iteration by the [IterationSetup]s below for the build (Add) category. private Trie trie = null!; private Dictionary dict = null!; @@ -38,8 +43,16 @@ public void Setup() prefixes[b] = $"{(char)('a' + b / 26)}{(char)('a' + b % 26)}"; keys = new string[ItemCount]; + slices = new (int, int)[ItemCount]; + var text = new System.Text.StringBuilder(); for (int i = 0; i < ItemCount; i++) + { keys[i] = $"{prefixes[i % PrefixBuckets]}_{i:D8}"; + slices[i] = (text.Length, keys[i].Length); + text.Append(keys[i]).Append(' '); + } + + buffer = text.ToString().ToCharArray(); trieFull = new Trie(); dictFull = new Dictionary(ItemCount); @@ -128,4 +141,39 @@ public long Trie_PrefixMatch() } return acc; } + + // ---- SpanLookup: the caller holds spans, not strings ---------------------------------------------- + // The baseline is what a net8.0 caller must do to probe any string-keyed collection: allocate the + // string first. The trie descends the span directly, so the whole allocation and copy vanish — which + // is why this category is the one with allocation numbers worth reading. It also flips the Lookup + // verdict: the trie's character walk no longer competes against a bare hash, but against a hash plus + // an allocation. (.NET 9's Dictionary.GetAlternateLookup closes this gap on that runtime; + // this project targets net8.0, the floor where the BCL has no answer.) + + [Benchmark(Baseline = true)] + [BenchmarkCategory("SpanLookup")] + public long Dictionary_SpanLookup() + { + long acc = 0; + for (int i = 0; i < slices.Length; i++) + { + (int offset, int length) = slices[i]; + acc += dictFull[new string(buffer, offset, length)]; + } + return acc; + } + + [Benchmark] + [BenchmarkCategory("SpanLookup")] + public long Trie_SpanLookup() + { + long acc = 0; + for (int i = 0; i < slices.Length; i++) + { + (int offset, int length) = slices[i]; + trieFull.TryGetValue(buffer.AsSpan(offset, length), out int value); + acc += value; + } + return acc; + } } diff --git a/src/Celerity.Fuzz/Differential.cs b/src/Celerity.Fuzz/Differential.cs index 1ae3ec0f..3e713a75 100644 --- a/src/Celerity.Fuzz/Differential.cs +++ b/src/Celerity.Fuzz/Differential.cs @@ -45,6 +45,7 @@ public static readonly (string Name, Action Run)[] All = ("CelerityMultiMap", CelerityMultiMapCase), ("FrozenCelerityDictionary", FrozenCase), ("FrozenCeleritySet", FrozenSetCase), + ("StringInternTable", StringInternTableCase), ("BloomFilter", BloomFilterCase), ("CuckooFilter", CuckooFilterCase), ("XorFilter", XorFilterCase), @@ -1237,6 +1238,72 @@ private static void FrozenCase(Random rng) Check(seen == oracle.Count, $"enumeration count {seen} != {oracle.Count}"); } + private static void StringInternTableCase(Random rng) + { + // A tiny, duplicate-rich token universe so hits dominate misses and probe chains stay + // dense; the table starts undersized so the run drives several resizes. + var table = new StringInternTable(capacity: 2); + + // The oracle maps a token's contents to the canonical instance first handed out. + var oracle = new Dictionary(StringComparer.Ordinal); + + int steps = rng.Next(0, 400); + for (int i = 0; i < steps; i++) + { + string token = $"tok_{rng.Next(0, 30)}"; + int op = rng.Next(100); + + if (op < 50) + { + // Intern from a span carved out of a larger buffer — the parser shape. Only a + // miss may materialize a string; a hit must return the instance already held. + string padded = $"<<{token}>>"; + string interned = table.GetOrAdd(padded.AsSpan(2, token.Length)); + + Check(interned == token, $"GetOrAdd(span) returned {interned} for {token}"); + if (oracle.TryGetValue(token, out string? canonical)) + Check(ReferenceEquals(canonical, interned), $"GetOrAdd(span) re-allocated {token}"); + else + oracle[token] = interned; + } + else if (op < 70) + { + string supplied = new string(token.ToCharArray()); + string interned = table.GetOrAdd(supplied); + + if (oracle.TryGetValue(token, out string? canonical)) + Check(ReferenceEquals(canonical, interned), $"GetOrAdd(string) re-allocated {token}"); + else + { + Check(ReferenceEquals(supplied, interned), $"GetOrAdd(string) did not adopt {token}"); + oracle[token] = interned; + } + } + else if (op < 90) + { + bool expected = oracle.TryGetValue(token, out string? canonical); + bool actual = table.TryGet(token.AsSpan(), out string? found); + Check(expected == actual, $"TryGet({token}) {actual} != {expected}"); + Check(!expected || ReferenceEquals(canonical, found), $"TryGet({token}) returned a non-canonical instance"); + Check(table.Contains(token.AsSpan()) == expected, $"Contains(span {token}) != {expected}"); + Check(table.Contains(token) == expected, $"Contains(string {token}) != {expected}"); + } + else + { + int seen = 0; + foreach (string s in table) + { + Check(oracle.TryGetValue(s, out string? canonical) && ReferenceEquals(canonical, s), + $"enumeration yielded a non-canonical or absent {s}"); + seen++; + } + Check(seen == oracle.Count, $"enumeration count {seen} != {oracle.Count}"); + } + + Check(table.Count == oracle.Count, $"Count {table.Count} != {oracle.Count}"); + } + } + private static void FrozenSetCase(Random rng) { // A deliberately tiny, duplicate-rich element universe so the build's dedupe diff --git a/src/Celerity.Hashing/ISpanHashProvider.cs b/src/Celerity.Hashing/ISpanHashProvider.cs new file mode 100644 index 00000000..cbaf4ac9 --- /dev/null +++ b/src/Celerity.Hashing/ISpanHashProvider.cs @@ -0,0 +1,44 @@ +namespace Celerity.Hashing; + +/// +/// Provides a hash function over a of , +/// so a caller holding a slice of an existing buffer can probe a string-keyed collection +/// without first materializing a . +/// +/// +/// +/// The contract. For every string s an implementation must satisfy +/// Hash(s) == Hash(s.AsSpan()). A collection that probes with a span and compares +/// against stored keys relies on this: a divergence would not merely +/// be slow, it would report a stored key as absent. Implementations are expected to share +/// a single body between the two overloads rather than maintain two copies. +/// +/// +/// This interface deliberately does not derive from +/// , mirroring the independent +/// sibling: is generic in +/// its key type, and a ref struct such as could not be +/// used as a generic type argument before allows ref struct (C# 13 / .NET 9) — +/// while net8.0 remains this library's floor. Expressing the span overload as a +/// separate non-generic interface sidesteps that entirely, because the span is a method +/// parameter rather than a type argument. In practice every built-in +/// String*Hasher implements both, so a single struct serves both call shapes. +/// +/// +/// Like , implementations must be value types (structs) so +/// the JIT can devirtualize and inline calls made through a generic type parameter +/// constrained to where THasher : struct, IHashProvider<string>, ISpanHashProvider. +/// +/// +public interface ISpanHashProvider +{ + /// + /// Computes a hash code for the specified character span. + /// + /// The characters to hash. + /// + /// A 32-bit signed integer hash code, equal to the code the same hasher's + /// Hash(string) overload returns for a string with the same contents. + /// + int Hash(ReadOnlySpan key); +} diff --git a/src/Celerity.Hashing/StringAdler32Hasher.cs b/src/Celerity.Hashing/StringAdler32Hasher.cs index 77bbae40..a7d77755 100644 --- a/src/Celerity.Hashing/StringAdler32Hasher.cs +++ b/src/Celerity.Hashing/StringAdler32Hasher.cs @@ -66,7 +66,7 @@ namespace Celerity.Hashing; /// sentinel. /// /// -public struct StringAdler32Hasher : IHashProvider +public struct StringAdler32Hasher : IHashProvider, ISpanHashProvider { /// /// The Adler-32 modulus: 65521, the largest prime below 2^16. @@ -89,7 +89,21 @@ public struct StringAdler32Hasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes the standard Adler-32 (RFC 1950 / zlib) of the specified character span + /// over the full little-endian UTF-16 byte stream (both bytes of every character). + /// + /// The characters to hash. + /// + /// The signed 32-bit Adler-32 of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { uint a = 1u; uint b = 0u; diff --git a/src/Celerity.Hashing/StringCityHash64Hasher.cs b/src/Celerity.Hashing/StringCityHash64Hasher.cs index 9abc5f3c..74291203 100644 --- a/src/Celerity.Hashing/StringCityHash64Hasher.cs +++ b/src/Celerity.Hashing/StringCityHash64Hasher.cs @@ -67,11 +67,11 @@ namespace Celerity.Hashing; /// The algorithm carries 64 bits of state internally, so the type also implements /// : returns that state un-folded, which is /// what the probabilistic sketches want (see for why the extra -/// 32 bits matter there and not in a hash table). is unchanged — it is +/// 32 bits matter there and not in a hash table). is unchanged — it is /// exactly h ^ (h >> 32) of the 64-bit result. /// /// -public struct StringCityHash64Hasher : IHashProvider, IHashProvider64 +public struct StringCityHash64Hasher : IHashProvider, IHashProvider64, ISpanHashProvider { // CityHash v1.1 mixing constants (Geoff Pike & Jyrki Alakuijala, public domain). private const ulong K0 = 0xC3A5C85C97CB3127UL; @@ -96,7 +96,23 @@ public struct StringCityHash64Hasher : IHashProvider, IHashProvider64 + /// Computes the CityHash64 hash of the specified character span, xor-folded to a + /// signed 32-bit result. + /// + /// The characters to hash. + /// + /// The signed 32-bit, xor-folded CityHash64 hash of — the same + /// value returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { + ulong h64 = Hash64Core(key); return unchecked((int)(h64 ^ (h64 >> 32))); } @@ -117,8 +133,16 @@ public int Hash(string key) public ulong Hash64(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash64Core(key.AsSpan()); + } - int charLen = key.Length; // count of UTF-16 code units (chars) + // The single 64-bit body. Both Hash64(string) and the span-based Hash(ReadOnlySpan) + // route through it, so the string and span overloads cannot drift apart — the parity the + // ISpanHashProvider contract requires. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Hash64Core(ReadOnlySpan key) + { + int charLen = key.Length; // count of UTF-16 code units (chars) ulong byteLen = (ulong)charLen * 2UL; // CityHash operates on the byte length ulong h = byteLen <= 32UL @@ -133,7 +157,7 @@ public ulong Hash64(string key) // ── Length-classed hash bodies (mirror CityHash64's dispatch) ─────────────── - private static ulong HashLen0to16(string key, int charLen, ulong byteLen) + private static ulong HashLen0to16(ReadOnlySpan key, int charLen, ulong byteLen) { if (byteLen >= 8UL) { @@ -168,7 +192,7 @@ private static ulong HashLen0to16(string key, int charLen, ulong byteLen) return K2; } - private static ulong HashLen17to32(string key, int charLen, ulong byteLen) + private static ulong HashLen17to32(ReadOnlySpan key, int charLen, ulong byteLen) { ulong mul = K2 + byteLen * 2UL; ulong a = Lane(key, 0) * K1; @@ -181,7 +205,7 @@ private static ulong HashLen17to32(string key, int charLen, ulong byteLen) mul); } - private static ulong HashLen33to64(string key, int charLen, ulong byteLen) + private static ulong HashLen33to64(ReadOnlySpan key, int charLen, ulong byteLen) { ulong mul = K2 + byteLen * 2UL; ulong a = Lane(key, 0) * K2; @@ -204,7 +228,7 @@ private static ulong HashLen33to64(string key, int charLen, ulong byteLen) return b + x; } - private static ulong HashLong(string key, int charLen, ulong byteLen) + private static ulong HashLong(ReadOnlySpan key, int charLen, ulong byteLen) { // For strings over 64 bytes CityHash hashes the end first, then keeps 56 // bytes of state (v, w, x, y, z) and consumes 64 bytes (thirty-two chars) @@ -251,7 +275,7 @@ private static ulong HashLong(string key, int charLen, ulong byteLen) /// read over the native little-endian UTF-16 stream. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static ulong Lane(string key, int i) => + private static ulong Lane(ReadOnlySpan key, int i) => (ulong)key[i] | ((ulong)key[i + 1] << 16) | ((ulong)key[i + 2] << 32) @@ -263,7 +287,7 @@ private static ulong Lane(string key, int i) => /// 16 bits, the next char the high 16 bits. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static ulong Block(string key, int i) => + private static ulong Block(ReadOnlySpan key, int i) => (uint)key[i] | ((uint)key[i + 1] << 16); /// CityHash's ShiftMix: val ^ (val >> 47). @@ -296,7 +320,7 @@ private static ulong HashLen16(ulong u, ulong v, ulong mul) /// seeds, returning the two-word result. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static (ulong, ulong) WeakHashLen32WithSeeds(string key, int i, ulong a, ulong b) + private static (ulong, ulong) WeakHashLen32WithSeeds(ReadOnlySpan key, int i, ulong a, ulong b) { ulong w = Lane(key, i); ulong x = Lane(key, i + 4); diff --git a/src/Celerity.Hashing/StringCrc32Hasher.cs b/src/Celerity.Hashing/StringCrc32Hasher.cs index fabac101..3d936bdb 100644 --- a/src/Celerity.Hashing/StringCrc32Hasher.cs +++ b/src/Celerity.Hashing/StringCrc32Hasher.cs @@ -68,7 +68,7 @@ namespace Celerity.Hashing; /// with the empty-slot sentinel. /// /// -public struct StringCrc32Hasher : IHashProvider +public struct StringCrc32Hasher : IHashProvider, ISpanHashProvider { /// The reflected CRC-32 (ISO-HDLC / IEEE 802.3) generator polynomial. private const uint Polynomial = 0xEDB88320u; @@ -114,7 +114,22 @@ private static uint[] BuildTable() public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes the standard CRC-32 (ISO-HDLC / IEEE 802.3 / zlib) of the specified + /// character span over the full little-endian UTF-16 byte stream (both bytes of + /// every character). + /// + /// The characters to hash. + /// + /// The signed 32-bit CRC-32 of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { uint[] table = Table; uint crc = 0xFFFFFFFFu; diff --git a/src/Celerity.Hashing/StringDjb2AHasher.cs b/src/Celerity.Hashing/StringDjb2AHasher.cs index 477478be..fc935286 100644 --- a/src/Celerity.Hashing/StringDjb2AHasher.cs +++ b/src/Celerity.Hashing/StringDjb2AHasher.cs @@ -66,7 +66,7 @@ namespace Celerity.Hashing; /// sentinel. /// /// -public struct StringDjb2AHasher : IHashProvider +public struct StringDjb2AHasher : IHashProvider, ISpanHashProvider { /// /// Computes the djb2a (XOR-folding) hash of the specified string over the @@ -84,7 +84,21 @@ public struct StringDjb2AHasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes the djb2a (XOR-folding) hash of the specified character span over + /// the full little-endian UTF-16 byte stream (both bytes of every character). + /// + /// The characters to hash. + /// + /// The signed 32-bit djb2a hash of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { uint hash = 5381u; foreach (char c in key) { diff --git a/src/Celerity.Hashing/StringDjb2Hasher.cs b/src/Celerity.Hashing/StringDjb2Hasher.cs index 05e033e7..63e1cf6c 100644 --- a/src/Celerity.Hashing/StringDjb2Hasher.cs +++ b/src/Celerity.Hashing/StringDjb2Hasher.cs @@ -51,7 +51,7 @@ namespace Celerity.Hashing; /// the hasher, so this does not collide with the empty-slot sentinel. /// /// -public struct StringDjb2Hasher : IHashProvider +public struct StringDjb2Hasher : IHashProvider, ISpanHashProvider { /// /// Computes Daniel J. Bernstein's djb2 hash of the specified string over the @@ -69,7 +69,21 @@ public struct StringDjb2Hasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes Daniel J. Bernstein's djb2 hash of the specified character span + /// over the full little-endian UTF-16 byte stream (both bytes of every character). + /// + /// The characters to hash. + /// + /// The signed 32-bit djb2 hash of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { uint hash = 5381u; foreach (char c in key) { diff --git a/src/Celerity.Hashing/StringElfHasher.cs b/src/Celerity.Hashing/StringElfHasher.cs index 7d3863e4..493355e3 100644 --- a/src/Celerity.Hashing/StringElfHasher.cs +++ b/src/Celerity.Hashing/StringElfHasher.cs @@ -62,7 +62,7 @@ namespace Celerity.Hashing; /// calling the hasher, so this does not collide with the empty-slot sentinel. /// /// -public struct StringElfHasher : IHashProvider +public struct StringElfHasher : IHashProvider, ISpanHashProvider { /// /// Computes the PJW / ELF hash of the specified string over the full @@ -81,7 +81,22 @@ public struct StringElfHasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes the PJW / ELF hash of the specified character span over the full + /// little-endian UTF-16 byte stream (both bytes of every character). + /// + /// The characters to hash. + /// + /// The signed 32-bit ELF hash of (always in the range + /// [0, 0x0FFFFFFF], since the algorithm clears the top nibble) — the same + /// value returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { uint hash = 0u; foreach (char c in key) { diff --git a/src/Celerity.Hashing/StringFnV164Hasher.cs b/src/Celerity.Hashing/StringFnV164Hasher.cs index 4abda560..5181527c 100644 --- a/src/Celerity.Hashing/StringFnV164Hasher.cs +++ b/src/Celerity.Hashing/StringFnV164Hasher.cs @@ -65,11 +65,11 @@ namespace Celerity.Hashing; /// The algorithm carries 64 bits of state internally, so the type also implements /// : returns that state un-folded, which is /// what the probabilistic sketches want (see for why the extra -/// 32 bits matter there and not in a hash table). is unchanged — it is +/// 32 bits matter there and not in a hash table). is unchanged — it is /// exactly h ^ (h >> 32) of the 64-bit result. /// /// -public struct StringFnV164Hasher : IHashProvider, IHashProvider64 +public struct StringFnV164Hasher : IHashProvider, IHashProvider64, ISpanHashProvider { /// /// Computes the FNV-1 64-bit hash of the specified string over the full @@ -87,7 +87,23 @@ public struct StringFnV164Hasher : IHashProvider, IHashProvider64 + /// Computes the FNV-1 64-bit hash of the specified character span, xor-folded to a + /// signed 32-bit result. + /// + /// The characters to hash. + /// + /// The signed 32-bit, xor-folded FNV-1 hash of — the same + /// value returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { + ulong h64 = Hash64Core(key); return unchecked((int)(h64 ^ (h64 >> 32))); } @@ -108,7 +124,15 @@ public int Hash(string key) public ulong Hash64(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash64Core(key.AsSpan()); + } + // The single 64-bit body. Both Hash64(string) and the span-based Hash(ReadOnlySpan) + // route through it, so the string and span overloads cannot drift apart — the parity the + // ISpanHashProvider contract requires. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Hash64Core(ReadOnlySpan key) + { // The FNV-1 64-bit parameters (identical to FNV-1a 64-bit). const ulong fnvPrime = 1099511628211UL; const ulong offsetBasis = 14695981039346656037UL; diff --git a/src/Celerity.Hashing/StringFnV1A64Hasher.cs b/src/Celerity.Hashing/StringFnV1A64Hasher.cs index cca23992..cac080c5 100644 --- a/src/Celerity.Hashing/StringFnV1A64Hasher.cs +++ b/src/Celerity.Hashing/StringFnV1A64Hasher.cs @@ -45,11 +45,11 @@ namespace Celerity.Hashing; /// The algorithm carries 64 bits of state internally, so the type also implements /// : returns that state un-folded, which is /// what the probabilistic sketches want (see for why the extra -/// 32 bits matter there and not in a hash table). is unchanged — it is +/// 32 bits matter there and not in a hash table). is unchanged — it is /// exactly h ^ (h >> 32) of the 64-bit result. /// /// -public struct StringFnV1A64Hasher : IHashProvider, IHashProvider64 +public struct StringFnV1A64Hasher : IHashProvider, IHashProvider64, ISpanHashProvider { /// /// Computes the FNV-1a 64-bit hash of the specified string over the full @@ -67,7 +67,23 @@ public struct StringFnV1A64Hasher : IHashProvider, IHashProvider64 + /// Computes the FNV-1a 64-bit hash of the specified character span, xor-folded to a + /// signed 32-bit result. + /// + /// The characters to hash. + /// + /// The signed 32-bit, xor-folded FNV-1a hash of — the same + /// value returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { + ulong h64 = Hash64Core(key); return unchecked((int)(h64 ^ (h64 >> 32))); } @@ -88,7 +104,15 @@ public int Hash(string key) public ulong Hash64(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash64Core(key.AsSpan()); + } + // The single 64-bit body. Both Hash64(string) and the span-based Hash(ReadOnlySpan) + // route through it, so the string and span overloads cannot drift apart — the parity the + // ISpanHashProvider contract requires. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Hash64Core(ReadOnlySpan key) + { // The FNV-1a 64-bit parameters. const ulong fnvPrime = 1099511628211UL; const ulong offsetBasis = 14695981039346656037UL; diff --git a/src/Celerity.Hashing/StringFnV1AFullHasher.cs b/src/Celerity.Hashing/StringFnV1AFullHasher.cs index d4ebc4fd..c77a4721 100644 --- a/src/Celerity.Hashing/StringFnV1AFullHasher.cs +++ b/src/Celerity.Hashing/StringFnV1AFullHasher.cs @@ -41,7 +41,7 @@ namespace Celerity.Hashing; /// the hasher, so this does not collide with the empty-slot sentinel. /// /// -public struct StringFnV1AFullHasher : IHashProvider +public struct StringFnV1AFullHasher : IHashProvider, ISpanHashProvider { /// /// Computes the FNV-1a 32-bit hash of the specified string over the full @@ -59,7 +59,21 @@ public struct StringFnV1AFullHasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes the FNV-1a 32-bit hash of the specified character span over the full + /// little-endian UTF-16 byte stream (both bytes of every character). + /// + /// The characters to hash. + /// + /// The signed 32-bit FNV-1a hash of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { // The FNV-1a 32-bit parameters. const uint fnvPrime = 16777619u; const uint offsetBasis = 2166136261u; diff --git a/src/Celerity.Hashing/StringFnV1AHasher.cs b/src/Celerity.Hashing/StringFnV1AHasher.cs index 262d0711..aeb96b8b 100644 --- a/src/Celerity.Hashing/StringFnV1AHasher.cs +++ b/src/Celerity.Hashing/StringFnV1AHasher.cs @@ -11,7 +11,7 @@ namespace Celerity.Hashing; /// so strings that differ only in high bytes of non-ASCII characters may collide. /// For Unicode-heavy workloads, a full UTF-8 or UTF-16 hash is preferable. /// -public struct StringFnV1AHasher : IHashProvider +public struct StringFnV1AHasher : IHashProvider, ISpanHashProvider { /// /// Computes the FNV-1a 32-bit hash of the specified string. @@ -28,7 +28,20 @@ public struct StringFnV1AHasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes the FNV-1a 32-bit hash of the specified character span. + /// + /// The characters to hash. + /// + /// The signed 32-bit FNV-1a hash of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { // The FNV-1a 32-bit parameters const uint fnvPrime = 16777619; const uint offsetBasis = 2166136261; diff --git a/src/Celerity.Hashing/StringFnV1Hasher.cs b/src/Celerity.Hashing/StringFnV1Hasher.cs index 13ee81e0..31371bfa 100644 --- a/src/Celerity.Hashing/StringFnV1Hasher.cs +++ b/src/Celerity.Hashing/StringFnV1Hasher.cs @@ -53,7 +53,7 @@ namespace Celerity.Hashing; /// the hasher, so this does not collide with the empty-slot sentinel. /// /// -public struct StringFnV1Hasher : IHashProvider +public struct StringFnV1Hasher : IHashProvider, ISpanHashProvider { /// /// Computes the FNV-1 32-bit hash of the specified string over the full @@ -71,7 +71,21 @@ public struct StringFnV1Hasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes the FNV-1 32-bit hash of the specified character span over the full + /// little-endian UTF-16 byte stream (both bytes of every character). + /// + /// The characters to hash. + /// + /// The signed 32-bit FNV-1 hash of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { // The FNV-1 32-bit parameters (identical to FNV-1a). const uint fnvPrime = 16777619u; const uint offsetBasis = 2166136261u; diff --git a/src/Celerity.Hashing/StringHalfSipHash24Hasher.cs b/src/Celerity.Hashing/StringHalfSipHash24Hasher.cs index bd27a5dd..c2a0c4ca 100644 --- a/src/Celerity.Hashing/StringHalfSipHash24Hasher.cs +++ b/src/Celerity.Hashing/StringHalfSipHash24Hasher.cs @@ -73,7 +73,7 @@ namespace Celerity.Hashing; /// hasher, so this does not collide with the empty-slot sentinel. /// /// -public struct StringHalfSipHash24Hasher : IHashProvider +public struct StringHalfSipHash24Hasher : IHashProvider, ISpanHashProvider { // Canonical HalfSipHash reference key (test-vector key, bytes 00..07), read as // two little-endian 32-bit halves. Fixed because collections build the hasher @@ -103,7 +103,22 @@ public struct StringHalfSipHash24Hasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes the HalfSipHash-2-4 hash of the specified character span over its native + /// little-endian UTF-16 byte stream (using this type's fixed built-in key), returning + /// the native 32-bit result as a signed integer. + /// + /// The characters to hash. + /// + /// The signed 32-bit HalfSipHash-2-4 hash of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { int charLen = key.Length; // count of UTF-16 code units (chars) uint byteLen = (uint)charLen * 2U; // HalfSipHash operates on the byte length @@ -156,7 +171,7 @@ public int Hash(string key) /// HalfSipHash would read over the native little-endian UTF-16 stream. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static uint Word(string key, int i) => + private static uint Word(ReadOnlySpan key, int i) => key[i] | ((uint)key[i + 1] << 16); /// diff --git a/src/Celerity.Hashing/StringHighwayHash64Hasher.cs b/src/Celerity.Hashing/StringHighwayHash64Hasher.cs index efbb0122..c4da046b 100644 --- a/src/Celerity.Hashing/StringHighwayHash64Hasher.cs +++ b/src/Celerity.Hashing/StringHighwayHash64Hasher.cs @@ -80,11 +80,11 @@ namespace Celerity.Hashing; /// The algorithm carries 64 bits of state internally, so the type also implements /// : returns that state un-folded, which is /// what the probabilistic sketches want (see for why the extra -/// 32 bits matter there and not in a hash table). is unchanged — it is +/// 32 bits matter there and not in a hash table). is unchanged — it is /// exactly h ^ (h >> 32) of the 64-bit result. /// /// -public struct StringHighwayHash64Hasher : IHashProvider, IHashProvider64 +public struct StringHighwayHash64Hasher : IHashProvider, IHashProvider64, ISpanHashProvider { // Fixed 256-bit key: the canonical HighwayHash reference test key (bytes // 00..1f read as four little-endian 64-bit words). Fixed because collections @@ -110,7 +110,23 @@ public struct StringHighwayHash64Hasher : IHashProvider, IHashProvider64 [MethodImpl(MethodImplOptions.AggressiveInlining)] public int Hash(string key) { - ulong h64 = Hash64(key); + ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + + /// + /// Computes the HighwayHash-64 hash of the specified character span (using this type's + /// fixed built-in key), xor-folded to a signed 32-bit result. + /// + /// The characters to hash. + /// + /// The signed 32-bit, xor-folded HighwayHash-64 hash of — the same + /// value returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { + ulong h64 = Hash64Core(key); return unchecked((int)(h64 ^ (h64 >> 32))); } @@ -131,7 +147,15 @@ public int Hash(string key) public ulong Hash64(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash64Core(key.AsSpan()); + } + // The single 64-bit body. Both Hash64(string) and the span-based Hash(ReadOnlySpan) + // route through it, so the string and span overloads cannot drift apart — the parity the + // ISpanHashProvider contract requires. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Hash64Core(ReadOnlySpan key) + { // Four lanes of 64-bit state, held on the stack (no heap allocation). Span v0 = stackalloc ulong[4]; Span v1 = stackalloc ulong[4]; @@ -277,7 +301,7 @@ private static void PermuteAndUpdate( /// [MethodImpl(MethodImplOptions.AggressiveInlining)] private static void UpdateRemainder( - string key, int tailStartChar, int sizeMod32, + ReadOnlySpan key, int tailStartChar, int sizeMod32, Span v0, Span v1, Span mul0, Span mul1) { int sizeMod4 = sizeMod32 & 3; // 0 or 2 on an always-even byte stream @@ -340,7 +364,7 @@ private static void Rotate32By(uint count, Span v1) /// over the native little-endian UTF-16 stream. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static ulong Lane(string key, int i) => + private static ulong Lane(ReadOnlySpan key, int i) => (ulong)key[i] | ((ulong)key[i + 1] << 16) | ((ulong)key[i + 2] << 32) @@ -353,7 +377,7 @@ private static ulong Lane(string key, int i) => /// offsets its high byte. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static byte TailByte(string key, int tailStartChar, int b) => + private static byte TailByte(ReadOnlySpan key, int tailStartChar, int b) => (byte)(key[tailStartChar + (b >> 1)] >> ((b & 1) << 3)); /// Rotates a 64-bit value by 32 bits (swaps its two 32-bit halves). diff --git a/src/Celerity.Hashing/StringJenkinsOaatHasher.cs b/src/Celerity.Hashing/StringJenkinsOaatHasher.cs index 0d7f4bf4..ac067e41 100644 --- a/src/Celerity.Hashing/StringJenkinsOaatHasher.cs +++ b/src/Celerity.Hashing/StringJenkinsOaatHasher.cs @@ -50,7 +50,7 @@ namespace Celerity.Hashing; /// does not collide with the empty-slot sentinel. /// /// -public struct StringJenkinsOaatHasher : IHashProvider +public struct StringJenkinsOaatHasher : IHashProvider, ISpanHashProvider { /// /// Computes Bob Jenkins' one-at-a-time hash of the specified string over the @@ -68,7 +68,21 @@ public struct StringJenkinsOaatHasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes Bob Jenkins' one-at-a-time hash of the specified character span over + /// the full little-endian UTF-16 byte stream (both bytes of every character). + /// + /// The characters to hash. + /// + /// The signed 32-bit one-at-a-time hash of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { uint hash = 0u; foreach (char c in key) { diff --git a/src/Celerity.Hashing/StringMetroHash64Hasher.cs b/src/Celerity.Hashing/StringMetroHash64Hasher.cs index 94d25869..2b6efbb5 100644 --- a/src/Celerity.Hashing/StringMetroHash64Hasher.cs +++ b/src/Celerity.Hashing/StringMetroHash64Hasher.cs @@ -57,11 +57,11 @@ namespace Celerity.Hashing; /// The algorithm carries 64 bits of state internally, so the type also implements /// : returns that state un-folded, which is /// what the probabilistic sketches want (see for why the extra -/// 32 bits matter there and not in a hash table). is unchanged — it is +/// 32 bits matter there and not in a hash table). is unchanged — it is /// exactly h ^ (h >> 32) of the 64-bit result. /// /// -public struct StringMetroHash64Hasher : IHashProvider, IHashProvider64 +public struct StringMetroHash64Hasher : IHashProvider, IHashProvider64, ISpanHashProvider { // metrohash64_1 mixing constants (J. Andrew Rogers, public-domain reference). private const ulong K0 = 0xC83A91E1UL; @@ -85,7 +85,23 @@ public struct StringMetroHash64Hasher : IHashProvider, IHashProvider64 + /// Computes the MetroHash64 hash of the specified character span, xor-folded to a + /// signed 32-bit result. + /// + /// The characters to hash. + /// + /// The signed 32-bit, xor-folded MetroHash64 hash of — the same + /// value returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { + ulong h64 = Hash64Core(key); return unchecked((int)(h64 ^ (h64 >> 32))); } @@ -106,8 +122,16 @@ public int Hash(string key) public ulong Hash64(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash64Core(key.AsSpan()); + } - int length = key.Length; // count of UTF-16 code units (chars) + // The single 64-bit body. Both Hash64(string) and the span-based Hash(ReadOnlySpan) + // route through it, so the string and span overloads cannot drift apart — the parity the + // ISpanHashProvider contract requires. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Hash64Core(ReadOnlySpan key) + { + int length = key.Length; // count of UTF-16 code units (chars) ulong byteLength = (ulong)length * 2UL; // MetroHash mixes in the byte length // Seed 0: hash starts at ((0 + k2) * k0) + len. @@ -185,7 +209,7 @@ public ulong Hash64(string key) /// read over the native little-endian UTF-16 stream. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static ulong Lane(string key, int i) => + private static ulong Lane(ReadOnlySpan key, int i) => (ulong)key[i] | ((ulong)key[i + 1] << 16) | ((ulong)key[i + 2] << 32) @@ -197,6 +221,6 @@ private static ulong Lane(string key, int i) => /// 16 bits, the next char the high 16 bits. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static uint Block(string key, int i) => + private static uint Block(ReadOnlySpan key, int i) => (uint)key[i] | ((uint)key[i + 1] << 16); } diff --git a/src/Celerity.Hashing/StringMurmur2Hasher.cs b/src/Celerity.Hashing/StringMurmur2Hasher.cs index 9e188a0d..bdd3d2ef 100644 --- a/src/Celerity.Hashing/StringMurmur2Hasher.cs +++ b/src/Celerity.Hashing/StringMurmur2Hasher.cs @@ -56,7 +56,7 @@ namespace Celerity.Hashing; /// sentinel. /// /// -public struct StringMurmur2Hasher : IHashProvider +public struct StringMurmur2Hasher : IHashProvider, ISpanHashProvider { private const uint M = 0x5bd1e995u; private const int R = 24; @@ -78,7 +78,21 @@ public struct StringMurmur2Hasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes the MurmurHash2 (32-bit, seed 0) hash of the specified character + /// span over the full little-endian UTF-16 byte stream (both bytes of every character). + /// + /// The characters to hash. + /// + /// The signed 32-bit MurmurHash2 hash of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { int length = key.Length; // MurmurHash2 seeds the accumulator with seed ^ byteLength; seed is 0 here, diff --git a/src/Celerity.Hashing/StringMurmur3Hasher.cs b/src/Celerity.Hashing/StringMurmur3Hasher.cs index fd29f91a..97f4a8f8 100644 --- a/src/Celerity.Hashing/StringMurmur3Hasher.cs +++ b/src/Celerity.Hashing/StringMurmur3Hasher.cs @@ -40,7 +40,7 @@ namespace Celerity.Hashing; /// calling the hasher, so this does not collide with the empty-slot sentinel. /// /// -public struct StringMurmur3Hasher : IHashProvider +public struct StringMurmur3Hasher : IHashProvider, ISpanHashProvider { private const uint C1 = 0xcc9e2d51u; private const uint C2 = 0x1b873593u; @@ -61,7 +61,21 @@ public struct StringMurmur3Hasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes the MurmurHash3 x86_32 hash of the specified character span over its + /// UTF-16 code units. + /// + /// The characters to hash. + /// + /// The signed 32-bit MurmurHash3 hash of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { uint h = 0; int length = key.Length; diff --git a/src/Celerity.Hashing/StringSdbmHasher.cs b/src/Celerity.Hashing/StringSdbmHasher.cs index cdcea7cf..3962d448 100644 --- a/src/Celerity.Hashing/StringSdbmHasher.cs +++ b/src/Celerity.Hashing/StringSdbmHasher.cs @@ -59,7 +59,7 @@ namespace Celerity.Hashing; /// calling the hasher, so this does not collide with the empty-slot sentinel. /// /// -public struct StringSdbmHasher : IHashProvider +public struct StringSdbmHasher : IHashProvider, ISpanHashProvider { /// /// Computes the sdbm hash of the specified string over the full little-endian @@ -77,7 +77,21 @@ public struct StringSdbmHasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes the sdbm hash of the specified character span over the full + /// little-endian UTF-16 byte stream (both bytes of every character). + /// + /// The characters to hash. + /// + /// The signed 32-bit sdbm hash of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { uint hash = 0u; foreach (char c in key) { diff --git a/src/Celerity.Hashing/StringSipHash13Hasher.cs b/src/Celerity.Hashing/StringSipHash13Hasher.cs index e10df40c..4287ed68 100644 --- a/src/Celerity.Hashing/StringSipHash13Hasher.cs +++ b/src/Celerity.Hashing/StringSipHash13Hasher.cs @@ -81,11 +81,11 @@ namespace Celerity.Hashing; /// The algorithm carries 64 bits of state internally, so the type also implements /// : returns that state un-folded, which is /// what the probabilistic sketches want (see for why the extra -/// 32 bits matter there and not in a hash table). is unchanged — it is +/// 32 bits matter there and not in a hash table). is unchanged — it is /// exactly h ^ (h >> 32) of the 64-bit result. /// /// -public struct StringSipHash13Hasher : IHashProvider, IHashProvider64 +public struct StringSipHash13Hasher : IHashProvider, IHashProvider64, ISpanHashProvider { // Canonical SipHash reference key (RFC-draft test-vector key, bytes 00..0f), // read as two little-endian 64-bit halves. Fixed because collections build the @@ -115,7 +115,23 @@ public struct StringSipHash13Hasher : IHashProvider, IHashProvider64 + /// Computes the SipHash-1-3 hash of the specified character span (using this type's + /// fixed built-in key), xor-folded to a signed 32-bit result. + /// + /// The characters to hash. + /// + /// The signed 32-bit, xor-folded SipHash-1-3 hash of — the same + /// value returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { + ulong h64 = Hash64Core(key); return unchecked((int)(h64 ^ (h64 >> 32))); } @@ -136,8 +152,16 @@ public int Hash(string key) public ulong Hash64(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash64Core(key.AsSpan()); + } - int charLen = key.Length; // count of UTF-16 code units (chars) + // The single 64-bit body. Both Hash64(string) and the span-based Hash(ReadOnlySpan) + // route through it, so the string and span overloads cannot drift apart — the parity the + // ISpanHashProvider contract requires. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Hash64Core(ReadOnlySpan key) + { + int charLen = key.Length; // count of UTF-16 code units (chars) ulong byteLen = (ulong)charLen * 2UL; // SipHash operates on the byte length ulong v0 = Init0 ^ K0; @@ -190,7 +214,7 @@ public ulong Hash64(string key) /// would read over the native little-endian UTF-16 stream. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static ulong Word(string key, int i) => + private static ulong Word(ReadOnlySpan key, int i) => (ulong)key[i] | ((ulong)key[i + 1] << 16) | ((ulong)key[i + 2] << 32) diff --git a/src/Celerity.Hashing/StringSipHash24Hasher.cs b/src/Celerity.Hashing/StringSipHash24Hasher.cs index 51370f45..a7d1d525 100644 --- a/src/Celerity.Hashing/StringSipHash24Hasher.cs +++ b/src/Celerity.Hashing/StringSipHash24Hasher.cs @@ -75,11 +75,11 @@ namespace Celerity.Hashing; /// The algorithm carries 64 bits of state internally, so the type also implements /// : returns that state un-folded, which is /// what the probabilistic sketches want (see for why the extra -/// 32 bits matter there and not in a hash table). is unchanged — it is +/// 32 bits matter there and not in a hash table). is unchanged — it is /// exactly h ^ (h >> 32) of the 64-bit result. /// /// -public struct StringSipHash24Hasher : IHashProvider, IHashProvider64 +public struct StringSipHash24Hasher : IHashProvider, IHashProvider64, ISpanHashProvider { // Canonical SipHash reference key (RFC-draft test-vector key, bytes 00..0f), // read as two little-endian 64-bit halves. Fixed because collections build the @@ -109,7 +109,23 @@ public struct StringSipHash24Hasher : IHashProvider, IHashProvider64 + /// Computes the SipHash-2-4 hash of the specified character span (using this type's + /// fixed built-in key), xor-folded to a signed 32-bit result. + /// + /// The characters to hash. + /// + /// The signed 32-bit, xor-folded SipHash-2-4 hash of — the same + /// value returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { + ulong h64 = Hash64Core(key); return unchecked((int)(h64 ^ (h64 >> 32))); } @@ -130,8 +146,16 @@ public int Hash(string key) public ulong Hash64(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash64Core(key.AsSpan()); + } - int charLen = key.Length; // count of UTF-16 code units (chars) + // The single 64-bit body. Both Hash64(string) and the span-based Hash(ReadOnlySpan) + // route through it, so the string and span overloads cannot drift apart — the parity the + // ISpanHashProvider contract requires. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Hash64Core(ReadOnlySpan key) + { + int charLen = key.Length; // count of UTF-16 code units (chars) ulong byteLen = (ulong)charLen * 2UL; // SipHash operates on the byte length ulong v0 = Init0 ^ K0; @@ -186,7 +210,7 @@ public ulong Hash64(string key) /// would read over the native little-endian UTF-16 stream. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static ulong Word(string key, int i) => + private static ulong Word(ReadOnlySpan key, int i) => (ulong)key[i] | ((ulong)key[i + 1] << 16) | ((ulong)key[i + 2] << 32) diff --git a/src/Celerity.Hashing/StringXxHash32Hasher.cs b/src/Celerity.Hashing/StringXxHash32Hasher.cs index 497f7b67..1903b5ca 100644 --- a/src/Celerity.Hashing/StringXxHash32Hasher.cs +++ b/src/Celerity.Hashing/StringXxHash32Hasher.cs @@ -47,7 +47,7 @@ namespace Celerity.Hashing; /// the hasher, so this does not collide with the empty-slot sentinel. /// /// -public struct StringXxHash32Hasher : IHashProvider +public struct StringXxHash32Hasher : IHashProvider, ISpanHashProvider { private const uint Prime1 = 2654435761u; private const uint Prime2 = 2246822519u; @@ -71,7 +71,21 @@ public struct StringXxHash32Hasher : IHashProvider public int Hash(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash(key.AsSpan()); + } + /// + /// Computes the xxHash32 hash (seed 0) of the specified character span over + /// its native little-endian UTF-16 byte stream. + /// + /// The characters to hash. + /// + /// The signed 32-bit xxHash32 hash of — the same value + /// returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { int length = key.Length; // count of UTF-16 code units (chars) uint byteLength = (uint)length * 2u; // XXH32 mixes in the byte length @@ -147,7 +161,7 @@ public int Hash(string key) /// byte-oriented XXH32 would read over the native little-endian UTF-16 stream. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static uint Block(string key, int i) => + private static uint Block(ReadOnlySpan key, int i) => (uint)key[i] | ((uint)key[i + 1] << 16); /// The XXH32 accumulator round: rotl(acc + lane * PRIME32_2, 13) * PRIME32_1. diff --git a/src/Celerity.Hashing/StringXxHash3Hasher.cs b/src/Celerity.Hashing/StringXxHash3Hasher.cs index 82e26850..3351357c 100644 --- a/src/Celerity.Hashing/StringXxHash3Hasher.cs +++ b/src/Celerity.Hashing/StringXxHash3Hasher.cs @@ -63,11 +63,11 @@ namespace Celerity.Hashing; /// The algorithm carries 64 bits of state internally, so the type also implements /// : returns that state un-folded, which is /// what the probabilistic sketches want (see for why the extra -/// 32 bits matter there and not in a hash table). is unchanged — it is +/// 32 bits matter there and not in a hash table). is unchanged — it is /// exactly h ^ (h >> 32) of the 64-bit result. /// /// -public struct StringXxHash3Hasher : IHashProvider, IHashProvider64 +public struct StringXxHash3Hasher : IHashProvider, IHashProvider64, ISpanHashProvider { private const ulong Prime64_1 = 0x9E3779B185EBCA87UL; private const ulong Prime64_2 = 0xC2B2AE3D27D4EB4FUL; @@ -124,7 +124,23 @@ public struct StringXxHash3Hasher : IHashProvider, IHashProvider64 + /// Computes the XXH3-64 hash of the specified character span, xor-folded to a + /// signed 32-bit result. + /// + /// The characters to hash. + /// + /// The signed 32-bit, xor-folded XXH3-64 hash of — the same + /// value returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { + ulong h64 = Hash64Core(key); return unchecked((int)(h64 ^ (h64 >> 32))); } @@ -145,7 +161,15 @@ public int Hash(string key) public ulong Hash64(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash64Core(key.AsSpan()); + } + // The single 64-bit body. Both Hash64(string) and the span-based Hash(ReadOnlySpan) + // route through it, so the string and span overloads cannot drift apart — the parity the + // ISpanHashProvider contract requires. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Hash64Core(ReadOnlySpan key) + { // n = count of UTF-16 code units (chars); the conceptual byte length is 2 * n, // which is always even — so every XXH3 read in the 4-to-240-byte and long // paths lands on a char boundary. The single odd-aligned read in XXH3 (the @@ -197,7 +221,7 @@ public ulong Hash64(string key) // ── Short-length code paths (seed 0) ────────────────────────────────────── [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static ulong Len4to8(string s, int n) + private static ulong Len4to8(ReadOnlySpan s, int n) { ulong byteLength = (ulong)(2 * n); uint input1 = Key32(s, 0); // bytes [0, 4) @@ -208,7 +232,7 @@ private static ulong Len4to8(string s, int n) } [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static ulong Len9to16(string s, int n) + private static ulong Len9to16(ReadOnlySpan s, int n) { ulong byteLength = (ulong)(2 * n); ulong bitflip1 = Sec64(24) ^ Sec64(32); @@ -222,7 +246,7 @@ private static ulong Len9to16(string s, int n) return Xxh3Avalanche(acc); } - private static ulong Len17to128(string s, int n) + private static ulong Len17to128(ReadOnlySpan s, int n) { int len = 2 * n; // byte length, in (16, 128] ulong acc = (ulong)len * Prime64_1; @@ -250,7 +274,7 @@ private static ulong Len17to128(string s, int n) return Xxh3Avalanche(acc); } - private static ulong Len129to240(string s, int n) + private static ulong Len129to240(ReadOnlySpan s, int n) { int len = 2 * n; // byte length, in (128, 240] ulong acc = (ulong)len * Prime64_1; @@ -276,7 +300,7 @@ private static ulong Len129to240(string s, int n) // ── Long code path (> 240 bytes), default secret / seed 0 ───────────────── - private static ulong HashLong(string s, int n) + private static ulong HashLong(ReadOnlySpan s, int n) { long byteLength = 2L * n; @@ -311,7 +335,7 @@ private static ulong HashLong(string s, int n) } [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void Accumulate(Span acc, string s, int startChar, int nbStripes) + private static void Accumulate(Span acc, ReadOnlySpan s, int startChar, int nbStripes) { for (int i = 0; i < nbStripes; i++) { @@ -321,7 +345,7 @@ private static void Accumulate(Span acc, string s, int startChar, int nbS } [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static void AccumulateStripe(Span acc, string s, int stripeChar, int secretOffset) + private static void AccumulateStripe(Span acc, ReadOnlySpan s, int stripeChar, int secretOffset) { for (int lane = 0; lane < 8; lane++) { @@ -367,7 +391,7 @@ private static ulong MergeAccs(Span acc, int secretOffset, ulong start) /// through a 128-bit multiply. Seed is 0. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static ulong Mix16B(string s, int ci, int so) + private static ulong Mix16B(ReadOnlySpan s, int ci, int so) { ulong inputLo = Key64(s, ci); ulong inputHi = Key64(s, ci + 4); @@ -420,7 +444,7 @@ private static ulong Rrmxmx(ulong h, ulong length) /// byte-oriented XXH3 would read over the native little-endian UTF-16 stream. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static ulong Key64(string s, int ci) => + private static ulong Key64(ReadOnlySpan s, int ci) => (ulong)s[ci] | ((ulong)s[ci + 1] << 16) | ((ulong)s[ci + 2] << 32) @@ -431,7 +455,7 @@ private static ulong Key64(string s, int ci) => /// starting at char index . /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static uint Key32(string s, int ci) => + private static uint Key32(ReadOnlySpan s, int ci) => (uint)s[ci] | ((uint)s[ci + 1] << 16); [MethodImpl(MethodImplOptions.AggressiveInlining)] diff --git a/src/Celerity.Hashing/StringXxHash64Hasher.cs b/src/Celerity.Hashing/StringXxHash64Hasher.cs index 5dfde73e..999979a9 100644 --- a/src/Celerity.Hashing/StringXxHash64Hasher.cs +++ b/src/Celerity.Hashing/StringXxHash64Hasher.cs @@ -58,11 +58,11 @@ namespace Celerity.Hashing; /// The algorithm carries 64 bits of state internally, so the type also implements /// : returns that state un-folded, which is /// what the probabilistic sketches want (see for why the extra -/// 32 bits matter there and not in a hash table). is unchanged — it is +/// 32 bits matter there and not in a hash table). is unchanged — it is /// exactly h ^ (h >> 32) of the 64-bit result. /// /// -public struct StringXxHash64Hasher : IHashProvider, IHashProvider64 +public struct StringXxHash64Hasher : IHashProvider, IHashProvider64, ISpanHashProvider { private const ulong Prime1 = 11400714785074694791UL; private const ulong Prime2 = 14029467366897019727UL; @@ -86,7 +86,23 @@ public struct StringXxHash64Hasher : IHashProvider, IHashProvider64 + /// Computes the xxHash64 (XXH64, seed 0) hash of the specified character span, + /// xor-folded to a signed 32-bit result. + /// + /// The characters to hash. + /// + /// The signed 32-bit, xor-folded xxHash64 hash of — the same + /// value returns for a string with the same contents. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public int Hash(ReadOnlySpan key) + { + ulong h64 = Hash64Core(key); return unchecked((int)(h64 ^ (h64 >> 32))); } @@ -107,8 +123,16 @@ public int Hash(string key) public ulong Hash64(string key) { ArgumentNullException.ThrowIfNull(key); + return Hash64Core(key.AsSpan()); + } - int length = key.Length; // count of UTF-16 code units (chars) + // The single 64-bit body. Both Hash64(string) and the span-based Hash(ReadOnlySpan) + // route through it, so the string and span overloads cannot drift apart — the parity the + // ISpanHashProvider contract requires. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static ulong Hash64Core(ReadOnlySpan key) + { + int length = key.Length; // count of UTF-16 code units (chars) ulong byteLength = (ulong)length * 2UL; // XXH64 mixes in the byte length ulong h64; @@ -200,7 +224,7 @@ public ulong Hash64(string key) /// over the native little-endian UTF-16 stream. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static ulong Lane(string key, int i) => + private static ulong Lane(ReadOnlySpan key, int i) => (ulong)key[i] | ((ulong)key[i + 1] << 16) | ((ulong)key[i + 2] << 32) @@ -212,7 +236,7 @@ private static ulong Lane(string key, int i) => /// 16 bits, the next char the high 16 bits. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static uint Block(string key, int i) => + private static uint Block(ReadOnlySpan key, int i) => (uint)key[i] | ((uint)key[i + 1] << 16); /// The XXH64 accumulator round: rotl(acc + lane * PRIME64_2, 31) * PRIME64_1. diff --git a/src/Celerity.Tests/Collections/SpanLookupTests.cs b/src/Celerity.Tests/Collections/SpanLookupTests.cs new file mode 100644 index 00000000..4ae77abc --- /dev/null +++ b/src/Celerity.Tests/Collections/SpanLookupTests.cs @@ -0,0 +1,311 @@ +using Celerity.Collections; +using Celerity.Hashing; + +namespace Celerity.Tests.Collections; + +/// +/// Cross-collection tests for the span-keyed lookup surface: the same check run across every +/// string-keyed type that gained it — , +/// , , +/// , and . +/// +/// +/// The contract under test is that the span path and the string path are indistinguishable: +/// same hits, same misses, same values. Every row therefore asserts the span result +/// against the type's own string overload rather than against a hard-coded +/// expectation, so a divergence shows up wherever it is introduced. +/// +public class SpanLookupTests +{ + // A key set that exercises the corners: the empty string, keys that are prefixes of one + // another, keys differing only in a high byte (which the default low-byte hasher collides), + // and a surrogate pair. + private static readonly string[] Keys = + [ + "", + "a", + "ab", + "abc", + "alpha", + "alphabet", + "A", + "Ł", // U+0141 — collides with "A" under the low-byte StringFnV1AHasher + "日本語", + "emoji-\U0001F600", // surrogate pair + "x\0y", // embedded NUL + new string('q', 300), + ]; + + private static readonly string[] Misses = + [ + "z", + "alph", + "alphabets", + "Ń", + "emoji-\U0001F601", + new string('q', 299), + ]; + + private static KeyValuePair[] Pairs() => + Keys.Select((k, i) => new KeyValuePair(k, i)).ToArray(); + + // ── FrozenCelerityDictionary ────────────────────────────────────────────── + + [Fact] + public void TryGetValue_ShouldAgreeWithStringOverload_WhenFrozenCelerityDictionary() + { + var dict = new FrozenCelerityDictionary(Pairs()); + + foreach (string key in Keys.Concat(Misses)) + { + bool expected = dict.TryGetValue(key, out int expectedValue); + bool actual = dict.TryGetValue(key.AsSpan(), out int actualValue); + + Assert.Equal(expected, actual); + Assert.Equal(expectedValue, actualValue); + Assert.Equal(dict.ContainsKey(key), dict.ContainsKey(key.AsSpan())); + } + } + + [Fact] + public void TryGetValue_ShouldAgreeWithStringOverload_WhenFrozenCelerityDictionaryFallsBackToProbing() + { + // "A" and "Ł" share a raw code under the low-byte hasher, so no seed can separate them + // and the build takes the linear-probing fallback. The span probe must mirror it. + var dict = new FrozenCelerityDictionary(Pairs()); + Assert.False(dict.IsPerfectlyHashed); + + foreach (string key in Keys.Concat(Misses)) + { + bool expected = dict.TryGetValue(key, out int expectedValue); + bool actual = dict.TryGetValue(key.AsSpan(), out int actualValue); + + Assert.Equal(expected, actual); + Assert.Equal(expectedValue, actualValue); + } + } + + [Fact] + public void TryGetValue_ShouldNotMatchTheNullKey_WhenSpanIsEmpty() + { + // A span has no null state, so an empty span is the key "" and never the out-of-band + // null key. Here "" is absent and null is present: the empty span must miss. + var dict = new FrozenCelerityDictionary( + [new KeyValuePair(null!, 7), new KeyValuePair("a", 1)]); + + Assert.True(dict.TryGetValue((string)null!, out int viaNull)); + Assert.Equal(7, viaNull); + Assert.False(dict.TryGetValue(ReadOnlySpan.Empty, out _)); + Assert.False(dict.ContainsKey(ReadOnlySpan.Empty)); + } + + [Fact] + public void TryGetValue_ShouldThrowArgumentNullException_WhenFrozenCelerityDictionaryIsNull() + { + FrozenCelerityDictionary dict = null!; + Assert.Throws(() => dict.TryGetValue("a".AsSpan(), out int _)); + Assert.Throws(() => dict.ContainsKey("a".AsSpan())); + } + + // ── FrozenCeleritySet ───────────────────────────────────────────────────── + + [Fact] + public void Contains_ShouldAgreeWithStringOverload_WhenFrozenCeleritySet() + { + var set = new FrozenCeleritySet(Keys); + + foreach (string key in Keys.Concat(Misses)) + Assert.Equal(set.Contains(key), set.Contains(key.AsSpan())); + } + + [Fact] + public void Contains_ShouldAgreeWithStringOverload_WhenFrozenCeleritySetFallsBackToProbing() + { + var set = new FrozenCeleritySet(Keys); + Assert.False(set.IsPerfectlyHashed); + + foreach (string key in Keys.Concat(Misses)) + Assert.Equal(set.Contains(key), set.Contains(key.AsSpan())); + } + + [Fact] + public void Contains_ShouldThrowArgumentNullException_WhenFrozenCeleritySetIsNull() + { + FrozenCeleritySet set = null!; + Assert.Throws(() => set.Contains("a".AsSpan())); + } + + // ── CelerityDictionary ──────────────────────────────────────────────────── + + [Fact] + public void TryGetValue_ShouldAgreeWithStringOverload_WhenCelerityDictionary() + { + var dict = new CelerityDictionary(); + foreach (KeyValuePair pair in Pairs()) + dict.Add(pair.Key, pair.Value); + + foreach (string key in Keys.Concat(Misses)) + { + bool expected = dict.TryGetValue(key, out int expectedValue); + bool actual = dict.TryGetValue(key.AsSpan(), out int actualValue); + + Assert.Equal(expected, actual); + Assert.Equal(expectedValue, actualValue); + Assert.Equal(dict.ContainsKey(key), dict.ContainsKey(key.AsSpan())); + } + } + + [Fact] + public void TryGetValue_ShouldSeeSubsequentMutations_WhenCelerityDictionary() + { + var dict = new CelerityDictionary(); + Assert.False(dict.ContainsKey("late".AsSpan())); + + dict.Add("late", 42); + Assert.True(dict.TryGetValue("late".AsSpan(), out int value)); + Assert.Equal(42, value); + + dict.Remove("late"); + Assert.False(dict.TryGetValue("late".AsSpan(), out _)); + } + + [Fact] + public void TryGetValue_ShouldSurviveAResize_WhenCelerityDictionary() + { + // Grow well past the initial threshold so the span probe runs against a rehashed table. + var dict = new CelerityDictionary(capacity: 4); + for (int i = 0; i < 500; i++) + dict.Add($"key-{i}", i); + + for (int i = 0; i < 500; i++) + { + Assert.True(dict.TryGetValue($"key-{i}".AsSpan(), out int value)); + Assert.Equal(i, value); + } + + Assert.False(dict.TryGetValue("key-500".AsSpan(), out _)); + } + + [Fact] + public void TryGetValue_ShouldNotMatchTheNullKey_WhenCelerityDictionarySpanIsEmpty() + { + var dict = new CelerityDictionary(); + dict.Add(null!, 7); + + Assert.True(dict.TryGetValue((string)null!, out int viaNull)); + Assert.Equal(7, viaNull); + Assert.False(dict.TryGetValue(ReadOnlySpan.Empty, out _)); + + dict.Add(string.Empty, 9); + Assert.True(dict.TryGetValue(ReadOnlySpan.Empty, out int viaSpan)); + Assert.Equal(9, viaSpan); + } + + [Fact] + public void TryGetValue_ShouldThrowArgumentNullException_WhenCelerityDictionaryIsNull() + { + CelerityDictionary dict = null!; + Assert.Throws(() => dict.TryGetValue("a".AsSpan(), out int _)); + Assert.Throws(() => dict.ContainsKey("a".AsSpan())); + } + + // ── CeleritySet ─────────────────────────────────────────────────────────── + + [Fact] + public void Contains_ShouldAgreeWithStringOverload_WhenCeleritySet() + { + var set = new CeleritySet(); + foreach (string key in Keys) + set.Add(key); + + foreach (string key in Keys.Concat(Misses)) + Assert.Equal(set.Contains(key), set.Contains(key.AsSpan())); + } + + [Fact] + public void Contains_ShouldSurviveAResize_WhenCeleritySet() + { + var set = new CeleritySet(capacity: 4); + for (int i = 0; i < 500; i++) + set.Add($"item-{i}"); + + for (int i = 0; i < 500; i++) + Assert.True(set.Contains($"item-{i}".AsSpan())); + + Assert.False(set.Contains("item-500".AsSpan())); + } + + [Fact] + public void Contains_ShouldThrowArgumentNullException_WhenCeleritySetIsNull() + { + CeleritySet set = null!; + Assert.Throws(() => set.Contains("a".AsSpan())); + } + + // ── Trie ────────────────────────────────────────────────────────────────── + + [Fact] + public void TryGetValue_ShouldAgreeWithStringOverload_WhenTrie() + { + var trie = new Trie(); + foreach (KeyValuePair pair in Pairs()) + trie.Add(pair.Key, pair.Value); + + foreach (string key in Keys.Concat(Misses)) + { + bool expected = trie.TryGetValue(key, out int expectedValue); + bool actual = trie.TryGetValue(key.AsSpan(), out int actualValue); + + Assert.Equal(expected, actual); + Assert.Equal(expectedValue, actualValue); + Assert.Equal(trie.ContainsKey(key), trie.ContainsKey(key.AsSpan())); + Assert.Equal(trie.ContainsPrefix(key), trie.ContainsPrefix(key.AsSpan())); + } + } + + [Fact] + public void ContainsPrefix_ShouldAgreeWithStringOverload_WhenTriePrefixIsPartial() + { + var trie = new Trie { ["alphabet"] = 1 }; + + Assert.True(trie.ContainsPrefix("alph".AsSpan())); + Assert.False(trie.ContainsKey("alph".AsSpan())); + Assert.True(trie.ContainsPrefix(ReadOnlySpan.Empty)); + Assert.False(trie.ContainsPrefix("beta".AsSpan())); + } + + // ── The span may be a slice of a caller-owned buffer, not a whole string ── + + [Fact] + public void SpanLookups_ShouldMatchOnASliceOfALargerBuffer_AcrossEveryType() + { + // The shape a real parser hands in: the key sits inside a bigger buffer with + // neighbouring characters on both sides. + const string Buffer = ">>>alphabet<<<"; + ReadOnlySpan slice = Buffer.AsSpan(3, "alphabet".Length); + + var frozenDict = new FrozenCelerityDictionary(Pairs()); + var frozenSet = new FrozenCeleritySet(Keys); + + var dict = new CelerityDictionary(); + var set = new CeleritySet(); + var trie = new Trie(); + foreach (KeyValuePair pair in Pairs()) + { + dict.Add(pair.Key, pair.Value); + set.Add(pair.Key); + trie.Add(pair.Key, pair.Value); + } + + int expected = Array.IndexOf(Keys, "alphabet"); + + Assert.True(frozenDict.TryGetValue(slice, out int a)); + Assert.Equal(expected, a); + Assert.True(frozenSet.Contains(slice)); + Assert.True(dict.TryGetValue(slice, out int b)); + Assert.Equal(expected, b); + Assert.True(set.Contains(slice)); + Assert.True(trie.TryGetValue(slice, out int c)); + Assert.Equal(expected, c); + } +} diff --git a/src/Celerity.Tests/Collections/StringInternTableDifferentialTests.cs b/src/Celerity.Tests/Collections/StringInternTableDifferentialTests.cs new file mode 100644 index 00000000..532ea6ed --- /dev/null +++ b/src/Celerity.Tests/Collections/StringInternTableDifferentialTests.cs @@ -0,0 +1,127 @@ +using Celerity.Collections; +using Celerity.Hashing; + +namespace Celerity.Tests.Collections; + +/// +/// Deterministic, seeded differential coverage for . Each seed +/// drives the same random stream of span interns, string interns, lookups, and clears into the table and +/// into an independent oracle keyed by +/// , then asserts after every operation that the two agree on count, +/// per-token membership, and — the property that makes the type worth having — that the table hands back +/// the same reference for every repeat of a token it has already seen. +/// +/// +/// Tokens are drawn from a tiny alphabet at short lengths so hash collisions and probe chains are dense, +/// and the run interleaves the span and string entry points so a divergence between them surfaces. +/// +public class StringInternTableDifferentialTests +{ + [Theory] + [InlineData(1)] + [InlineData(2)] + [InlineData(7)] + [InlineData(42)] + [InlineData(1234)] + public void RandomizedOperations_MatchDictionaryOracle_WithWeakHasher(int seed) => + RunCase(seed); + + [Theory] + [InlineData(1)] + [InlineData(42)] + [InlineData(1234)] + public void RandomizedOperations_MatchDictionaryOracle_WithStrongHasher(int seed) => + RunCase(seed); + + private static void RunCase(int seed) + where THasher : struct, IHashProvider, ISpanHashProvider + { + var rand = new Random(seed); + var table = new StringInternTable(capacity: 2); + + // The oracle maps a token's contents to the canonical instance the table returned first. + var oracle = new Dictionary(StringComparer.Ordinal); + + const int Steps = 4000; + for (int step = 0; step < Steps; step++) + { + string token = RandomToken(rand); + int op = rand.Next(100); + + if (op < 45) + { + // Intern from a span carved out of a larger buffer — the parser shape. + string padded = "<<" + token + ">>"; + string interned = table.GetOrAdd(padded.AsSpan(2, token.Length)); + + Assert.Equal(token, interned); + if (oracle.TryGetValue(token, out string? canonical)) + Assert.Same(canonical, interned); + else + oracle[token] = interned; + } + else if (op < 70) + { + // Intern from a freshly allocated string: on a miss the supplied instance itself + // becomes canonical; on a hit the already-held instance comes back. + string supplied = new string(token.ToCharArray()); + string interned = table.GetOrAdd(supplied); + + if (oracle.TryGetValue(token, out string? canonical)) + { + Assert.Same(canonical, interned); + } + else + { + Assert.Same(supplied, interned); + oracle[token] = interned; + } + } + else if (op < 88) + { + // Pure lookup: never mutates, and agrees with the oracle on both entry points. + bool expected = oracle.TryGetValue(token, out string? canonical); + Assert.Equal(expected, table.TryGet(token.AsSpan(), out string? actual)); + Assert.Equal(expected, table.Contains(token.AsSpan())); + Assert.Equal(expected, table.Contains(token)); + if (expected) + Assert.Same(canonical, actual); + else + Assert.Null(actual); + } + else if (op < 97) + { + // Enumeration yields exactly the canonical instances, once each. + var seen = new List(); + foreach (string s in table) + seen.Add(s); + + Assert.Equal(oracle.Count, seen.Count); + foreach (string s in seen) + { + Assert.True(oracle.TryGetValue(s, out string? canonical)); + Assert.Same(canonical, s); + } + } + else + { + table.Clear(); + oracle.Clear(); + } + + Assert.Equal(oracle.Count, table.Count); + } + } + + // A tiny alphabet at short lengths, so tokens repeat constantly and probe chains stay dense. + private static string RandomToken(Random rand) + { + const string Alphabet = "abcŁ"; + int length = rand.Next(0, 5); + return string.Create(length, rand, static (span, rng) => + { + for (int i = 0; i < span.Length; i++) + span[i] = Alphabet[rng.Next(Alphabet.Length)]; + }); + } +} diff --git a/src/Celerity.Tests/Collections/StringInternTableTests.cs b/src/Celerity.Tests/Collections/StringInternTableTests.cs new file mode 100644 index 00000000..6e983096 --- /dev/null +++ b/src/Celerity.Tests/Collections/StringInternTableTests.cs @@ -0,0 +1,326 @@ +using Celerity.Collections; +using Celerity.Hashing; + +namespace Celerity.Tests.Collections; + +/// +/// Dedicated tests for / . +/// +public class StringInternTableTests +{ + // ── Construction / validation ───────────────────────────────────────────── + + [Fact] + public void Constructor_ShouldStartEmpty_WhenDefault() + { + var table = new StringInternTable(); + Assert.Equal(0, table.Count); + Assert.Empty(table); + } + + [Theory] + [InlineData(-1)] + [InlineData(int.MinValue)] + public void Constructor_ShouldThrowArgumentOutOfRangeException_WhenCapacityIsNegative(int capacity) + { + Assert.Throws(() => new StringInternTable(capacity)); + Assert.Throws(() => new StringInternTable(capacity)); + } + + [Theory] + [InlineData(0f)] + [InlineData(1f)] + [InlineData(-0.5f)] + [InlineData(1.5f)] + public void Constructor_ShouldThrowArgumentOutOfRangeException_WhenLoadFactorIsOutOfRange(float loadFactor) + { + Assert.Throws(() => new StringInternTable(16, loadFactor)); + Assert.Throws(() => new StringInternTable(16, loadFactor)); + } + + [Fact] + public void Constructor_ShouldAcceptZeroCapacity_WhenRoundedUp() + { + var table = new StringInternTable(0); + Assert.Same(table.GetOrAdd("a".AsSpan()), table.GetOrAdd("a".AsSpan())); + } + + // ── The headline behaviour: allocate once, return the same reference ────── + + [Fact] + public void GetOrAdd_ShouldReturnTheSameReference_WhenTheSameCharactersAreSeenAgain() + { + var table = new StringInternTable(); + + string first = table.GetOrAdd("token".AsSpan()); + string second = table.GetOrAdd("token".AsSpan()); + + Assert.Equal("token", first); + Assert.Same(first, second); + Assert.Equal(1, table.Count); + } + + [Fact] + public void GetOrAdd_ShouldReturnTheSameReference_WhenSpansComeFromDifferentBuffers() + { + var table = new StringInternTable(); + + string first = table.GetOrAdd("xx-token-yy".AsSpan(3, 5)); + char[] buffer = "..token..".ToCharArray(); + string second = table.GetOrAdd(buffer.AsSpan(2, 5)); + + Assert.Equal("token", first); + Assert.Same(first, second); + Assert.Equal(1, table.Count); + } + + [Fact] + public void GetOrAdd_ShouldNotAllocateANewString_WhenGivenAStringOnAMiss() + { + var table = new StringInternTable(); + string supplied = new string('k', 5); + + Assert.Same(supplied, table.GetOrAdd(supplied)); + } + + [Fact] + public void GetOrAdd_ShouldReturnTheAlreadyInternedInstance_WhenGivenAnEqualString() + { + var table = new StringInternTable(); + string canonical = table.GetOrAdd("token".AsSpan()); + string duplicate = new string("token".ToCharArray()); + + Assert.NotSame(canonical, duplicate); + Assert.Same(canonical, table.GetOrAdd(duplicate)); + Assert.Equal(1, table.Count); + } + + [Fact] + public void GetOrAdd_ShouldThrowArgumentNullException_WhenStringIsNull() + { + var table = new StringInternTable(); + Assert.Throws(() => table.GetOrAdd((string)null!)); + } + + [Fact] + public void GetOrAdd_ShouldTreatTheEmptySpanAsTheEmptyString() + { + var table = new StringInternTable(); + + string interned = table.GetOrAdd(ReadOnlySpan.Empty); + + Assert.Equal(string.Empty, interned); + Assert.Equal(1, table.Count); + Assert.True(table.Contains(string.Empty)); + Assert.Same(interned, table.GetOrAdd(string.Empty)); + } + + [Fact] + public void GetOrAdd_ShouldKeepEveryDistinctToken_WhenTheTableResizes() + { + var table = new StringInternTable(capacity: 2); + var canonical = new string[400]; + + for (int i = 0; i < canonical.Length; i++) + canonical[i] = table.GetOrAdd($"token-{i}".AsSpan()); + + Assert.Equal(canonical.Length, table.Count); + + // Every token still resolves to the instance handed out before the growth. + for (int i = 0; i < canonical.Length; i++) + Assert.Same(canonical[i], table.GetOrAdd($"token-{i}".AsSpan())); + } + + // ── TryGet / Contains ───────────────────────────────────────────────────── + + [Fact] + public void TryGet_ShouldNotIntern_WhenTheCharactersAreAbsent() + { + var table = new StringInternTable(); + + Assert.False(table.TryGet("absent".AsSpan(), out string? value)); + Assert.Null(value); + Assert.Equal(0, table.Count); + } + + [Fact] + public void TryGet_ShouldReturnTheInternedInstance_WhenPresent() + { + var table = new StringInternTable(); + string canonical = table.GetOrAdd("present".AsSpan()); + + Assert.True(table.TryGet("present".AsSpan(), out string? value)); + Assert.Same(canonical, value); + } + + [Fact] + public void Contains_ShouldAgreeAcrossTheSpanAndStringOverloads() + { + var table = new StringInternTable(); + table.GetOrAdd("present".AsSpan()); + + Assert.True(table.Contains("present")); + Assert.True(table.Contains("present".AsSpan())); + Assert.False(table.Contains("absent")); + Assert.False(table.Contains("absent".AsSpan())); + } + + [Fact] + public void Contains_ShouldThrowArgumentNullException_WhenStringIsNull() + { + var table = new StringInternTable(); + Assert.Throws(() => table.Contains((string)null!)); + } + + // ── Clear ───────────────────────────────────────────────────────────────── + + [Fact] + public void Clear_ShouldDropEveryInternedString() + { + var table = new StringInternTable(); + string before = table.GetOrAdd("token".AsSpan()); + table.GetOrAdd("other".AsSpan()); + + table.Clear(); + + Assert.Equal(0, table.Count); + Assert.False(table.Contains("token".AsSpan())); + + // A fresh instance is minted after a Clear — the old one is no longer canonical. + Assert.NotSame(before, table.GetOrAdd("token".AsSpan())); + } + + [Fact] + public void Clear_ShouldBeANoOp_WhenAlreadyEmpty() + { + var table = new StringInternTable(); + table.Clear(); + Assert.Equal(0, table.Count); + } + + // ── Enumeration ─────────────────────────────────────────────────────────── + + [Fact] + public void GetEnumerator_ShouldYieldEveryInternedString() + { + var table = new StringInternTable(); + string[] tokens = ["a", "bb", "ccc", string.Empty]; + foreach (string token in tokens) + table.GetOrAdd(token.AsSpan()); + + var seen = new List(); + foreach (string s in table) + seen.Add(s); + + Assert.Equal(tokens.OrderBy(s => s, StringComparer.Ordinal), seen.OrderBy(s => s, StringComparer.Ordinal)); + } + + [Fact] + public void GetEnumerator_ShouldYieldTheCanonicalInstances() + { + var table = new StringInternTable(); + string canonical = table.GetOrAdd("token".AsSpan()); + + Assert.Same(canonical, Assert.Single(table)); + } + + [Fact] + public void MoveNext_ShouldThrowInvalidOperationException_WhenTheTableIsModifiedDuringEnumeration() + { + var table = new StringInternTable(); + table.GetOrAdd("a".AsSpan()); + + StringInternTable.Enumerator enumerator = table.GetEnumerator(); + table.GetOrAdd("b".AsSpan()); + + Assert.Throws(() => enumerator.MoveNext()); + } + + [Fact] + public void Reset_ShouldRestartTheEnumeration() + { + var table = new StringInternTable(); + table.GetOrAdd("a".AsSpan()); + + StringInternTable.Enumerator enumerator = table.GetEnumerator(); + Assert.True(enumerator.MoveNext()); + Assert.False(enumerator.MoveNext()); + + enumerator.Reset(); + Assert.True(enumerator.MoveNext()); + enumerator.Dispose(); + } + + [Fact] + public void Reset_ShouldThrowInvalidOperationException_WhenTheTableWasModified() + { + var table = new StringInternTable(); + table.GetOrAdd("a".AsSpan()); + + StringInternTable.Enumerator enumerator = table.GetEnumerator(); + table.GetOrAdd("b".AsSpan()); + + Assert.Throws(enumerator.Reset); + } + + [Fact] + public void GetEnumerator_ShouldWorkThroughTheGenericInterface() + { + var table = new StringInternTable(); + table.GetOrAdd("a".AsSpan()); + + IEnumerable asEnumerable = table; + Assert.Single(asEnumerable); + + System.Collections.IEnumerable nonGeneric = table; + var seen = new List(); + foreach (object? item in nonGeneric) + seen.Add(item); + Assert.Single(seen); + } + + [Fact] + public void Current_ShouldBeNullBeforeTheFirstMoveNext() + { + var table = new StringInternTable(); + table.GetOrAdd("a".AsSpan()); + + System.Collections.IEnumerator enumerator = ((IEnumerable)table).GetEnumerator(); + Assert.Null(enumerator.Current); + } + + // ── Hasher parameterization ─────────────────────────────────────────────── + + [Fact] + public void GetOrAdd_ShouldBehaveIdentically_AcrossHashers() + { + var weak = new StringInternTable(); + var strong = new StringInternTable(); + + // "A" and "Ł" share a raw code under the low-byte hasher; the table must still keep + // them distinct, because the resolution is the ordinal span compare, not the hash. + string[] tokens = ["A", "Ł", "A", "Ł", "日本語"]; + foreach (string token in tokens) + { + weak.GetOrAdd(token.AsSpan()); + strong.GetOrAdd(token.AsSpan()); + } + + Assert.Equal(3, weak.Count); + Assert.Equal(3, strong.Count); + Assert.NotSame(weak.GetOrAdd("A".AsSpan()), weak.GetOrAdd("Ł".AsSpan())); + } + + [Fact] + public void GetOrAdd_ShouldKeepSurrogatePairsDistinct() + { + var table = new StringInternTable(); + + string a = table.GetOrAdd("\U0001F600".AsSpan()); + string b = table.GetOrAdd("\U0001F601".AsSpan()); + + Assert.NotSame(a, b); + Assert.Equal(2, table.Count); + Assert.Equal("\U0001F600", a); + } +} diff --git a/src/Celerity.Tests/Hashing/SpanHashParityTests.cs b/src/Celerity.Tests/Hashing/SpanHashParityTests.cs new file mode 100644 index 00000000..7ad0fff7 --- /dev/null +++ b/src/Celerity.Tests/Hashing/SpanHashParityTests.cs @@ -0,0 +1,209 @@ +using Celerity.Hashing; + +namespace Celerity.Tests.Hashing; + +/// +/// Family-wide contract tests for : which built-in hashers +/// implement it, and the one invariant every implementation must hold — +/// Hash(s) == Hash(s.AsSpan()). +/// +/// +/// +/// This is not a nice-to-have. The span lookups on the string-keyed collections hash a span +/// and then compare the result against keys that were placed using the string +/// overload. If the two overloads ever disagreed for some input, the lookup would not be +/// slow — it would report a stored key as absent. That failure is silent, data-dependent, +/// and would survive every existing test, so it is pinned here directly. +/// +/// +/// The parity assertions run over , which already sweeps every +/// length class the block-oriented hashers branch on (empty, sub-word tails, exact word and +/// stripe boundaries, several bulk-loop iterations) plus non-ASCII characters whose high byte +/// is set. Each string is checked both as a whole-string span and as a slice of a larger +/// buffer — the shape a real parser hands in, and the one that would catch an +/// implementation that read past its span or keyed off the buffer rather than the slice. +/// +/// +public class SpanHashParityTests +{ + /// + /// The built-in hashers that hash a character span. Every String* hasher qualifies: + /// each already walks the characters, so the span overload is the same body. The integer + /// and hashers do not — a character span is not their key shape. + /// + public static readonly string[] ExpectedSpanHashers = + [ + nameof(StringAdler32Hasher), + nameof(StringCityHash64Hasher), + nameof(StringCrc32Hasher), + nameof(StringDjb2AHasher), + nameof(StringDjb2Hasher), + nameof(StringElfHasher), + nameof(StringFnV164Hasher), + nameof(StringFnV1A64Hasher), + nameof(StringFnV1AFullHasher), + nameof(StringFnV1AHasher), + nameof(StringFnV1Hasher), + nameof(StringHalfSipHash24Hasher), + nameof(StringHighwayHash64Hasher), + nameof(StringJenkinsOaatHasher), + nameof(StringMetroHash64Hasher), + nameof(StringMurmur2Hasher), + nameof(StringMurmur3Hasher), + nameof(StringSdbmHasher), + nameof(StringSipHash13Hasher), + nameof(StringSipHash24Hasher), + nameof(StringXxHash32Hasher), + nameof(StringXxHash3Hasher), + nameof(StringXxHash64Hasher), + ]; + + private static IEnumerable AllHasherTypes() => + typeof(IHashProvider<>).Assembly + .GetExportedTypes() + .Where(t => t.IsValueType && Array.Exists(t.GetInterfaces(), IsHashProvider32)); + + private static bool IsHashProvider32(Type i) => + i.IsGenericType && i.GetGenericTypeDefinition() == typeof(IHashProvider<>); + + private static bool IsSpanHashProvider(Type t) => + Array.Exists(t.GetInterfaces(), i => i == typeof(ISpanHashProvider)); + + // ── Roster ──────────────────────────────────────────────────────────────── + + [Fact] + public void ISpanHashProvider_ShouldBeImplementedByExactlyTheExpectedHashers() + { + string[] actual = AllHasherTypes() + .Where(IsSpanHashProvider) + .Select(t => t.Name) + .OrderBy(n => n, StringComparer.Ordinal) + .ToArray(); + + Assert.Equal( + ExpectedSpanHashers.OrderBy(n => n, StringComparer.Ordinal).ToArray(), + actual); + } + + [Fact] + public void ISpanHashProvider_ShouldNotBeImplementedByTheNonStringHashers() + { + string[] nonStringSpanHashers = AllHasherTypes() + .Where(t => IsSpanHashProvider(t) && !t.Name.StartsWith("String", StringComparison.Ordinal)) + .Select(t => t.Name) + .ToArray(); + + Assert.Empty(nonStringSpanHashers); + } + + // ── The Hash(s) == Hash(s.AsSpan()) contract, per hasher ────────────────── + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenAdler32() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenCityHash64() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenCrc32() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenDjb2() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenDjb2A() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenElf() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenFnV1() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenFnV164() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenFnV1A() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenFnV1A64() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenFnV1AFull() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenHalfSipHash24() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenHighwayHash64() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenJenkinsOaat() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenMetroHash64() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenMurmur2() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenMurmur3() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenSdbm() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenSipHash13() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenSipHash24() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenXxHash32() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenXxHash3() => AssertParity(); + + [Fact] + public void Hash_ShouldMatchStringOverload_WhenXxHash64() => AssertParity(); + + // ── Null handling on the string overload is unchanged ───────────────────── + + [Fact] + public void Hash_ShouldThrowArgumentNullException_WhenStringKeyIsNull() + { + var hasher = new StringFnV1AHasher(); + Assert.Throws(() => hasher.Hash((string)null!)); + } + + [Fact] + public void Hash_ShouldReturnTheEmptyStringCode_WhenSpanIsDefault() + { + // A span has no null state: default(ReadOnlySpan) is empty, and empty means "". + var hasher = new StringFnV1AHasher(); + Assert.Equal(hasher.Hash(string.Empty), hasher.Hash(default(ReadOnlySpan))); + } + + private static void AssertParity() + where THasher : struct, IHashProvider, ISpanHashProvider + { + var hasher = default(THasher); + + foreach (string s in HasherStringCorpus.Strings) + { + int fromString = hasher.Hash(s); + + Assert.Equal(fromString, hasher.Hash(s.AsSpan())); + + // The same characters as a slice of a larger buffer, so the hasher cannot be + // reading past the span or keying off anything but the slice's contents. + string padded = "ÿÿ" + s + "ÿÿ"; + Assert.Equal(fromString, hasher.Hash(padded.AsSpan(2, s.Length))); + + // And as a slice of a char[] the caller owns, which is not a string at all. + char[] buffer = new char[s.Length + 3]; + s.AsSpan().CopyTo(buffer.AsSpan(1)); + Assert.Equal(fromString, hasher.Hash(buffer.AsSpan(1, s.Length))); + } + } +} diff --git a/src/Celerity/Collections/CelerityDictionary.cs b/src/Celerity/Collections/CelerityDictionary.cs index 63689dd3..86cd896b 100644 --- a/src/Celerity/Collections/CelerityDictionary.cs +++ b/src/Celerity/Collections/CelerityDictionary.cs @@ -1,4 +1,5 @@ using System.Collections; +using System.Diagnostics; using System.Runtime.CompilerServices; using System.Runtime.InteropServices; using Celerity.Hashing; @@ -803,6 +804,54 @@ private int ProbeForKey(TKey key) } } + // The hasher instance the span-lookup extension methods hand back to the probe below. + // The field is private and the extension methods live outside the type, so they need a + // way to reach it without the class itself carrying an ISpanHashProvider constraint — + // which could not be added to THasher without breaking every existing instantiation. + internal THasher Hasher => _hasher; + + // Reads the value parked alongside a slot the span probe already located. + internal TValue? ValueAt(int index) => _values[index]; + + // The span twin of ProbeForKey. Generic in its own hasher type parameter (rather than + // reusing THasher) so the ISpanHashProvider constraint lives on the method: the class + // constraint stays exactly what it has always been, and the JIT still devirtualizes the + // hash call because TSpanHasher is a struct type parameter. + // + // Only reachable from SpanLookupExtensions, whose signatures pin TKey to string — which + // is what makes the reinterpretation of the slot below sound. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + internal int ProbeForKey(ReadOnlySpan key, TSpanHasher hasher) + where TSpanHasher : struct, ISpanHashProvider + { + // The slot reinterpretation below is a no-op only while TKey really is string. Nothing in + // the type system says so — the extension methods' signatures do — so assert it. Debug.Assert + // is [Conditional("DEBUG")], so this costs nothing in a release build and the hot path is + // unchanged; it exists to catch a future in-assembly caller that forgets the precondition. + Debug.Assert(typeof(TKey) == typeof(string), + "The span probe reinterprets each slot as a string; it is only valid for TKey == string."); + + TKey?[] keys = _keys; + ref TKey? keysRef = ref MemoryMarshal.GetArrayDataReference(keys); + int mask = keys.Length - 1; + int index = hasher.Hash(key) & mask; + + while (true) + { + ref TKey? slot = ref Unsafe.Add(ref keysRef, (nint)(uint)index); + if (EmptySlot.Is(slot)) + return -1; + + // TKey is string at every call site (see above), so this is a no-op + // reinterpretation rather than a cast. Comparing the spans is ordinal, matching + // the EqualityComparer.Default the string-keyed probe uses. + if (key.SequenceEqual(Unsafe.As(ref slot).AsSpan())) + return index; + + index = (index + 1) & mask; + } + } + private void Resize() => Resize(FastUtils.DoubleCapacity(_keys.Length)); // Rehashes every live entry into a freshly allocated table of the given power-of-two size. diff --git a/src/Celerity/Collections/CeleritySet.cs b/src/Celerity/Collections/CeleritySet.cs index 87453f9f..eba8a189 100644 --- a/src/Celerity/Collections/CeleritySet.cs +++ b/src/Celerity/Collections/CeleritySet.cs @@ -1,4 +1,5 @@ using System.Collections; +using System.Diagnostics; using System.Runtime.CompilerServices; using System.Runtime.InteropServices; using Celerity.Hashing; @@ -569,6 +570,51 @@ public void Dispose() { } private static bool IsDefaultValue(T item) => EmptySlot.Is(item); + // The hasher instance the span-lookup extension methods hand back to the probe below. + // The field is private and the extension methods live outside the type, so they need a + // way to reach it without the class itself carrying an ISpanHashProvider constraint — + // which could not be added to THasher without breaking every existing instantiation. + internal THasher Hasher => _hasher; + + // The span twin of ProbeForItem. Generic in its own hasher type parameter (rather than + // reusing THasher) so the ISpanHashProvider constraint lives on the method: the class + // constraint stays exactly what it has always been, and the JIT still devirtualizes the + // hash call because TSpanHasher is a struct type parameter. + // + // Only reachable from SpanLookupExtensions, whose signatures pin T to string — which is + // what makes the reinterpretation of the slot below sound. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + internal int ProbeForItem(ReadOnlySpan item, TSpanHasher hasher) + where TSpanHasher : struct, ISpanHashProvider + { + // The slot reinterpretation below is a no-op only while T really is string. Nothing in the + // type system says so — the extension methods' signatures do — so assert it. Debug.Assert is + // [Conditional("DEBUG")], so this costs nothing in a release build and the hot path is + // unchanged; it exists to catch a future in-assembly caller that forgets the precondition. + Debug.Assert(typeof(T) == typeof(string), + "The span probe reinterprets each slot as a string; it is only valid for T == string."); + + T?[] slots = _slots; + ref T? slotsRef = ref MemoryMarshal.GetArrayDataReference(slots); + int mask = slots.Length - 1; + int index = hasher.Hash(item) & mask; + + while (true) + { + ref T? slot = ref Unsafe.Add(ref slotsRef, (nint)(uint)index); + if (EmptySlot.Is(slot)) + return -1; + + // T is string at every call site (see above), so this is a no-op + // reinterpretation rather than a cast. Comparing the spans is ordinal, matching + // the EqualityComparer.Default the string-keyed probe uses. + if (item.SequenceEqual(Unsafe.As(ref slot).AsSpan())) + return index; + + index = (index + 1) & mask; + } + } + [MethodImpl(MethodImplOptions.AggressiveInlining)] private int ProbeForItem(T item) { diff --git a/src/Celerity/Collections/FrozenCelerityDictionary.cs b/src/Celerity/Collections/FrozenCelerityDictionary.cs index ebe23720..3ec1c41a 100644 --- a/src/Celerity/Collections/FrozenCelerityDictionary.cs +++ b/src/Celerity/Collections/FrozenCelerityDictionary.cs @@ -468,6 +468,47 @@ private static void BuildFallback( // Returns the slot holding , or -1 if absent. In perfect // mode this is a single index + equality check; in fallback mode it linear-probes // until it finds the key or hits an empty slot. The caller guarantees key != null. + // The hasher instance the span-lookup extension methods hand back to the probe below. + // The field is private and the extension methods live outside the type, so they need a + // way to reach it without the class itself carrying an ISpanHashProvider constraint — + // which could not be added to THasher without breaking every existing instantiation. + internal THasher Hasher => _hasher; + + // Reads the value parked alongside a slot the span probe already located. + internal TValue? ValueAt(int index) => _values[index]; + + // The span twin of FindSlot. Generic in its own hasher type parameter (rather than + // reusing THasher) so the ISpanHashProvider constraint lives on the method: the class + // constraint stays exactly what it has always been, and the JIT still devirtualizes the + // hash call because TSpanHasher is a struct type parameter. Both the perfect and the + // linear-probing fallback shapes are mirrored exactly, including the same Mix/seed, so + // the span path and the string path resolve to the same slot. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + internal int FindSlot(ReadOnlySpan key, TSpanHasher hasher) + where TSpanHasher : struct, ISpanHashProvider + { + string?[] keys = _keys; + ref string? keysRef = ref MemoryMarshal.GetArrayDataReference(keys); + int mask = _mask; + int slot = (int)(Mix(unchecked((uint)hasher.Hash(key)), _seed) & (uint)mask); + + if (_isPerfect) + { + string? candidate = Unsafe.Add(ref keysRef, (nint)(uint)slot); + return candidate is not null && key.SequenceEqual(candidate.AsSpan()) ? slot : -1; + } + + while (true) + { + string? candidate = Unsafe.Add(ref keysRef, (nint)(uint)slot); + if (candidate is null) + return -1; + if (key.SequenceEqual(candidate.AsSpan())) + return slot; + slot = (slot + 1) & mask; + } + } + [MethodImpl(MethodImplOptions.AggressiveInlining)] private int FindSlot(string key) { diff --git a/src/Celerity/Collections/FrozenCeleritySet.cs b/src/Celerity/Collections/FrozenCeleritySet.cs index f3d24ddb..8d521d7a 100644 --- a/src/Celerity/Collections/FrozenCeleritySet.cs +++ b/src/Celerity/Collections/FrozenCeleritySet.cs @@ -481,6 +481,44 @@ private static void BuildFallback( // mode this is a single index + equality check; in fallback mode it linear-probes // until it finds the element or hits an empty slot. The caller guarantees // item != null. + // The hasher instance the span-lookup extension methods hand back to the probe below. + // The field is private and the extension methods live outside the type, so they need a + // way to reach it without the class itself carrying an ISpanHashProvider constraint — + // which could not be added to THasher without breaking every existing instantiation. + internal THasher Hasher => _hasher; + + // The span twin of FindSlot. Generic in its own hasher type parameter (rather than + // reusing THasher) so the ISpanHashProvider constraint lives on the method: the class + // constraint stays exactly what it has always been, and the JIT still devirtualizes the + // hash call because TSpanHasher is a struct type parameter. Both the perfect and the + // linear-probing fallback shapes are mirrored exactly, including the same Mix/seed, so + // the span path and the string path resolve to the same slot. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + internal int FindSlot(ReadOnlySpan item, TSpanHasher hasher) + where TSpanHasher : struct, ISpanHashProvider + { + string?[] items = _items; + ref string? itemsRef = ref MemoryMarshal.GetArrayDataReference(items); + int mask = _mask; + int slot = (int)(Mix(unchecked((uint)hasher.Hash(item)), _seed) & (uint)mask); + + if (_isPerfect) + { + string? candidate = Unsafe.Add(ref itemsRef, (nint)(uint)slot); + return candidate is not null && item.SequenceEqual(candidate.AsSpan()) ? slot : -1; + } + + while (true) + { + string? candidate = Unsafe.Add(ref itemsRef, (nint)(uint)slot); + if (candidate is null) + return -1; + if (item.SequenceEqual(candidate.AsSpan())) + return slot; + slot = (slot + 1) & mask; + } + } + [MethodImpl(MethodImplOptions.AggressiveInlining)] private int FindSlot(string item) { diff --git a/src/Celerity/Collections/SpanLookupExtensions.cs b/src/Celerity/Collections/SpanLookupExtensions.cs new file mode 100644 index 00000000..f3852380 --- /dev/null +++ b/src/Celerity/Collections/SpanLookupExtensions.cs @@ -0,0 +1,185 @@ +using Celerity.Hashing; + +namespace Celerity.Collections; + +/// +/// Allocation-free lookups on the string-keyed Celerity collections from a +/// of — a slice of a buffer a parser +/// already holds — without first materializing a . +/// +/// +/// +/// A tokenizer, CSV/log reader, or route dispatcher that has a +/// ReadOnlySpan<char> over its input buffer would otherwise have to call +/// new string(span) to probe a string-keyed collection: one allocation plus a copy +/// per lookup, on the hot path of exactly the workloads these types exist for. +/// These overloads delete both. The stored keys are compared ordinally against the span, +/// which is what does for , +/// so a span lookup and the equivalent string lookup always agree. +/// +/// +/// Why extension methods. The collections are generic in their hasher +/// (where THasher : struct, IHashProvider<string>) and the span probe needs +/// as well. Adding that to the class constraint would break +/// every existing instantiation, so the extra constraint lives on these methods instead: +/// they bind only when the hasher supplies both, are resolved statically (no boxing, and the +/// JIT still devirtualizes the hash call through the struct type parameter), and leave the +/// collections' own signatures untouched. +/// +/// +/// The empty span. A span has no null state, so an empty span means +/// the empty string "" — an ordinary key — and never the out-of-band null key. +/// Look the null key up through the string overload. +/// +/// +/// takes no hasher and carries its span overloads as ordinary +/// instance methods; see . +/// To go the other way — turning a span into a that is allocated only +/// the first time it is seen — use . +/// +/// +/// +/// +/// var routes = new FrozenCelerityDictionary<int, StringXxHash3Hasher>(pairs); +/// +/// ReadOnlySpan<char> path = requestLine.AsSpan(4, length); +/// if (routes.TryGetValue(path, out int handler)) +/// Dispatch(handler); // no string was allocated +/// +/// +public static class SpanLookupExtensions +{ + /// + /// Attempts to get the value associated with the characters in . + /// + /// The type of the stored values. + /// The dictionary's hasher; must also hash spans. + /// The dictionary to probe. + /// The characters to look up. An empty span means the key "". + /// + /// When this method returns, contains the value associated with + /// if found; otherwise the default value of . + /// + /// true if the key was found; otherwise, false. + /// is null. + public static bool TryGetValue( + this FrozenCelerityDictionary dictionary, + ReadOnlySpan key, + out TValue? value) + where THasher : struct, IHashProvider, ISpanHashProvider + { + ArgumentNullException.ThrowIfNull(dictionary); + + int index = dictionary.FindSlot(key, dictionary.Hasher); + if (index < 0) + { + value = default; + return false; + } + + value = dictionary.ValueAt(index); + return true; + } + + /// + /// Determines whether the characters in are present as a key. + /// + /// The type of the stored values. + /// The dictionary's hasher; must also hash spans. + /// The dictionary to probe. + /// The characters to look up. An empty span means the key "". + /// true if the key is found; otherwise, false. + /// is null. + public static bool ContainsKey( + this FrozenCelerityDictionary dictionary, + ReadOnlySpan key) + where THasher : struct, IHashProvider, ISpanHashProvider + { + ArgumentNullException.ThrowIfNull(dictionary); + return dictionary.FindSlot(key, dictionary.Hasher) >= 0; + } + + /// + /// Determines whether the characters in are present in the set. + /// + /// The set's hasher; must also hash spans. + /// The set to probe. + /// The characters to look up. An empty span means the element "". + /// true if the element is found; otherwise, false. + /// is null. + public static bool Contains( + this FrozenCeleritySet set, + ReadOnlySpan item) + where THasher : struct, IHashProvider, ISpanHashProvider + { + ArgumentNullException.ThrowIfNull(set); + return set.FindSlot(item, set.Hasher) >= 0; + } + + /// + /// Attempts to get the value associated with the characters in . + /// + /// The type of the stored values. + /// The dictionary's hasher; must also hash spans. + /// The dictionary to probe. + /// The characters to look up. An empty span means the key "". + /// + /// When this method returns, contains the value associated with + /// if found; otherwise the default value of . + /// + /// true if the key was found; otherwise, false. + /// is null. + public static bool TryGetValue( + this CelerityDictionary dictionary, + ReadOnlySpan key, + out TValue? value) + where THasher : struct, IHashProvider, ISpanHashProvider + { + ArgumentNullException.ThrowIfNull(dictionary); + + int index = dictionary.ProbeForKey(key, dictionary.Hasher); + if (index < 0) + { + value = default; + return false; + } + + value = dictionary.ValueAt(index); + return true; + } + + /// + /// Determines whether the characters in are present as a key. + /// + /// The type of the stored values. + /// The dictionary's hasher; must also hash spans. + /// The dictionary to probe. + /// The characters to look up. An empty span means the key "". + /// true if the key is found; otherwise, false. + /// is null. + public static bool ContainsKey( + this CelerityDictionary dictionary, + ReadOnlySpan key) + where THasher : struct, IHashProvider, ISpanHashProvider + { + ArgumentNullException.ThrowIfNull(dictionary); + return dictionary.ProbeForKey(key, dictionary.Hasher) >= 0; + } + + /// + /// Determines whether the characters in are present in the set. + /// + /// The set's hasher; must also hash spans. + /// The set to probe. + /// The characters to look up. An empty span means the element "". + /// true if the element is found; otherwise, false. + /// is null. + public static bool Contains( + this CeleritySet set, + ReadOnlySpan item) + where THasher : struct, IHashProvider, ISpanHashProvider + { + ArgumentNullException.ThrowIfNull(set); + return set.ProbeForItem(item, set.Hasher) >= 0; + } +} diff --git a/src/Celerity/Collections/StringInternTable.cs b/src/Celerity/Collections/StringInternTable.cs new file mode 100644 index 00000000..346e314a --- /dev/null +++ b/src/Celerity/Collections/StringInternTable.cs @@ -0,0 +1,404 @@ +using System.Collections; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using Celerity.Hashing; +using Celerity.Primitives; + +namespace Celerity.Collections; + +/// +/// A canonicalizing table of s that is probed with a +/// of and allocates a +/// only on a miss, using +/// . Supply a different string hasher via the +/// generic overload. +/// +public sealed class StringInternTable : StringInternTable +{ + /// + /// Initializes a new with the specified capacity + /// and load factor. + /// + /// The initial capacity, rounded up to the next power of two. + /// + /// The fraction of the table that can be filled before it grows. + /// + /// + /// is negative, or is not + /// in the open interval (0, 1). + /// + public StringInternTable( + int capacity = DEFAULT_CAPACITY, + float loadFactor = DEFAULT_LOAD_FACTOR) + : base(capacity, loadFactor) + { + } +} + +/// +/// A canonicalizing table of s that is probed with a +/// of and allocates a +/// only on a miss. +/// +/// +/// The hasher used to compute key hashes. Must be a value type implementing both +/// over and +/// , so the JIT can devirtualize and inline it and so the +/// span and string probes agree. +/// +/// +/// +/// The workload. A parser walking a 10M-cell CSV or a log stream holds each +/// token as a slice of its input buffer. If the token set is small — say a hundred distinct +/// column values — it wants a hundred s, not ten million. Feeding each +/// slice to returns the one canonical instance and +/// materializes a only the first time a token is seen. Downstream +/// reference equality then works, and the GC never sees the other 9,999,900 copies. +/// +/// +/// Why the BCL cannot do this before .NET 9. +/// HashSet<string>.TryGetValue takes a , so you must +/// allocate the string before you can discover you already had it — the allocation +/// this type exists to avoid. .NET 9's +/// Dictionary<string,V>.GetAlternateLookup<ReadOnlySpan<char>>() +/// closes that gap on .NET 9+; this type works the same way on net8.0, which is +/// Celerity's floor, and stays available on all three target frameworks. +/// +/// +/// Not . The runtime intern pool is +/// process-wide, never collected for the life of the process, and — decisively — still +/// requires a to hand it. A is +/// an ordinary object: its scope is yours, releases everything it holds, +/// and dropping the table drops the interned strings with it. +/// +/// +/// Keys are compared ordinally, matching for +/// . The empty string is an ordinary entry (an empty span means ""); +/// null is not storable, and the string overloads reject it. The type is +/// single-threaded and does not guarantee enumeration order. +/// +/// +/// +/// +/// var interned = new StringInternTable(); +/// +/// foreach (ReadOnlySpan<char> cell in Cells(line)) +/// { +/// string token = interned.GetOrAdd(cell); // allocates only the first time +/// Consume(token); +/// } +/// +/// +public class StringInternTable : IReadOnlyCollection + where THasher : struct, IHashProvider, ISpanHashProvider +{ + /// + /// The default initial capacity of the table if no capacity is specified. + /// + protected const int DEFAULT_CAPACITY = 16; + + /// + /// The default load factor of the table if no load factor is specified. + /// + protected const float DEFAULT_LOAD_FACTOR = 0.75f; + + private string?[] _slots; + private int _count; + private readonly float _loadFactor; + private int _threshold; + private readonly THasher _hasher; + + // Incremented on every structural mutation so active enumerators can detect + // concurrent modification and throw, matching BCL semantics. + private int _version; + + /// + /// Initializes a new with the specified + /// capacity and load factor. + /// + /// The initial capacity, rounded up to the next power of two. + /// + /// The fraction of the table that can be filled before it grows. + /// + /// + /// is negative, or is not + /// in the open interval (0, 1). + /// + public StringInternTable( + int capacity = DEFAULT_CAPACITY, + float loadFactor = DEFAULT_LOAD_FACTOR) + { + if (capacity < 0) + throw new ArgumentOutOfRangeException(nameof(capacity), capacity, "Capacity must be non-negative."); + if (loadFactor <= 0f || loadFactor >= 1f) + throw new ArgumentOutOfRangeException(nameof(loadFactor), loadFactor, "Load factor must be between 0 (exclusive) and 1 (exclusive)."); + + int size = FastUtils.NextPowerOfTwo(capacity); + + _slots = new string?[size]; + _loadFactor = loadFactor; + _threshold = (int)(size * _loadFactor); + _hasher = default; + } + + /// + /// Gets the number of distinct strings the table has interned. + /// + public int Count => _count; + + /// + /// Returns the canonical for the characters in + /// , allocating one only if those characters are not + /// already interned. + /// + /// The characters to canonicalize. An empty span means "". + /// + /// The interned instance. Two calls with equal contents return the same reference, so + /// callers may compare the results with . + /// + public string GetOrAdd(ReadOnlySpan key) + { + int index = Probe(key, out bool wasEmpty); + if (!wasEmpty) + return _slots[index]!; + + // The miss path is the only one that materializes a string. + string materialized = key.ToString(); + + if (_count >= _threshold) + { + Resize(); + index = Probe(key, out _); + } + + _slots[index] = materialized; + _count++; + _version++; + return materialized; + } + + /// + /// Returns the canonical for , interning + /// itself if its contents are not already present. + /// + /// The string to canonicalize. + /// + /// The interned instance — itself if it was the first with those + /// contents, otherwise the instance already held. + /// + /// is null. + /// + /// This overload never allocates: on a miss the supplied instance becomes the canonical + /// one. It is the shape to use when you already hold a and want to + /// collapse duplicates. + /// + public string GetOrAdd(string key) + { + ArgumentNullException.ThrowIfNull(key); + + int index = Probe(key.AsSpan(), out bool wasEmpty); + if (!wasEmpty) + return _slots[index]!; + + if (_count >= _threshold) + { + Resize(); + index = Probe(key.AsSpan(), out _); + } + + _slots[index] = key; + _count++; + _version++; + return key; + } + + /// + /// Looks up the characters in without interning them. + /// + /// The characters to look up. An empty span means "". + /// + /// When this method returns, the interned instance if the contents were already present; + /// otherwise null. The table is not modified either way. + /// + /// true if the contents were already interned; otherwise false. + public bool TryGet(ReadOnlySpan key, out string? value) + { + int index = Probe(key, out bool wasEmpty); + if (wasEmpty) + { + value = null; + return false; + } + + value = _slots[index]; + return true; + } + + /// + /// Determines whether the characters in are already interned. + /// + /// The characters to look up. An empty span means "". + /// true if the contents are present; otherwise false. + public bool Contains(ReadOnlySpan key) + { + Probe(key, out bool wasEmpty); + return !wasEmpty; + } + + /// + /// Determines whether 's contents are already interned. + /// + /// The string to look up. + /// true if the contents are present; otherwise false. + /// is null. + public bool Contains(string key) + { + ArgumentNullException.ThrowIfNull(key); + return Contains(key.AsSpan()); + } + + /// + /// Drops every interned string. The backing capacity is preserved. + /// + public void Clear() + { + if (_count == 0) + return; + + Array.Clear(_slots, 0, _slots.Length); + _count = 0; + _version++; + } + + /// + /// Returns an allocation-free enumerator over the interned strings. The order is + /// unspecified and may change across versions. If the table is modified during + /// enumeration, throws + /// . + /// + /// A struct enumerator over this table. + public Enumerator GetEnumerator() => new Enumerator(this); + + IEnumerator IEnumerable.GetEnumerator() => new Enumerator(this); + + IEnumerator IEnumerable.GetEnumerator() => new Enumerator(this); + + /// + /// A struct enumerator over a . Because it is a + /// struct, iterating it via foreach avoids the allocation a compiler-generated + /// IEnumerator<T> would incur. + /// + public struct Enumerator : IEnumerator + { + private readonly StringInternTable _table; + private readonly int _version; + private int _index; + private string _current; + + internal Enumerator(StringInternTable table) + { + _table = table; + _version = table._version; + _index = -1; + _current = null!; + } + + /// Gets the string at the current position of the enumerator. + public string Current => _current; + + object IEnumerator.Current => _current; + + /// Advances the enumerator to the next interned string. + /// + /// true if the enumerator advanced to a new entry; false if it has + /// passed the end of the table. + /// + /// + /// The table was modified since the enumerator was created. + /// + public bool MoveNext() + { + if (_version != _table._version) + throw new InvalidOperationException("Collection was modified; enumeration operation may not execute."); + + string?[] slots = _table._slots; + while (++_index < slots.Length) + { + string? slot = slots[_index]; + if (slot is not null) + { + _current = slot; + return true; + } + } + + _current = null!; + return false; + } + + /// Resets the enumerator to its initial position, before the first entry. + /// + /// The table was modified since the enumerator was created. + /// + public void Reset() + { + if (_version != _table._version) + throw new InvalidOperationException("Collection was modified; enumeration operation may not execute."); + + _index = -1; + _current = null!; + } + + /// Releases any resources held by the enumerator. No-op for this type. + public void Dispose() { } + } + + // Returns the slot the characters belong in. wasEmpty is true when the slot is vacant + // (a miss — the caller may write there) and false when it already holds a string with + // these contents. A vacant slot always exists because the load factor is < 1. + // + // The probe walks _slots via Unsafe.Add against a base reference taken at the top, so + // per-iteration bounds checks disappear; the bound is structural (mask = length - 1). + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private int Probe(ReadOnlySpan key, out bool wasEmpty) + { + string?[] slots = _slots; + ref string? slotsRef = ref MemoryMarshal.GetArrayDataReference(slots); + int mask = slots.Length - 1; + int index = _hasher.Hash(key) & mask; + + while (true) + { + string? slot = Unsafe.Add(ref slotsRef, (nint)(uint)index); + if (slot is null) { wasEmpty = true; return index; } + if (key.SequenceEqual(slot.AsSpan())) { wasEmpty = false; return index; } + index = (index + 1) & mask; + } + } + + // Rehashes every interned string into a table of twice the size. Entries are known to be + // distinct, so the reinsert loop only has to find the first vacant slot. + private void Resize() + { + int newSize = FastUtils.DoubleCapacity(_slots.Length); + int mask = newSize - 1; + string?[] oldSlots = _slots; + string?[] newSlots = new string?[newSize]; + + for (int i = 0; i < oldSlots.Length; i++) + { + string? slot = oldSlots[i]; + if (slot is null) + continue; + + int index = _hasher.Hash(slot) & mask; + while (newSlots[index] is not null) + index = (index + 1) & mask; + + newSlots[index] = slot; + } + + _slots = newSlots; + _threshold = (int)(newSize * _loadFactor); + } +} diff --git a/src/Celerity/Collections/Trie.cs b/src/Celerity/Collections/Trie.cs index a63fd547..90efd42a 100644 --- a/src/Celerity/Collections/Trie.cs +++ b/src/Celerity/Collections/Trie.cs @@ -259,6 +259,45 @@ public bool TryGetValue(string key, out TValue? value) return false; } + /// + /// Determines whether the trie contains the characters in , without + /// materializing a from the span. + /// + /// The characters to locate. An empty span means the key "". + /// true if the key is present; otherwise false. + /// + /// A descends its keys character by character, so a span key costs + /// exactly what a key costs — minus the new string(span) a caller + /// holding a slice of a parse buffer would otherwise have to allocate per lookup. + /// + public bool ContainsKey(ReadOnlySpan key) + { + Node? node = FindNode(key); + return node is not null && node.HasValue; + } + + /// + /// Attempts to get the value associated with the characters in , without + /// materializing a from the span. + /// + /// The characters to locate. An empty span means the key "". + /// + /// When this method returns, the associated value if the key was found; otherwise the default + /// value of . + /// + /// true if the key was found; otherwise false. + public bool TryGetValue(ReadOnlySpan key, out TValue? value) + { + Node? node = FindNode(key); + if (node is not null && node.HasValue) + { + value = node.Value; + return true; + } + value = default; + return false; + } + /// Removes from the trie. /// The key to remove. /// true if the key was found and removed; otherwise false. @@ -372,6 +411,19 @@ public bool ContainsPrefix(string prefix) return node is not null && (node.HasValue || node.ChildCount != 0); } + /// + /// Determines whether any stored key starts with the characters in + /// (a key equal to the prefix counts), without materializing a from the + /// span. The empty span matches whenever the trie is non-empty. + /// + /// The characters to test as a prefix. + /// true if at least one key has as a prefix; otherwise false. + public bool ContainsPrefix(ReadOnlySpan prefix) + { + Node? node = FindNode(prefix); + return node is not null && (node.HasValue || node.ChildCount != 0); + } + /// /// Enumerates every entry whose key starts with (an entry whose key equals the /// prefix is included), in ascending ordinal key order. The empty prefix enumerates the whole trie. @@ -490,7 +542,9 @@ public bool TryGetLongestPrefix(string query, out string? key, out TValue? value // ---- internal machinery ---------------------------------------------------------------------- // Walks the key from the root and returns the node it ends on, or null if the path breaks. - private Node? FindNode(string key) + // Takes a span so the string and span lookup surfaces share one descent; a string key is + // handed in as key.AsSpan(), which is free. + private Node? FindNode(ReadOnlySpan key) { Node node = _root; for (int i = 0; i < key.Length; i++) diff --git a/web/dev/bench/detail.html b/web/dev/bench/detail.html index 683182c2..c4485aa0 100644 --- a/web/dev/bench/detail.html +++ b/web/dev/bench/detail.html @@ -384,6 +384,7 @@ { key: 'DisjointSet', title: 'DisjointSet', vs: 'Dictionary> merge' }, { key: 'IndexedPriorityQueue', title: 'IndexedPriorityQueue', vs: 'PriorityQueue' }, { key: 'Trie', title: 'Trie', vs: 'Dictionary' }, + { key: 'StringInternTable', title: 'StringInternTable', vs: 'HashSet / Dictionary' }, { key: 'FenwickTree', title: 'FenwickTree', vs: 'long[] (naive prefix sum)' }, { key: 'BTreeDictionary', title: 'BTreeDictionary', vs: 'SortedDictionary' }, { key: 'BTreeSet', title: 'BTreeSet', vs: 'SortedSet' }, diff --git a/web/dev/bench/index.html b/web/dev/bench/index.html index 9ebf6bd9..054114af 100644 --- a/web/dev/bench/index.html +++ b/web/dev/bench/index.html @@ -419,7 +419,7 @@

Hash function throughput

{ key: 'SwissDictionary', title: 'SwissDictionary', vs: 'Dictionary', ops: ['Insert', 'Lookup', 'Remove'] }, { key: 'HashCachingDictionary', title: 'HashCachingDictionary', vs: 'Dictionary', ops: ['Insert', 'Lookup', 'Remove'] }, { key: 'PooledCelerityDictionary', title: 'PooledCelerityDictionary', vs: 'Dictionary', ops: ['Insert', 'Lookup', 'Remove'] }, - { key: 'FrozenCelerityDictionary', title: 'FrozenCelerityDictionary', vs: 'FrozenDictionary', ops: ['Build', 'Lookup'] }, + { key: 'FrozenCelerityDictionary', title: 'FrozenCelerityDictionary', vs: 'FrozenDictionary', ops: ['Build', 'Lookup', 'SpanLookup'] }, { key: 'CelerityMultiMap', title: 'CelerityMultiMap', vs: 'Dictionary>', ops: ['Insert', 'Lookup', 'Remove'] }, { key: 'CelerityMultiSet', title: 'CelerityMultiSet', vs: 'Dictionary', ops: ['Count', 'Lookup', 'Remove'] }, // SmallDictionary is a small-n collection: it is benchmarked at 8 / 64 items @@ -459,7 +459,11 @@

Hash function throughput

// IndexedPriorityQueue is an addressable heap benchmarked against the BCL PriorityQueue (lazy-deletion decrease-key). { key: 'IndexedPriorityQueue', title: 'IndexedPriorityQueue', vs: 'PriorityQueue', ops: ['Enqueue', 'DecreaseKey'] }, // Trie is an ordered prefix tree benchmarked against Dictionary; PrefixMatch is the win. - { key: 'Trie', title: 'Trie', vs: 'Dictionary', ops: ['Add', 'Lookup', 'PrefixMatch'] }, + // SpanLookup probes from a ReadOnlySpan slice, which the baseline must first turn into a string. + { key: 'Trie', title: 'Trie', vs: 'Dictionary', ops: ['Add', 'Lookup', 'PrefixMatch', 'SpanLookup'] }, + // StringInternTable canonicalizes tokens probed as spans: it allocates one string per distinct token, + // where the BCL arms must allocate one per occurrence before they can dedupe. Allocation is the headline. + { key: 'StringInternTable', title: 'StringInternTable', vs: 'HashSet / Dictionary', ops: ['Dedupe', 'Lookup'] }, // FenwickTree is a Binary Indexed Tree benchmarked against a plain long[] holding the raw values: point // updates are O(1) there, but every prefix / range sum re-adds the slice (O(n)). The tree keeps both O(log n). { key: 'FenwickTree', title: 'FenwickTree', vs: 'long[] (naive prefix sum)', ops: ['Mixed', 'RangeSum'] }, diff --git a/web/index.html b/web/index.html index 2295e39a..db43dbe2 100644 --- a/web/index.html +++ b/web/index.html @@ -305,6 +305,7 @@

What ships in the box

DisjointSet<T>
Union-find over arbitrary elements: partitions them into disjoint sets with near-O(1) amortized Union/Find/Connected via union-by-size and path halving. The union-find the BCL lacks — incremental connectivity, connected components, and Kruskal MST in near-linear time, where a Dictionary+HashSet set-merge is quadratic.
IndexedPriorityQueue<E, P, H>
Addressable binary min-heap: unlike the BCL PriorityQueue it can change a queued element's priority (Update / decrease-key) and remove an arbitrary element in O(log n), and answer Contains/TryGetPriority in O(1). The heap the priority-relaxation loop of Dijkstra / Prim / A* needs — no lazy-deletion heap growth.
Trie<TValue>
Ordered prefix tree mapping string keys to values: GetByPrefix lists every entry under a prefix in O(prefix + matches) and TryGetLongestPrefix finds the longest stored prefix of a query in O(query). The trie the BCL lacks — autocomplete, routing, and ordered iteration, where a Dictionary must scan every key and run StartsWith.
+
StringInternTable
Canonicalizing token table probed with a ReadOnlySpan<char>: GetOrAdd returns the one shared string for those characters and allocates only on a miss, so a 10M-cell parse over 100 distinct tokens creates 100 strings, not 10,000,000. The collection you cannot build on the pre-.NET-9 BCL — HashSet<string> makes you allocate the string before you can discover you already had it. The same span-keyed lookups also ship on FrozenCelerityDictionary, FrozenCeleritySet, CelerityDictionary, CeleritySet, and Trie.
FenwickTree<T>
Binary Indexed Tree over a fixed-length numeric sequence: point update and prefix / range sum both in O(log n), in one array with no per-node overhead. The prefix-sum structure the BCL lacks — running aggregates, rank counters and cumulative-frequency tables, where a plain array is O(n) per query or O(n) per update.
BTreeDictionary<K, V, C>
Sorted map backed by a B-tree with up to 31 keys per node in flat arrays, so a lookup visits ~log32(n) nodes instead of chasing ~log2(n) pointers. The B-tree the BCL lacks — Min/Max, lower/upper bound and O(log n + k) range scans, where SortedDictionary is a red-black tree with an object per entry and SortedList memmoves on every insert.
BTreeSet<T, C>
The set counterpart: ordered elements packed 31 to a node, with the same ordered surface and an in-order walk over contiguous arrays instead of successor pointers. Beats SortedSet on the interleaved insert + membership + range-scan workload, and stores no values, so the memory saving is larger still.