feat: let a consumer install the per-run hash seed - #119
Conversation
The zkEVM build has no entropy source, so its `UInt256.GetHashCode` seeds are compile-time constants. Every guest of a given version therefore buckets a given key identically for ever: a colliding key set can be found offline against the published binary and replayed against every prover, turning constant-time lookups linear. EIP-8025 asks a guest to mix a per-payload value - `new_payload_request_root` - into its hash function, and there was no way to hand one to this library. `UInt256.SeedHashes(uint)` is that way. Both builds honour it and both replace every seed they hash with, so the two differ only in where they start: the zkEVM build from constants, the standard build from the seed it already draws per process. `Int256.GetHashCode` delegates here, so it follows. Two details worth the review: - The seeds move to a nested `RunSeed` type. Mutating them then leaves `UInt256`'s own statics immutable after their constructor, which is what lets NativeAOT freeze them - and on the standard build it also takes the RNG call out of `UInt256`'s own class constructor. - The 32-bit seed is spread over 64 bits by splitmix64's finalizer, chained once per derived seed, so one changed seed bit moves a whole seed word rather than half of one, and no two run seeds can land on the same derived one.
The type test inside GenericEqualityComparer<T> put one key type's problem in a comparer that serves every key type. UInt256Comparer is that comparer instead: it hashes a slot key through the run-seeded mixer, and GetOptimized() hands it to the guest and null to the host, the same shape GenericEqualityComparer.GetOptimized already uses. PersistentStorageProvider had a private comparer doing exactly this, so it now shares this one and the duplicate is gone. Also marks where UInt256.SeedHashes belongs once Nethermind.Numerics.Int256 ships it (NethermindEth/int256#119), which is what makes the routing above a second line rather than the only one.
Accept a UInt256 seed and mix its four limbs before the scalar hash loses input bits, preventing the old CRC collision family from surviving every seed. Preserve the AES and standard XxHash paths, deriving their seeds from the new input. Document that callers must install a private, cryptographically random 256-bit seed before creating hash-keyed collections and replace the full seed between runs. Add regression coverage for CRC collision families, cross-seed cancellation sets, each seed limb, distribution, and an independent BigInteger reference. Remove the unused CRC mixer and its superseded tests.
`low ^ high` of a widening product is zero whenever either factor is, and `GetMultiplyHashCode` XORs the seed into the key limbs before multiplying the pairs. So a key matching the seed in one limb erased the limb folded with it, and matching one limb of each half collapsed every key onto a single value. Reseeding is not an answer: the guest's seed is `new_payload_request_root`, which is public and known to whoever assembles the payload, so the set is constructible for the payload in hand rather than precomputed against the binary. `MultiplyFold` now carries its factors past the product, which also covers `MumFold` and the AES final fold - fixing only the halves would leave `MumFold(a, b) == 0` reachable through `a == 0x9E3779B97F4A7C15`, collapsing every key with that low half onto hash 0. Separately, the product is commutative, so folding a pair without position-separating constants gave a half the same value when its two seed-masked words were exchanged - a colliding pair for every key, needing no cancellation at all. The pairs now fold through `MumFold`, whose asymmetric constants separate the positions. Measured on the guest build with intrinsics disabled, 512 keys per case: one limb matched, its partner free, 1 distinct hash before and 512 after; one limb of each half matched, 1 before and 512 after; a half's words exchanged, 512 collisions before and 0 after. Control and single-bit input avalanche are unchanged. `MultiplyHash_ReseedingBreaksCollisionSet` pinned the family, so it becomes `MultiplyHash_MatchingTheSeedDoesNotCancelALimb`, asserting the same constructed set is distributed under its own seed. The exchange case and an updated BigInteger reference follow.
|
@claude review - is the issue solved now? |
Carrying the factors past the product removed the *constant* output a zero factor used to give, but not the family: `low ^ high ^ a ^ b` with a zero factor returns the other factor verbatim, so the fold becomes the identity rather than a constant. Chaining that collapses the whole mixer. With `K1 = 0x9E3779B97F4A7C15` and `K2 = 0xBF58476D1CE4E5B9`, taking `u0 = s0^K1` and `u2 = s2^K1` zeroes each half's first factor, `u1 = s1^K2^K1` carries the low half's output onto the outer fold's constant, and the hash reduces to `FoldHash(u3^s3)` - so `u3 = s3 ^ (t | t<<32)` gives hash `0` for every `t`. Measured on the scalar mixer, 4096 such keys: 1 distinct hash. The AES path is reachable the same way, because a known seed means known round keys: pick the round key freely, invert both rounds to the `mixed` whose low word zeroes the final fold's first factor, and 4096 forged keys also gave 1 distinct hash on the standard x64 build. An avalanche finaliser does not answer this - splitmix64's is a bijection with a cheap inverse, so an attacker picks a preimage of any zero-folding value. `FoldHash` therefore closes with another `MumFold`, against a constant no key can reach. Its factor is a fixed non-zero value, so the fold does not degenerate and inverting it is the same problem the mixer already rests on: the two families above now measure 4096 distinct hashes each, and what remains is a 32-bit hash's generic search. Cost is one widening multiply per hash. `MultiplyHash_MatchingTheSeedDoesNotCancelALimb` never exercised the cancellation its docs described: once the halves went through `MumFold`, a factor is `key ^ seed ^ constant`, so pinning `key[i] == seed[i]` left it at `K1`/`K2` and it could not be zero. Reverting the factor carry left all four cases green. It becomes `MultiplyHash_ZeroingAFoldFactorDoesNotEraseItsPartner`, using the value that does zero a factor, and now fails on all four limbs without the carry. `MultiplyHash_ChainingZeroedFoldFactorsDoesNotCollapse` pins the closing fold, and the BigInteger reference models it. Also: the fixture drew a fresh random seed at teardown rather than leaving a known one installed process-wide, since `SeedHashes` is write-only and the fixture's seed would otherwise be inherited by whatever ran next; the zkEVM build caches its AES round keys as the standard build does, instead of rebuilding both vectors from the mutable static on every hash; and `MumFold` returns `ulong`, dropping the casts at its call sites.
# Conflicts: # src/Nethermind.Int256/UInt256.std.cs # src/Nethermind.Int256/UInt256.zkevm.cs
|
@LukaszRozmej fixed conflicts and did a codex review as follows: Findings [Test quality: hot path with no benchmark] UInt256.cs:1343-1352. GetHashCode gains a widening multiply per call, and the seeds move from static readonly primitives, which Tier-1 JIT folds into immediates and a constant vector load, to mutable statics that must be loaded on every hash. The PR reports code size only and says so. There is no GetHashCode benchmark in the benchmark project. UInt256 is the key type of the storage-cell dictionaries in Nethermind, so a BenchmarkDotNet number for the AES path and the scalar path, before and after, belongs in the PR before the version bump ships it. Summary |
Changes
The zkEVM build starts with fixed hash seeds and previously exposed no way for a guest to replace them. Add
UInt256.SeedHashes(in UInt256 seed)to install a full-width 256-bit seed in both standard and zkEVM builds.Int256.GetHashCodedelegates toUInt256and follows the same seed.RunSeedtype soUInt256's own statics remain immutable after initialization. Both builds cache the AES round keys there rather than rebuilding them per hash.low ^ highfold made collisions constructible rather than searchable once the seed is public, and all three are addressed:MultiplyFoldcarries its factors past the product (low ^ high ^ a ^ b), the scalar mixer folds each limb pair throughMumFoldrather thanMultiplyFold, andFoldHashcloses with a furtherMumFoldagainst a constant no key can reach. See below.GetXxHashCode, its dedicated test file, two unused imports, and theSystem.IO.Hashingpackage reference and central version. The packed library now has no consumer NuGet dependencies; SourceLink remains a private build dependency.Payload-derived seeding is motivated by the proposed EIP-8025 update and zkevm-standards#41. The guest's
new_payload_request_rootis public and identical across retries and provers for the same payload. Hash-flooding protection depends on the seed not being known before an adversary chooses keys; this API does not provide private entropy or cryptographic authentication.Mixer changes, and why a public seed needs them
A per-payload seed removes the precomputed colliding set EIP-8025 is about, but the guest's seed is
public, so it does not remove a set built for the payload in hand. Three properties of
low ^ highmadethat set free to construct rather than something to search for.
A zero product erased its factors.
low ^ highis zero whenever either factor is, and the scalarmixer XORs the seed into the key limbs before multiplying the pairs. So a key matching the seed in one
limb erased the limb folded with it, and matching one limb of each half collapsed every key onto a
single value.
MultiplyFoldcarries the factors past the product, which is why the carry lives thererather than in the mixer: fixing only the halves leaves
MumFold(a, b) == 0reachable througha == 0x9E3779B97F4A7C15, collapsing every key with that low half onto hash0regardless of its highhalf.
The product is commutative. Folding a pair without position-separating constants gave a half the
same value when its two seed-masked words were exchanged — a colliding pair for every key, requiring no
cancellation at all.
MumFold's asymmetric constants separate the positions, so the limb pairs gothrough it.
The carried factors chained. Carrying the factors past the product removes the constant output a
zero factor gave, but not the family:
low ^ high ^ a ^ bwith a zero factor returns the other factorverbatim, so the fold becomes the identity rather than a constant, and three of those compose. With
K1 = 0x9E3779B97F4A7C15andK2 = 0xBF58476D1CE4E5B9,u0 = s0^K1andu2 = s2^K1zero each half'sfirst factor,
u1 = s1^K2^K1carries the low half's output onto the outer fold's constant, and the hashreduces to
FoldHash(u3^s3)— sou3 = s3 ^ (t | t<<32)gives hash0for everyt. The AES path isreachable the same way: a known seed means known round keys, so picking the round key freely and
inverting both rounds reaches any mixer input, including one whose low word zeroes the final fold's
first factor.
An avalanche finaliser does not answer this. splitmix64's is a bijection with a cheap inverse, so an
adversary picks a preimage of any value that folds to zero.
FoldHashtherefore closes with anotherMumFold, against a constant no key can reach: its factor is a fixed non-zero value, so the fold cannotdegenerate to the identity, and inverting it is the same problem the mixer already rests on. Cost is one
widening multiply per hash.
Measured on the zkEVM build with intrinsics disabled, so
GetHashCode()takes the scalar mixer, 512keys per case under one installed seed:
low ^ highSince the halves now fold through
MumFold, the value that zeroes a factor isseed ^ constantratherthan the seed itself; the first two rows describe the input as it was before that routing change, and
the regression tests use the value that zeroes a factor today.
The chained family, 4096 keys per case, before and after the closing fold:
What remains is the generic bound of a 32-bit hash, which is not specific to this mixer. With the seed
known, searching for a colliding set costs roughly
2^(32(k-1)/k)draws for a k-way set; measured on thesame build, a pair took a mean of 98,366 draws over 20 trials, three-way 2.8 M, four-way 31.6 M and
five-way 86.0 M. Flooding a bucket needs many keys in it, so that cost climbs steeply — which is the
difference the three fixes above make, since each property handed out a set of unbounded size for free.
This is an argument about three structural shortcuts, not a cryptanalysis of the multiply structure.
Code generation
Review measurements at
39276e3found the standard AES wrapper at 109 bytes without a stack frame, versus 98 bytes with a stack slot in the base. Each round-key load is a 16-byte memory operand; the second round consumes the other seed half. UsingMultiply64removes the previous stack store/load around multiplication. Both scalar variants inline without calls (134 bytes for the hardware multiply form, 363 bytes for the software form). These are code-generation observations, not end-to-end throughput measurements, and they predate both fold changes: the carry adds two XOR instructions per fold, and the closing fold adds one widening multiply plus its XORs per hash.Testing
UInt256HashSeedTestscovers replacement, distribution, and every seed bit both individually andpaired with its counterpart 128 bits away. Existing hash tests exercise the public path and the scalar
mixer. The fixture draws a fresh random seed at teardown, since
SeedHashesis write-only and a seedleft installed there would otherwise be inherited by whatever runs next.
Each fix is pinned by a test that fails without it:
MultiplyHash_ZeroingAFoldFactorDoesNotEraseItsPartnerconstructs the cancellation family under thevalue that zeroes a factor and asserts it distributed. It replaces
MultiplyHash_MatchingTheSeedDoesNotCancelALimb, which pinned nothing once the halves went throughMumFold: a factor iskey ^ seed ^ constant, sokey[i] == seed[i]left it atK1/K2and itcould not be zero — reverting the carry left all four of its cases green. The replacement fails on all
four limbs without the carry.
MultiplyHash_ExchangingAHalfsWordsChangesTheHashcovers the commutativity half.MultiplyHash_ChainingZeroedFoldFactorsDoesNotCollapsecovers the chain, and fails with the closingfold removed.
MultiplyHash_MatchesBigIntegerReferencemodels the composition explicitly, which is what pins thehalves going through
MumFoldand the closing fold being there at all.Full suite, Release on x64, in the four configurations CI covers:
The skip is the AES-only distribution test.
dotnet packsucceeds. Inspection of the resulting package confirms bothimpl/stdandimpl/zkevmassemblies and an empty consumer dependency group.Consumer
Nethermind #13166 installs the full payload root into its own span mixers and uses
UInt256Comparerfor guest storage-slot maps. It still uses the published int256 package and does not yet call this API. A subsequent package upgrade and explicitUInt256.SeedHashes(in root)call are needed to seed the package's default comparer too. The version bump remains a separate PR.