Add state diff serialization #15250

potuz · 2025-05-05T18:49:53Z

Adds custom functions to create differences between beacon states, serialize them and apply them.

This adds a package hdiff which exposes one type and two functions

type HdiffBytes struct {
	StateDiff      []byte
	ValidatorDiffs []byte
	BalancesDiff   []byte
}

func Diff(source, target state.ReadOnlyBeaconState) (HdiffBytes, error)
 
func ApplyDiff(ctx context.Context, source state.BeaconState, diff HdiffBytes) (state.BeaconState, error)

The type is exposed and the members are accessible because in a future PR we will expose functions to apply differences only to the validator set and the balance slice.

Current benchmarks for applications of a diff between a state 6 months ago in Deneb vs current Electra state point to 1.1" for deserialization and application of the diff, on AMD Ryzen 5 3600. Creation and serialization of the diff takes 530ms on this computer and produces a 27MB diff.

consensus-types/hdiff/state_diff.go

james-prysm · 2025-09-24T21:50:22Z

consensus-types/hdiff/state_diff.go

+	}
+	ret.eth1VotesAppend = ((*data)[0] == nilMarker)
+	eth1DataVotesLength := int(binary.LittleEndian.Uint64((*data)[1 : 1+8])) // lint:ignore uintcast
+	if len(*data) < 1+8+eth1DataVotesLength*eth1DataLength {


maybe we should do a max() check either 9 or 1+8+eth1DataVotesLength*eth1DataLength whichever is longer?

The length cannot be known until after checking for the first 8 bytes. Since the nilmarker anyway needs to be present, that's why I check for 9 instead of just 9

consensus-types/hdiff/state_diff.go

james-prysm · 2025-09-24T21:55:37Z

consensus-types/hdiff/state_diff.go

+	if err := ret.readSlot(&data); err != nil {
+		return nil, err
+	}
+	if err := ret.readFork(&data); err != nil {


I feel like it would be very easy to forget something here, I wonder what would be the best way to keep track to make sure we add a new time every time we touch the state

consensus-types/hdiff/state_diff.go

rkapka · 2025-09-25T11:08:36Z

consensus-types/hdiff/state_diff.go

+	earliestExitEpoch             primitives.Epoch // override.
+	consolidationBalanceToConsume primitives.Gwei  // override.
+	earliestConsolidationEpoch    primitives.Epoch // override.
+


Suggested change

consensus-types/hdiff/state_diff.go

rkapka · 2025-09-25T11:28:35Z

consensus-types/hdiff/state_diff.go

+	if err != nil {
+		return nil, err
+	}
+	ret.currentSyncCommittee, err = target.CurrentSyncCommittee()


What about using a nil marker when there is no diff in the sync committee? Same for execution payload header

consensus-types/hdiff/state_diff.go

prestonvanloon · 2025-09-30T00:29:10Z

beacon-chain/core/electra/upgrade.go

 )

+// ConvertToElectra converts a Deneb beacon state to an Electra beacon state. It does not perform any fork logic.
+func ConvertToElectra(beaconState state.BeaconState) (state.BeaconState, error) {


Yeah, i think you should reuse the code and reduce the duplicate LOC

prestonvanloon · 2025-09-30T00:33:09Z

beacon-chain/state/state-native/setters_withdrawal.go

+	b.sharedFieldReferences[types.PendingPartialWithdrawals] = stateutil.NewRef(1)
+
+	b.pendingPartialWithdrawals = pendingPartialWithdrawals
+	b.markFieldAsDirty(types.PendingPartialWithdrawals)


yeah, that could be an issue. @rkapka what do you think?

prestonvanloon · 2025-09-30T00:36:04Z

beacon-chain/state/state-native/state_trie.go

 		proposerLookahead[i] = primitives.ValidatorIndex(v)
 	}
+	// Proposer lookahead must be exactly 2 * SLOTS_PER_EPOCH in length. We fill in with zeroes instead of erroring out here
+	for i := len(proposerLookahead); i < 2*fieldparams.SlotsPerEpoch; i++ {


I think it's supposed to be params.BeaconConfig().MinSeedLookahead + 1, not just 2, although it is 2 in both cases.

prestonvanloon · 2025-09-30T00:42:26Z

consensus-types/helpers/comparisons.go

@@ -0,0 +1,109 @@
+package helpers


Sorry, we need tests for all of these 😱 . Maybe a friendly AI could help with some of that work?

Fixed in 1bdd78

Adds serialization code for state diffs. Adds code to create and apply state diffs Adds fuzz tests and benchmarks for serialization/deserialization Co-authored-by: Claude <[email protected]>

🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Inspector-Butters · 2025-10-08T11:47:08Z

consensus-types/helpers/comparisons.go

@@ -0,0 +1,109 @@
+package helpers


why not use proto.Equal(s, t) for all of these functions?

I generally dislike proto.Equal as it requires an extra serialization/allocation and it fails in an obscure manner. Eventually we want to move away from protobufs so it would even be backwards-looking

james-prysm · 2025-10-13T19:50:10Z

consensus-types/hdiff/state_diff.go

+			vals = append(vals, v.WithdrawalCredentials...)
+		}
+		vals = binary.LittleEndian.AppendUint64(vals, v.EffectiveBalance)
+		if v.Slashed {


can we reverse this to be
if !v.Slashed {
als = append(vals, nilMarker)
else {
vals = append(vals, notNilMarker)
}

why? this is just an arbitrary choice, but generally, I've used that the default value should always serialize to zero because that way the default diff is smaller. Wouldn't want to change this.

consensus-types/hdiff/state_diff.go

consensus-types/hdiff/state_diff.md

Bast Co-authored-by: Bastin <[email protected]>

potuz force-pushed the hdiff branch 4 times, most recently from d9f6001 to c461552 Compare May 5, 2025 19:34

terencechain reviewed May 5, 2025

View reviewed changes

consensus-types/hdiff/state_diff.go Outdated Show resolved Hide resolved

potuz force-pushed the hdiff branch 25 times, most recently from f41008d to 1beba7a Compare May 12, 2025 21:03

james-prysm reviewed Sep 24, 2025

View reviewed changes

consensus-types/hdiff/state_diff.go Outdated Show resolved Hide resolved

james-prysm reviewed Sep 24, 2025

View reviewed changes

consensus-types/hdiff/state_diff.go Show resolved Hide resolved

james-prysm reviewed Sep 24, 2025

View reviewed changes

consensus-types/hdiff/state_diff.go Outdated Show resolved Hide resolved

rkapka reviewed Sep 25, 2025

View reviewed changes

potuz force-pushed the hdiff branch from 0622f3c to 603aaa8 Compare September 30, 2025 00:34

prestonvanloon reviewed Sep 30, 2025

View reviewed changes

potuz and others added 8 commits October 2, 2025 10:58

Add serialization code for state diffs

79eb396

Adds serialization code for state diffs. Adds code to create and apply state diffs Adds fuzz tests and benchmarks for serialization/deserialization Co-authored-by: Claude <[email protected]>

Add Fulu support

446c342

Review #1

47bf9fa

gazelle

e8eb3d0

Fix some fuzzers

2da4900

Failing cases from the fuzzers in consensus-types/hdiff

eba2f3d

Fix more fuzz tests

be05cbb

🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

add comparison tests

7da79a8

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

potuz force-pushed the hdiff branch from 1bdd78e to 7da79a8 Compare October 2, 2025 13:58

potuz added 3 commits October 7, 2025 09:53

Use ConvertToElectra in UpgradeToElectra

bf1ea5c

Add comments on constants

228b2ea

Fix readEth1Data

3c55689

potuz force-pushed the hdiff branch from 1959ff7 to 3c55689 Compare October 7, 2025 13:55

remove colons from error messages

55f9d8b

Inspector-Butters reviewed Oct 8, 2025

View reviewed changes

rkapka previously approved these changes Oct 8, 2025

View reviewed changes

james-prysm reviewed Oct 13, 2025

View reviewed changes

consensus-types/hdiff/state_diff.go Show resolved Hide resolved

Add design doc

a9ed709

potuz dismissed rkapka’s stale review via a9ed709 October 14, 2025 17:36

Inspector-Butters reviewed Oct 15, 2025

View reviewed changes

Apply suggestions from code review

ae7a072

Bast Co-authored-by: Bastin <[email protected]>

Add state diff serialization #15250

Are you sure you want to change the base?

Add state diff serialization #15250

Uh oh!

Conversation

potuz commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rkapka Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

potuz commented May 5, 2025 •

edited

Loading

rkapka Sep 25, 2025 •

edited

Loading