LightHouse Decentralized Checkpoint Sync Project Proposal Update by AbolareRoheemah · Pull Request #383 · eth-protocol-fellows/cohort-seven

AbolareRoheemah · 2026-09-02T11:59:16Z

Update: Decentralized CL Checkpoint Sync proposal

Revises Phase 1 wording to align with Etan's new spec and correct two inaccuracies
Also adds a testing subsection for Phase 1a/1b (unit, integration, consensus-spec, p2p, adversarial) and a phase-interdependency diagram for clarity.

…ed the backfill endpoint suggested by Etan

…into project-proposal

aarishnaiyer · 2026-09-03T21:41:18Z

Looks great Roheemah, the updated spec alignment and interdependency diagram really clarify the division of work. One thing is that my Phase 3 work on --wss-checkpoint verification (PR #9917) fits into the Phase 1b/3 transition since it adds the trusted root verification before checkpoint initialization. I'll sync with Yee on the BeaconStateSnapshot interface before starting the BeaconStatePartsByRange implementation.

AbolareRoheemah · 2026-09-05T13:58:19Z

Hi @taxmeifyoucan @etan-status. Trust you're doing good.
We made changes to our proposal based on the new decentralized CL checkpoint sync spec draft by Etan. Your reviews would be very much appreciated 🙇‍♀️

etan-status

I like the "No changes to scope, timeline, or fellow assignments" in the PR description 😅 But yeah, it's still alike in spirit.

etan-status · 2026-09-06T03:04:05Z

+
+This task:
+- Spawns once the node reaches `SyncState::Synced`, gated on `store.get_historic_state_limits()` rather than `BackFillState::Completed` — block backfill only guarantees historical *blocks* are available, not the *states* this task needs, so it doesn't need to wait for it and can run concurrently with it
+- Walks backward, from the most recently finalized sync committee period toward the node's earliest available state, so the periods most likely to be requested by peers are backfilled first


Mind that the lighthouse database may not yet support filling the data in reverse order, due to certain assumptions in the logic accessing it. I'm not deeply familiar with their implementation, but point this out mostly as a general aspect when dealing with databases.

etan-status · 2026-09-06T03:06:35Z

+**The Problem:** Lighthouse already computes light client proofs for every block during sync, but the recency guard in `import_block_update_metrics_and_events` and the bounded channel (`LIGHT_CLIENT_SERVER_CHANNEL_CAPACITY = 32`) prevent historical data from reaching the database. Additionally, `get_light_client_bootstrap` explicitly lacks a backfill mechanism. The result: checkpoint-synced nodes cannot serve historical LC data to peers.
+
+**The Solution** 
+**Phase 1a - Local Collection:** Implement a post-sync backfill task that walks the finalized chain backward, from the most recently finalized sync committee period to the node's earliest available *state* (per `store.get_historic_state_limits()`). (Note: this is bounded by whichever `BeaconState`s the node actually retains — a checkpoint-synced, non-archive node cannot walk back to Altair; only an archive node can. Backfill coverage is therefore a function of node configuration, not a project guarantee of full Altair-to-present coverage on every node type.) For each sync committee period, call the existing block-import light client update path — `recompute_and_cache_updates` — for every block in the period. It already retains only the spec-best update per period via `is_better_light_client_update`, so no separate candidate-selection step is needed; it stores the canonical `LightClientUpdate` and `SyncCommitteeBranch` for whichever block wins that comparison.


You'll likely have to replay BeaconState from the database to achieve this, which is a computationally heavy operation. Must be careful to not interfere with normal node operations / validator duties.

For Nimbus, we only compute historical data when the user explicitly chooses --light-client-data-import-mode=full or =on-demand, with full taking hours to initialize (https://nimbus.guide/light-client-data.html#configuration), and on-demand risking remotely triggerable lag-spikes; it's not advised to replay states unnecessarily on a node that also has validators

etan-status · 2026-09-06T03:09:52Z

+**Phase 1a - Local Collection:** Implement a post-sync backfill task that walks the finalized chain backward, from the most recently finalized sync committee period to the node's earliest available *state* (per `store.get_historic_state_limits()`). (Note: this is bounded by whichever `BeaconState`s the node actually retains — a checkpoint-synced, non-archive node cannot walk back to Altair; only an archive node can. Backfill coverage is therefore a function of node configuration, not a project guarantee of full Altair-to-present coverage on every node type.) For each sync committee period, call the existing block-import light client update path — `recompute_and_cache_updates` — for every block in the period. It already retains only the spec-best update per period via `is_better_light_client_update`, so no separate candidate-selection step is needed; it stores the canonical `LightClientUpdate` and `SyncCommitteeBranch` for whichever block wins that comparison.
+
+This task:
+- Spawns once the node reaches `SyncState::Synced`, gated on `store.get_historic_state_limits()` rather than `BackFillState::Completed` — block backfill only guarantees historical *blocks* are available, not the *states* this task needs, so it doesn't need to wait for it and can run concurrently with it


To form a LightClientUpdate, you need the corresponding BeaconState.

Block backfill is for ~5 months of data before the earliest available BeaconState, you can't locally compute the LightClientUpdate for these (as you won't have the state)

There is no libp2p protocol for backfilling historical states.

etan-status · 2026-09-06T03:12:46Z

+This task:
+- Spawns once the node reaches `SyncState::Synced`, gated on `store.get_historic_state_limits()` rather than `BackFillState::Completed` — block backfill only guarantees historical *blocks* are available, not the *states* this task needs, so it doesn't need to wait for it and can run concurrently with it
+- Walks backward, from the most recently finalized sync committee period toward the node's earliest available state, so the periods most likely to be requested by peers are backfilled first
+- Is resumable (checks `store.get_light_client_update(period)` before processing)


In Nimbus, had to add a separate flag to indicate whether all data for the period was considered. Otherwise, you'll have a present light client update from the second block in each period, but there may still be a better one later (e.g., one with supermajority, or one that finalizes the period's start slot). So, just checking whether you have a LightClientUpdate for the period doesn't mean that work for that period is necessarily done.

etan-status · 2026-09-06T03:15:53Z

+```

-The combined effect: a Lighthouse node that synced from genesis or checkpoint cannot serve `LightClientBootstrap` or historical `LightClientUpdates` because it never generated or stored the proofs.
+Implementation follows the pattern of the existing `handle_light_client_updates_by_range` handler in `network_beacon_processor::rpc_methods.rs`:


There's also a beacon-API that you can query with curl, can be useful for development.

etan-status · 2026-09-06T03:22:21Z

+- Stream responses back via `SendResponse`

-Our solution is a trust-minimized checkpoint sync strategy borrowed from/inspired by the LC sync strategy:
+`LightClientEpochData` contains per-epoch raw block data including `sync_committee_bits`, `sync_aggregate_branch`, `finalized_checkpoint`, and `current_sync_committee` — everything a receiver needs to independently simulate `is_better_update` and verify the canonical best update for a period.


Draft spec was updated based on feedback from @jeffoodchain, it's still same concepts but fields were slightly restructured and a couple edge cases are now handled better.

etan-status · 2026-09-06T03:24:33Z

 ## Possible challenges

- **State Availability for Backfill:** The backfill task requires loading historical `BeaconStates` from the freezer DB. If a node pruned states before backfill completed, or if it checkpoint-synced and never had old states, it cannot generate historical light client data. We must document that backfill requires either archive node configuration or fetching missing data from peers (the backfill API).
+- **Protocol evolution risk:** Etan's HackMD spec is explicitly marked as a draft ("TBD and likely not yet optimal"). The `LightClientEpochData` container, chunk sizes, and endpoint paths may change during implementation. We build defensively, pinning only the parts that are stable (the existing `LightClientUpdate` storage, the `is_better_update` ranking logic) and adapting the transport layer as the spec stabilizes.


There is no "TBD and likely not yet optimal" in that draft 😅 not sure where that citation comes from

although it indeed is tbd and likely not optimal. if you have suggestions to improve it, we can still integrate feedback.

etan-status · 2026-09-06T03:28:26Z

-   ```
-   GET /eth/v1/beacon/light_client/updates/backfill?from_period={u64}&count={u64}
-   ```
+**Option B — full per-slot storage for every backfilled epoch, matching the spec as drafted — is the eventual path**, targeted as a stretch goal beyond this project's MVP scope (or a follow-on after EPF). It requires new storage design (roughly 32x more records than Option A) and should ideally mirror Nimbus's existing implementation (PR #8445) rather than Lighthouse inventing its own independently.


Not sure where the 32x is coming from, what records do we mean?

@jeffoodchain is looking into this option for prysm

Thanks for flagging this. Will fix.

And good to know Jeff's looking at this for Prysm too

does the record mean LCEpochData?

etan-status · 2026-09-06T03:29:17Z

+1. **Unit tests**
+   - `recompute_and_cache_updates` produces a correct `LightClientUpdate` and `SyncCommitteeBranch` for a known-good historical period
+   - Constructing `LightClientEpochData` from stored `LightClientUpdate`/`SyncCommitteeBranch` data round-trips correctly against the container spec
+   - Best-candidate selection (highest sync aggregate participation) picked correctly across a period with missed slots


is_better_update is not just about highest participation, it also considers finality and has a couple tiebreakers, to be deterministic (i.e., everyone computes the same when running the algo).

etan-status · 2026-09-06T03:32:41Z

+**Storage scoping decision:** Phase 1a's storage (`LightClientUpdate`/`SyncCommitteeBranch`/`SyncCommittee`) retains only the winning block per sync committee period — not per-slot data for every epoch in that period. The `LightClientDataBackfillByRange` endpoint as drafted requests by individual `epoch`, and each `LightClientEpochData` response is expected to contain full per-slot `block_data` for that epoch, including epochs that did not win their period's `is_better_light_client_update` comparison. This is a real granularity gap: the endpoint's purpose (per the spec: "verifying every single field... then simulating is_better_update") is to let the requester independently recompute which block was best, rather than trust the server's selection — which requires access to the non-winning epochs' raw data too.

-2. **Design the backfill API** — Once nodes have the data, we need a way to request it. This is the first TBD endpoint and it should look something like:
+**For this project's MVP, we're building Option A: scope the endpoint to what Phase 1a already stores** — serve period-best `LightClientUpdate` history rather than the full per-epoch `LightClientEpochData` container. This requires no new storage, ships against the existing consensus-specs `light_client_data_collection` test format (which does not test `LightClientEpochData`), and fits the project timeline. The trade-off is accepted for MVP: a requester has to trust the server's "best block" selection rather than independently verify it from raw per-epoch data, so this does not yet deliver the endpoint's full trust-minimization goal as drafted.


So why is there an option B section below, if "we're building option A"?

And, the /eth2/beacon_chain/req/light_client_data_backfill_by_range/0/ is mentioned above, as phase 1b (is that option B?)

Should this mean "phase 1a" rather than "option A"?

Thanks for all the review @etan-status really appreciate them 🫡

So Option B is there since it's the more spec aligned option. It's like saying "this is what the spec really says but this is what we are building for now as part of the MVP"

AbolareRoheemah added 6 commits August 2, 2026 23:26

project proposal

f74d5b0

changed chunking strategy, assigned fellows to tasks

9f0c656

replaced partly wrong code under recency gap, reworded phase1 and add…

13b890d

…ed the backfill endpoint suggested by Etan

updated proposal based on Etan's new documentation

a4a2baf

add more details and testing information, majorly to phase 1

39fb262

Merge branch 'main' of https://github.com/AbolareRoheemah/cohort-seven …

170e3d2

…into project-proposal

AbolareRoheemah marked this pull request as draft September 2, 2026 11:59

updated several sections in Phase 1

a424d65

AbolareRoheemah changed the title ~~LighHouse Decentralized Checkpoint Sync Project Proposal Update~~ LightHouse Decentralized Checkpoint Sync Project Proposal Update Sep 5, 2026

etan-status approved these changes Sep 6, 2026

View reviewed changes

edited proposal based on Etan's review comments

f7cde2f

AbolareRoheemah marked this pull request as ready for review September 13, 2026 22:42

updated explanation on state availability for backfill

03d4dbc

AbolareRoheemah mentioned this pull request Sep 14, 2026

Lc historical backfill sigp/lighthouse#10050

Draft

Uh oh!

Conversation

AbolareRoheemah commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update: Decentralized CL Checkpoint Sync proposal

Uh oh!

aarishnaiyer commented Sep 3, 2026

Uh oh!

AbolareRoheemah commented Sep 5, 2026

Uh oh!

etan-status left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AbolareRoheemah commented Sep 2, 2026 •

edited

Loading