Downscore and retry custody failures #7510

dapplion · 2025-05-22T05:17:34Z

Issue Addressed

Partially addresses

Improve range sync with PeerDAS #6258

Proposed Changes

TBD

1000 lines are for new unit tests :)

Questions / TODO

Should custody_by_range requests try all possible peers before giving up? i.e. should they ignore the failures counter for custody failures? Add a test for this behaviour.

Tests to add

Peer does not send columns at all on specific index
We find no-one serving columns on a specific index, how to recover
Random tests where most peer are faulty and we have to keep cycling them. Use randomness and run the tests with different levels of fault % :=> build a simulator
Permanently fail a single column many times, but then resolve the rest such that we can reconstruct.
Send non-matching data columns
Test downscoring of custody failures
Test the fallback mechanism of peer selection
Test a syncing chain reaching 0 peers, both for forward sync and backwards sync
Test forwards sync on Fulu with blocks without data

dapplion · 2025-05-22T05:38:42Z

beacon_node/network/src/sync/manager.rs

@@ -545,6 +545,9 @@ impl<T: BeaconChainTypes> SyncManager<T> {
        for (id, result) in self.network.continue_custody_by_root_requests() {
            self.on_custody_by_root_result(id, result);
        }
+        for (id, result) in self.network.continue_custody_by_range_requests() {
+            self.on_custody_by_range_result(id, result);
+        }


Every interval (15 sec), we call continue_custody_by_range / by_root requests, which will cause the request to error if it has been alive for too long. This allows the requests to not error immediately if they do not have enough custody peers.

dapplion · 2025-05-22T05:39:02Z

beacon_node/network/src/sync/manager.rs

@@ -442,6 +439,9 @@ impl<T: BeaconChainTypes> SyncManager<T> {
        for (id, result) in self.network.continue_custody_by_root_requests() {
            self.on_custody_by_root_result(id, result);
        }
+        for (id, result) in self.network.continue_custody_by_range_requests() {
+            self.on_custody_by_range_result(id, result);
+        }


Every time a peer joins, attempt to progress custody_by_root and custody_by_range requests

dapplion · 2025-05-22T05:40:04Z

beacon_node/network/src/sync/mod.rs

@@ -3,7 +3,6 @@
 //! Stores the various syncing methods for the beacon chain.
 mod backfill_sync;
 mod block_lookups;
-mod block_sidecar_coupling;


Logic moved to beacon_node/network/src/sync/network_context/block_components_by_range.rs

dapplion · 2025-05-22T05:40:41Z

beacon_node/network/src/sync/network_context.rs

 };

-pub mod custody;


Renamed existing custody module to custody_by_root and added a new one custody_by_range

dapplion · 2025-05-22T05:41:40Z

beacon_node/network/src/sync/network_context.rs

+    }
+
+    /// Returns the ids of all active requests
+    pub fn active_requests(&mut self) -> impl Iterator<Item = (SyncRequestId, &PeerId)> {


Changed this signature for tests, to have access to all active RPC requests

dapplion · 2025-05-22T05:42:14Z

beacon_node/network/src/sync/network_context.rs

-
-                Ok((requests, column_to_peer_map))
-            })
-            .transpose()?;


network_context no longer spawns _by_range requests, this logic is now inside BlockComponentsByRangeRequest

dapplion · 2025-05-22T05:43:55Z

beacon_node/network/src/sync/network_context/block_components_by_range.rs

+        blocks_by_range_request:
+            ByRangeRequest<BlocksByRangeRequestId, Vec<Arc<SignedBeaconBlock<E>>>>,
+        blobs_by_range_request: ByRangeRequest<BlobsByRangeRequestId, Vec<Arc<BlobSidecar<E>>>>,
+    },


Maintains the same behaviour for mainnet:

deneb: issue blocks + blobs requests at the same time

fulu: issue blocks request first, then columns

dapplion · 2025-05-22T06:10:00Z

beacon_node/network/src/sync/network_context.rs

 }

 #[derive(Debug, PartialEq, Eq)]
 pub enum RpcRequestSendError {
-    /// No peer available matching the required criteria
-    NoPeer(NoPeerError),


Requests do not error now on send if they don't have peers. Instead, custody_by_root and custody_by_range requests are left idle for some time, expecting peers.

jimmygchen

@dapplion I've started reviewing this but haven't got to the meat of the changes - will continue tomorrow but I've submitted my comments so far.

jimmygchen · 2025-05-26T12:42:03Z

beacon_node/lighthouse_network/src/types/globals.rs

+        trusted_peers: Vec<PeerId>,
+        config: Arc<NetworkConfig>,
+        spec: Arc<ChainSpec>,
+        is_supernode: bool,


I don't think it matters currently - but thought might be worth mentioning - we use config.subscribe_all_data_column_subnets for determine the gossip topics, and it doesn't seem like we use it for determining is_supernode elsewhere, but something to keep in mind - if we want consistency maybe we can derive is_supernode from network config, but that may not be relevant from once we move to validator custody.

Good point, we should revisit after impl validator custody. Since this function is for unit tests without gossip, it doesn't matter

jimmygchen · 2025-05-26T13:12:53Z

beacon_node/network/src/sync/manager.rs

@@ -1186,10 +1189,9 @@ impl<T: BeaconChainTypes> SyncManager<T> {
        block: RpcEvent<Arc<SignedBeaconBlock<T::EthSpec>>>,
    ) {
        if let Some(resp) = self.network.on_blocks_by_range_response(id, peer_id, block) {
-            self.on_range_components_response(
+            self.on_block_components_by_range_response(


I often get confused here as we have layers of on_xxx_by_range_response functions, what about renaming:

resp to result

self.on_block_components_by_range_response to self.on_range_components_rpc_result, and add a comment describing it:

Suggested change

self.on_block_components_by_range_response(

// Process it further if the response results in a completed request or a failure that needs to be handled.

self.on_range_components_rpc_result(

also might help to add some comment to self.network.on_blocks_by_range_response. I wrote these notes while reviewing:

/// Processes the `BlocksByRange` response (`RpcEvent`), depending on the event, it may: /// - update the active request state; or do nothing. /// - complete the active request and return an `Ok` response containing a list of response objects. /// - propagate the error if necessary. pub(crate) fn on_blocks_by_range_response( &mut self, id: BlocksByRangeRequestId, peer_id: PeerId, rpc_event: RpcEvent<Arc<SignedBeaconBlock<T::EthSpec>>>, ) -> Option<RpcResponseResult<Vec<Arc<SignedBeaconBlock<T::EthSpec>>>>> {

beacon_node/network/src/sync/manager.rs

beacon_node/network/src/sync/network_context.rs

jimmygchen · 2025-05-27T05:49:51Z