Snap sync: StateSyncPivot.UpdateHeaderForcefully has no distance guard, so forced pivot updates re-pin the pivot to the head and livelock Phase 1 indefinitely

[stdevMac]

Description

StateSyncPivot.UpdateHeaderForcefully has no distance guard, so every forced pivot update re-pins the snap pivot to the current head. On a chain whose head moves between two consecutive failure streaks, each forced update invalidates the state root of every in-flight and queued range, which manufactures the next streak. The loop is self-sustaining, and because the pivot never falls behind the head it also starves the natural re-pivot path — the one that would place the pivot at a safe distance behind the head.

The observed result is a snap sync that makes literally zero progress, indefinitely, while looking busy: peers connected, requests dispatched and answered, and no error in the log.

This was reproduced on OP Mainnet, where the 2 s block time makes the loop easy to see, but the code path is not chain-specific and not backend-specific.

Steps to Reproduce

1. Start a fresh snap sync on a chain with a short block time (OP Mainnet reproduces it reliably).
2. Let it run. Watch for Changing sync ... at pivot: N lines with reason too many empty responses.
3. Compare the reported State Ranges (Phase 1) percentage over hours.

Actual behavior

Over a 60 hour window on a node started from scratch:

observable	value
forced pivot updates in the retained window	74,469
of which reason too many empty responses	74,469 (100%)
of which reason distance from HEAD (natural path)	0
rate, hourly, 60 consecutive hours	1,201 - 1,303, i.e. one every 2.88 s
average pivot step	1.5 blocks (max 3)
distinct values of State Ranges (Phase 1) over the whole history	exactly one: 0.00 %
nethermind_snap_synced_accounts / storage_slots / codes / state_synced	0 / 0 / 0 / 0
peers	5 (3 Geth, 1 Reth, 1 Erigon), MaxActivePeers=25
dominant peer-state line	Active: None | Sleeping: 3 Snap (566 of 713 samples)
samples with a snap request in flight	313 of 713

The pivot advances at exactly chain speed (1.5 blocks x 1,250/h is about 1,875 blocks/h, against the chain's 1,800 blocks/h), so it never falls 96 behind and the natural path never gets a turn. Not one account was committed in 60 hours.

Expected behavior

A forced pivot update should place the pivot a bounded distance behind the head, exactly as the natural path does, so that the ranges already in flight are not invalidated by the update that was meant to unstick them. Failing that, a snap sync that has committed nothing for hours should say so rather than churn silently.

Additional context

src/Nethermind/Nethermind.Synchronization/FastSync/StateSyncPivot.cs:

public BlockHeader? GetPivotHeader()          // the natural path
{
    ulong target = (blockTree.BestSuggestedHeader?.Number ?? 0UL) + syncConfig.StateMinDistanceFromHead;
    ulong best = _bestHeader?.Number ?? 0UL;
    if (_bestHeader is null || (target > best && target - best >= syncConfig.StateMaxDistanceFromHead))
        TrySetNewBestHeader($"distance from HEAD:{Diff}");
    return _bestHeader;
}

public void UpdateHeaderForcefully()          // the forced path
{
    ulong target = (blockTree.BestSuggestedHeader?.Number ?? 0UL) + syncConfig.StateMinDistanceFromHead;
    if (_bestHeader is null || target > _bestHeader.Number)
        TrySetNewBestHeader("too many empty responses");
}

Defaults are StateMinDistanceFromHead = 32 and StateMaxDistanceFromHead = 128 (SyncConfig.cs:85-86).

• The natural path fires only when head - pivot >= 96. That is the design: keep the pivot meaningfully behind the head.
• The forced path fires when head + 32 > pivot. Since pivot <= head always holds, that condition is always true. There is no guard; every call sets pivot := head, because TrySetNewBestHeader uses BestSuggestedHeader.

The caller is SnapSyncFeed.AnalyzeResponsePerPeer (SnapSyncFeed.cs:221 and :242), which calls UpdatePivot() after AllowedInvalidResponses = 5 consecutive non-OK results with no success in the window.

Not an 2.0.0-rc regression: StateSyncPivot.cs is byte-identical between e6d3d2f5ab and master.

Related: #13155 is a different way into the same "endless empty responses, pivot changed, nothing written" outcome — there the empty replies come from accounts whose storage was emptied between discovery and fetch. Both share the property that the client can churn forever with no warning and no abandon path. A fix for the pivot guard here does not remove the need for that one, or vice versa.

Desktop

• Operating System: Linux x64
• Version: 2.0.0-rc (e6d3d2f5ab); StateSyncPivot.cs identical on master
• Installation Method: Docker
• Consensus Client: op-node (OP Mainnet); L1 fully synced and the consensus client healthy throughout