Desync between redistribution game phase tracked by node and on-chain (need to investigate)

[sbackend123]

Context
An on-chain round is split into commit → reveal → claim. When round N enters claim, the agent publishes the internal sample phase and runs handleSample. At the start of handleSample, IsPlaying() is called — an eth_call to the contract’s isParticipatingInUpcomingRound(owner, committedDepth) — to check whether the node will fall in the neighbourhood for the next round (N+1). Building the sample can take a long time and may overlap with commit of round N+1; IsPlaying is not called again during that work (only once at the beginning of sample).

Problem
The bee_storageincentives_is_playing_errors metric keeps increasing steadily.
Logs on light testnet show phase failed, phase=sample, with errors like
IsPlaying: ... execution reverted: 0xe2586bcc. As I understand, selector decodes to WrongPhase().
In the contract, isParticipatingInUpcomingRound reverts with WrongPhase() only when the call lands in the on-chain reveal phase (currentPhaseReveal()). It should not revert for phase reasons during claim or commit.
If IsPlaying fails, no sample is created → the node does not participate in that game round.

Hypothesis
I suspect a desync between the phase the node believes it is in and the actual on-chain phase. Because of these frequent errors, the node rarely passes the gate before sampling and participates in the game only rarely

[sbackend123]

More context
The redistribution game runs in cycles of 152 blocks. Each round is divided into three phases:

commit — blocks 0–37
reveal — blocks 38–75
claim — blocks 76–151

Where the block number comes from
The RPC call for the block number is cached. Every n blocks (10 by default), BlockNumber fetches the real block from the network via HeaderByNumber. Between synchronizations, the node estimates the current block by adding to the last known number however many blocks have elapsed since the last sync, based on the network’s block-time.

Contract checks (IsPlaying, IsWinner) run against the real latest block via CallContract. Commit, Reveal, and Claim are sent as separate transactions, but they also execute against the current chain state.

That gives two sources of truth, and the node can run a few blocks ahead of the chain in block number.

What happens on claim entry
When phaseCheck sees claim, the claim handler runs handleClaim (checking for a win in the current round) and, in parallel, Publish(sample) → handleSample (preparing for the next round).

If no sample is created, the node will not participate in the next round.

Problem 1: the node cannot participate in the round (WrongPhase)
phaseCheck uses the computed BlockNumber and decides claim has already started, then starts the sample-creation flow. The contract, on the real latest block, is still in reveal and returns WrongPhase. After that, no sample is created.

This happens very often — in the logs, on the vast majority of claim-entry attempts.

Problem 2: false repeated claim detection at the end of the round
phaseCheck keeps prevPhase and publishes an event only when the phase changes. It assumes blocks grow monotonically: commit → reveal → claim → commit of the next round. But BlockNumber can “roll back”: after the computed block has moved into commit of the next round, the next sync with the network can return a block one lower, and phaseCheck again sees claim for the previous round. That retriggers the participation check and another attempt to create a sample.

This happens in roughly a quarter of claim detections (on some pods in some rounds). Block rollbacks occur on all pods (about 13% of phaseCheck transitions), but only about a third of those hit the claim/commit boundary and produce a false claim.

For the most part this does not block participation in the game, but it is still not correct behavior.

Possible fix
In phaseCheck, do not use the cached BlockNumber(). Instead, call HeaderByNumber and take the real latest block number from the chain. Also ignore responses where the block number has not increased — that guards against RPC rollbacks.

RPC call volume should barely change: phaseCheck already does not run on every block, only at an interval of blocks

@acud @gacevicljubisa

[acud]

That gives two sources of truth, and the node can run a few blocks ahead of the chain in block number.

that's sounds like a lot. if we check the real block number every 10 blocks (50 seconds), it means that in order to be at least one block ahead of the real number, the block time must vary by 10% on average during that time which sounds to me like a huge number. Also, for example, gnosis chain now shows avg block time 5.2s. for us to be ahead of the chain by "a few blocks" means we must be counting block time at least 20% faster than the chain which is either suggests a severe bug on our side or an unlikely scenario.

There is definitely a chance that the sample comes in late - the variance of sample creation time is through the roof (SIMD hasher should improve this - might be worth to check whether that improves it), and in many cases it comes in too late, causing to miss the commit-reveal windows.

In any case I am fine with not using the block number caching for the purpose of solving this specific problem (if it even does so - I have my doubts - if the problem is due to the time it takes to sample the reserve I'm not sure it would affect this)