fix(consensus): stabilize TestByzantinePrevoteEquivocation flake

[rootulp]

Summary

Stabilize TestByzantinePrevoteEquivocation — the most frequent flake in celestia-core CI (8 of the last 19 failed Test workflow runs on main between 2026-02-09 and 2026-04-15). Five small, independently-valuable changes:

1. Send both conflicting prevotes to every peer (consensus/byzantine_test.go). Splitting the votes across peers (one variant to half, the other to the rest) does not work reliably: the consensus reactor's HasVote gossip optimization (consensus/reactor.go:PickVoteToSend) excludes a validator's index from gossip selection once any peer reports holding any vote from that validator, regardless of which BlockID the vote is for. Once the byz-bit is set in each peer's HasVote bitarray, no peer ever forwards its own variant to peers holding the other variant, so no peer sees both conflicting votes and DuplicateVoteEvidence cannot form via gossip. Sending both votes directly to every peer makes the conflict detectable on first receipt without relying on gossip. Upstream maintainer cason identified this exact mechanism in consensus: TestByzantinePrevoteEquivocation is flaky cometbft/cometbft#1917 ("a node does not send one of the conflicting Prevote messages to a peer because it realized that that peer does not need it anymore").

2. Use the upstream mock ticker (consensus/byzantine_test.go). Upstream's TestByzantinePrevoteEquivocation uses newMockTickerFunc(true) (cometbft/cometbft internal/consensus/byzantine_test.go:45) — the celestia-core fork was using a real NewTimeoutTicker(). With a real ticker, TimeoutPropose can fire on the byzantine before the proposal arrives via gossip, causing doPrevote to run with rs.ProposalBlock = nil; prevote1 then collapses to a vote for nil identical to prevote2 and no equivocation evidence forms. The mock ticker fires the new-height timer once at startup and then never; consensus advances purely on +2/3 thresholds, eliminating that race.

3. Peer-mesh wait (consensus/byzantine_test.go). MakeConnectedSwitches dials synchronously but p2p handshake and peer registration complete asynchronously. Without a wait, the byzantine node could reach its doPrevote override with a partial peer set and fire conflicting votes at fewer than nValidators-1 peers — which the evidence pool cannot reconstruct into DuplicateVoteEvidence. New require.Eventually polls each reactor's peer set (10s budget, 50ms tick) until every validator sees nValidators-1 peers.

4. Two-stage polling replaces the 120s deadline. Stage 1 polls each validator's evidence pool (30s / 100ms) for a DuplicateVoteEvidence at the expected height. Stage 2 polls each validator's block store (60s / 200ms) for any block containing evidence. The split isolates "evidence detected" from "evidence committed" so that future failures point at the correct failing stage. Test wall time drops from up-to-120s to ~0.5s on a passing run.

5. Structured instrumentation. [byz] log lines capture peer count at prevote time and peer.Send return values, surfacing in CI logs so future flakes can be diagnosed without a local repro.

This PR supersedes the timeout-bump approaches in #2810 and #2850 — the old block-watch-limit and bare time.After are deleted entirely.

Test plan

• go test -tags deadlock -v -run '^TestByzantinePrevoteEquivocation$' ./consensus/ — passes locally in ~0.5s
• Local stress test: 50/50 pass with this PR (vs. 5/20 fails on `main` — 25% flake rate on macOS)
• CI is green on this PR
• Re-run CI a few times to observe flake rate over multiple runs (target: 0 failures)

Closes #2200.
Closes https://linear.app/celestia/issue/PROTOCO-580/flaky-test-testbyzantineprevoteequivocation-in-consensusbyzantine

🤖 Generated with Claude Code

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 3 additional findings.

[evan-forbes]

we were going to do something like this before generally not opposed however this test is notoriously complex yet important

I had codex run locally to confirm and actually still hit a failure but I'm unsure if this is separate

Failure was at /home/evan/src/celestiaorg/worktrees/celestia-core-pr-2950/consensus/byzantine_test.go:310: evidence pool never received
DuplicateVoteEvidence at height 2. The log showed the new peer wait succeeded in the important way: peer count was 3, and all three equivocation sends
returned sent=true. So either the peer-mesh wait is insufficient, or the new PendingEvidence polling is racing a transient pending state. Either way,
this PR does not yet justify “stabilize” for this test.

[rootulp]

[Claude generated]

Thanks for the careful review @evan-forbes. You were right that the original three changes werent sufficient — the failure you hit locally is the same flake the PR was supposed to fix. Digging deeper revealed a fourth root cause that the peer-mesh wait alone could not address:

The consensus reactors HasVote gossip optimization (consensus/reactor.go:PickVoteToSend) subtracts each peers HasVote bitarray from the local vote-set bitarray when picking a vote to gossip. The bitarray is indexed by validator, not by (validator, BlockID). So once any peer reports "byz has voted at h/r/type", every other peers gossip routine sees that index already covered and never forwards its own variant.

When the byzantine splits the votes (one variant to peer A, the other to peers B and C), peer A holds prevote1 (block) and peers B/C hold prevote2 (nil). Peer A broadcasts HasVote saying "byz has voted"; peers B/C do the same; everyone marks the byz-bit set in their peer-state for everyone else. Now peer As gossip wont send prevote1 to B or C (their peer-state says they already have byzs vote), and B/Cs gossip wont send prevote2 to A. Gossip cannot propagate the conflict between honest peers. The split-vote attack only forms DuplicateVoteEvidence when peer As HasVote for prevote1 happens to outrun B/Cs HasVote for prevote2; otherwise no peer sees both votes and the evidence pool stays empty — exactly your failure.

4d64bb1b2 sends both prevote1 and prevote2 directly to every peer, so each peer sees the conflict on first receipt without relying on gossip. The peer.Send returns are still logged so any future failures point cleanly at delivery vs. consensus-state issues. Locally on macOS this drops the stress-script failure rate from 5/20 on main to ~0/20 in the common case (some residual races remain when the receivers commit block 2 before processing both votes — happy to chase those in a follow-up if CI surfaces them).

[rootulp]

Quick update: I investigated the test on upstream cometbft and the same flake exists there too (cometbft/cometbft#1917, #2353). Upstream maintainer @cason identified the same root causes I found — the HasVote gossip optimization preventing conflicting-vote propagation, and receivers committing the next height before processing the second conflicting vote. They closed #1917 as deprioritized, and #2353 ("Evidence may not work consistently") is still open.

One concrete difference between upstream and the celestia-core fork: upstream uses newMockTickerFunc(true) for this test (reference) — the fork was using a real NewTimeoutTicker(). With a real ticker, TimeoutPropose can race with proposal gossip and cause doPrevote to run before rs.ProposalBlock is set; both signed prevotes then collapse to votes for nil and no equivocation evidence forms.

14e55357f switches to the upstream mock ticker. Local stress is now 50/50 on macOS (was 30/30 with real ticker, where Linux CI still flaked once at that rate). Re-running CI to confirm.