fix(discovery): prune stale discv5 peers from peer store so PX stops serving dead ENRs (#3933)#3939
Open
igor-sirotin wants to merge 1 commit into
Open
fix(discovery): prune stale discv5 peers from peer store so PX stops serving dead ENRs (#3933)#3939igor-sirotin wants to merge 1 commit into
igor-sirotin wants to merge 1 commit into
Conversation
|
You can find the image built from this PR at Built from fafac55 |
igor-sirotin
added a commit
to status-im/status-go
that referenced
this pull request
Jun 9, 2026
quay.io/wakuorg/nwaku-pr:3939 is the build of logos-messaging/logos-delivery#3939, the candidate fix for the light-client PX stale-ENR flake (#7513). Also enables store peer-exchange.
igor-sirotin
added a commit
to status-im/status-go
that referenced
this pull request
Jun 9, 2026
quay.io/wakuorg/nwaku-pr:3939 is the build of logos-messaging/logos-delivery#3939, the candidate fix for the light-client PX stale-ENR flake (#7513). Also enables store peer-exchange.
igor-sirotin
added a commit
to status-im/status-go
that referenced
this pull request
Jun 9, 2026
* test: discovery mix full and light clients * docs: known issue wakufleetconfig.json not found * test: enable bridge network by default to prevent discovery * fix: logs * test(waku): use nwaku PR #3939 fleet image + num-shards-in-network=65 quay.io/wakuorg/nwaku-pr:3939 is the build of logos-messaging/logos-delivery#3939, the candidate fix for the light-client PX stale-ENR flake (#7513). Also enables store peer-exchange. * test: allow finding more peers * chore(deps): bump go-waku with filter prefer-connected-peers fix Pulls waku-org/go-waku fix/filter-prefer-connected-peers, which makes peer selection prefer already-connected peers. This stops light-client filter/lightpush from churning on stale/unreachable peers learned via peer-exchange, which was causing the flaky light-client message tests (#7513). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(waku): use released nwaku v0.38.1 fleet (drop #3939) Test the go-waku prefer-connected-peers fix in isolation: the status-go image already carries it (go.mod -> go-waku fix branch), so point the fleet at the released wakuorg/nwaku:v0.38.1 instead of the #3939 build. The client-side fix should keep light clients off stale peers regardless of fleet-side stale-ENR pruning. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(deps): bump go-waku to v0.10.2 Release tag including the prefer-connected-peers selection fix (logos-messaging/logos-delivery-go#1309), which keeps light-client filter/lightpush off stale/unreachable peers and fixes the flaky light-client message tests (#7513). Supersedes the temporary branch pin. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: go mod tidy --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
igor-sirotin
added a commit
to status-im/status-go
that referenced
this pull request
Jun 20, 2026
The light-client compat flakiness is driven by peer-exchange pollution:
the shared test fleet's PX server hands each light client the ENRs of
every other parallel cell's backends (incl. torn-down ones). The light
client churns dialing those dead peers ('all dials failed') and its
filter peer-selection intermittently can't find the connected fleet node
for the community shard ('no suitable peers found') -> missed message ->
timeout. Affects both v10.34.1 and v10.33.2 light cells, flaky.
Point the fleet (boot-1 + store) at quay.io/wakuorg/nwaku-pr:3939, the
build of logos-messaging/logos-delivery#3939 which prunes stale
Discv5-origin peers from the peer store (reconcilePeerStoreWithRoutingTable
in the discv5 search loop), so PX stops advertising dead sibling ENRs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
igor-sirotin
marked this pull request as ready for review
June 21, 2026 10:00
Peer-exchange responds with ENRs read from the PeerStore ENRBook, which is fed add-only by the discv5 search loop and only pruned on capacity, never on staleness. discv5 revalidates its routing table and evicts unreachable nodes within seconds, but those evictions were not propagated to the peer store, so PX kept advertising ENRs of dead peers (only peers we personally failed to dial -> CannotConnect were skipped; disconnected peers are CanConnect and discovered-but-unconnected peers are NotConnected, both of which were still served). Reconcile the peer store against the live routing table at the end of each discv5 search iteration: drop Discv5-origin, non-connected peers that discv5 no longer knows about. Peers discovered in the same round are preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
igor-sirotin
force-pushed
the
fix/px-stale-enr-3933
branch
from
June 21, 2026 10:00
4f91380 to
e2f9efe
Compare
igor-sirotin
requested review from
Ivansete-status,
NagyZoltanPeter and
darshankabariya
June 21, 2026 10:01
Ivansete-status
left a comment
Ivansete-status
left a comment
Collaborator
There was a problem hiding this comment.
I think the responsibility should be only within peer_manager.nim. Discv5 should just discover and add peers whereas peer_manager.nim should decide when to delete them.
igor-sirotin
added a commit
to status-im/status-go
that referenced
this pull request
Jun 22, 2026
…un + fleet rate limits) (#7574) * fix(waku): set filter full-node rate limit to match the fleet (30/min) status-go enables the go-waku filter server on full nodes but only set WithMaxSubscribers, leaving the rate limiter at go-waku's default of 1 request/second, burst 1 — while the lightpush server right beside it got an explicit limiter. A light client joining a community fans out to many per-content-topic FilterSubscribeRequests at once, so a 1/s+burst-1 server rejects them with 429 ("filter request rejected due rate limit exceeded") for ~60s+, longer than the functional tests' 60s wait, so pushed community messages are dropped server-side ("no subscribed peers found") and the light receiver times out (flaky full-light/light-light compat cells). Set the filter full-node limiter to 30 requests/minute per peer (burst 30), matching the logos-delivery (nwaku) fleet default (FilterDefaultPerPeerRateLimit = (30, 1.minutes)). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(functional): serve community shards 128 & 256 in the waku fleet Old peers (v10.33.x) subscribe community channel filters on shards rs/16/128 and rs/16/256, but the test fleet only served shards 32 and 64. Those subscribes fail with 'no suitable peers found' and churn the light client's filter manager, collaterally destabilizing the shard-32 subscription that actually carries the community message -> the light receiver misses it and times out (flaky full-light / light-light compat cells against v10.33.2). Subscribe the fleet (boot-1 + store) to shards 128 and 256 as well, and bump --num-shards-in-network to 257 so those shards validate. This is a fleet-only change; the value is not propagated to the status-go backends (the fleet-config scanner only emits clusterId + node ENRs), so it does not affect how any backend derives shards. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(functional): use nwaku-pr:3939 fleet (PX stale-ENR prune fix) The light-client compat flakiness is driven by peer-exchange pollution: the shared test fleet's PX server hands each light client the ENRs of every other parallel cell's backends (incl. torn-down ones). The light client churns dialing those dead peers ('all dials failed') and its filter peer-selection intermittently can't find the connected fleet node for the community shard ('no suitable peers found') -> missed message -> timeout. Affects both v10.34.1 and v10.33.2 light cells, flaky. Point the fleet (boot-1 + store) at quay.io/wakuorg/nwaku-pr:3939, the build of logos-messaging/logos-delivery#3939 which prunes stale Discv5-origin peers from the peer store (reconcilePeerStoreWithRoutingTable in the discv5 search loop), so PX stops advertising dead sibling ENRs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(functional): fix fleet relay-mesh + filter rate limit for light-client compat Two fleet-side root causes of tests-rpc-compat light-client flakiness, both proven from CI fleet logs (compose.log), neither a status-go regression: 1. Relay-mesh fragmentation. nwaku runs AutoSharding, so each fleet node relay-subscribes to ALL `num-shards-in-network` shards. At 257 the 2-node gossipsub mesh is spread across 257 topics and rs/16/32 (the static community shard) frequently has zero mesh peers, so a light client's lightpush returns `not_published_to_any_peer` and the message never reaches the store. Evidence: num=65 -> 0 lightpush failures; num=257 -> 101. Revert to 65 and drop the --shard=128/256 subscriptions that forced it (status-go uses static 32/64; the observed failures were all on rs/16/32, never 128/256). 2. Filter rate limit. The fleet set no --rate-limit, so nwaku enforced its stock 30/min/peer filter limit. Under `pytest -n 12` + reruns many light clients subscribe concurrently and draw a 429, which go-waku turns into a permanent subscription backoff for the whole run (dropped pushes / stuck community joins). Lift filter+lightpush limits well above the test load. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(functional): point compat PEER_REFS at filter-rate-limit-fixed release branches TEMP experiment on #7574: the remaining compat flakiness is filter 429s from cross-test peer backends running released images that lack the filter full-node rate-limit fix (default 1/s burst 1). Build the peer images from the release branches WITH the fix cherry-picked in (tags compat-filterfix-10.34.x / compat-filterfix-10.33.x) to see whether that removes the 429-driven drops. Revert this override afterwards. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(functional): run compat tests serially + fleet filter rate limit Reduce the compat experiment to two fleet/test-harness changes (no status-go code change): - Run the compatibility marker serially (-n 0). They share one waku fleet, and under pytest-xdist the fleet's peer-exchange hands each light client the ENRs of other concurrent tests' torn-down backends, churning filter peer selection and intermittently dropping messages (#7513). - Lift the fleet nodes' stock filter/lightpush per-peer rate limits. Reverts the earlier status-go go-waku filter limiter and the PEER_REFS override. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(functional): bump compat test reruns to 6 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3933
Note (human-written)
Warning
I'm not sure if this is a good solution, look more like a workaround to me.
So please review carefully.
Description
Peer-exchange responds with ENRs read from the PeerStore
ENRBook, which is populated add-only by the discv5 search loop and only pruned on capacity, never on staleness. discv5 revalidates its own routing table and evicts unreachable nodes within seconds, but those evictions were never propagated to the peer store — so PX kept advertising ENRs of dead peers indefinitely (under capacity).The only liveness gate in
getEnrsFromStoreskipsCannotConnect, which is set only when we personally failed to dial a peer. A peer that merely disconnected isCanConnect, and a discv5-discovered-but-never-connected peer isNotConnected— both were still served by PX.Change
At the end of each discv5 search iteration, reconcile the peer store against the live (revalidated) routing table: drop
Discv5-origin, non-connected peers that discv5 no longer knows about. Peers discovered in the same round are preserved (they may not be in a bucket yet).reconcilePeerStoreWithRoutingTableinwaku/discovery/waku_discv5.nim, called fromsearchLoopafter adding discovered peers.This keeps PX responses tied to discv5's liveness without changing the PX protocol or its peer-store-backed read path.
Notes / open questions
ENRBook. Happy to take direction if maintainers prefer one of those.randomNodes(int.high)is used to enumerate the routing table each iteration (every ~5s); fine for typical table sizes, can be optimized if needed.nim check(compiles clean) andnph-formatted. Functional verification against the status-go light-client flake is in progress — see Flaky functional tests (light-client only): filter subscription churns on dead peer-exchange peers status-im/status-go#7513.🤖 Generated with Claude Code