eth_subscribe(newHeads) intermittently skips block headers (v0.46.0) #3415
Description
Description
When subscribing to eth_subscribe: "newHeads" over WebSocket, some intermediate canonical block headers are not delivered. The subscription jumps from block #12414308 to #12414313 without sending #12414309, #12414310, #12414311, or #12414312.
Environment
Version: 0.46.0-d7df89e7161
Launch cmd:
moonbeam --base-path=/_data/moonbeam/node --chain=moonbeam --sync=full --in-peers=128 --in-peers-light=64 --out-peers=64 --port=30333 --rpc-port=8545 --runtime-cache-size=64 --trie-cache-size=1073741824 --db-cache=25000 --disable-log-color --no-telemetry --prometheus-external --prometheus-port=9615 -- --port=30334 --sync=fast
Minimal reproduction
- Start the node with the flags above.
- Connect via WebSocket and subscribe to newHeads:
./websocat.x86_64-unknown-linux-musl ws://127.0.0.1:8545
{"id":1,"jsonrpc":"2.0","method":"eth_subscribe","params":["newHeads"]}
- Observe delivered headers.
Observed output (excerpt)
{"jsonrpc":"2.0","id":1,"result":"0xe44a086d5b0bb32ffc5c6805bb7b5b8a"}
{"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xe44a086d5b0bb32ffc5c6805bb7b5b8a","result":{
"number":"0xbd6d64", // 12414308
"hash":"0xe29d85b9bb94c18776f420ad21a33a0104c21e1c77989715f32a8431f1bff9b5",
...
}}}
{"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xe44a086d5b0bb32ffc5c6805bb7b5b8a","result":{
"number":"0xbd6d69", // 12414313
"hash":"0x1a092068d39c508baef655633864ec7626dfb1b66c008c2b33b98dc731bc07a7",
...
}}}
Node log around the same time
2025-09-01 11:44:24 [🌗] 🏆 Imported #12414307 (0xabaa…3fbe → 0x1ec1…2089)
2025-09-01 11:44:24 [🌗] 🏆 Imported #12414308 (0x1ec1…2089 → 0xce06…367f)
2025-09-01 11:44:29 [Relaychain] 💤 Idle (14 peers), best: #27578013 (0x5e1b…39b9), finalized #27578011 (0xb514…ac4e), ⬇ 299.3kiB/s ⬆ 588.1kiB/s
2025-09-01 11:44:29 [🌗] 💤 Idle (65 peers), best: #12414308 (0xce06…367f), finalized #12414303 (0xc7be…14d4), ⬇ 186.3kiB/s ⬆ 161.7kiB/s
2025-09-01 11:44:30 [🌗] 🏆 Imported #12414309 (0xce06…367f → 0xa02b…8697)
2025-09-01 11:44:30 [Relaychain] 🏆 Imported #27578014 (0x5e1b…39b9 → 0x5715…331e)
2025-09-01 11:44:34 [Relaychain] 💤 Idle (14 peers), best: #27578014 (0x5715…331e), finalized #27578011 (0xb514…ac4e), ⬇ 253.4kiB/s ⬆ 449.6kiB/s
2025-09-01 11:44:34 [🌗] 💤 Idle (65 peers), best: #12414309 (0xa02b…8697), finalized #12414303 (0xc7be…14d4), ⬇ 62.0kiB/s ⬆ 227.6kiB/s
2025-09-01 11:44:37 [Relaychain] 🏆 Imported #27578015 (0x5715…331e → 0xdaef…4264)
2025-09-01 11:44:39 [Relaychain] 💤 Idle (14 peers), best: #27578015 (0xdaef…4264), finalized #27578011 (0xb514…ac4e), ⬇ 328.5kiB/s ⬆ 276.5kiB/s
2025-09-01 11:44:39 [🌗] 💤 Idle (65 peers), best: #12414309 (0xa02b…8697), finalized #12414303 (0xc7be…14d4), ⬇ 3.4kiB/s ⬆ 4.8kiB/s
2025-09-01 11:44:42 [🌗] 🏆 Imported #12414310 (0xa02b…8697 → 0x142d…a06c)
2025-09-01 11:44:42 [🌗] 🆕 Imported #12414310 (0xa02b…8697 → 0xb19e…42ed)
2025-09-01 11:44:42 [Relaychain] 🏆 Imported #27578016 (0xdaef…4264 → 0xf3f5…d20e)
2025-09-01 11:44:42 [Relaychain] ♻️ Reorg on #27578016,0xf3f5…d20e to #27578016,0xee25…9522, common ancestor #27578015,0xdaef…4264
2025-09-01 11:44:42 [Relaychain] 🏆 Imported #27578016 (0xdaef…4264 → 0xee25…9522)
2025-09-01 11:44:42 [🌗] ♻️ Reorg on #12414310,0x142d…a06c to #12414311,0xa8ce…6154, common ancestor #12414309,0xa02b…8697
2025-09-01 11:44:42 [🌗] 🏆 Imported #12414311 (0xb19e…42ed → 0xa8ce…6154)
2025-09-01 11:44:44 [Relaychain] 💤 Idle (15 peers), best: #27578016 (0xee25…9522), finalized #27578012 (0x1f93…ebdf), ⬇ 307.1kiB/s ⬆ 512.3kiB/s
2025-09-01 11:44:44 [🌗] 💤 Idle (65 peers), best: #12414311 (0xa8ce…6154), finalized #12414304 (0x03f9…1e37), ⬇ 154.6kiB/s ⬆ 752.0kiB/s
2025-09-01 11:44:48 [Relaychain] 🏆 Imported #27578017 (0xee25…9522 → 0x6e07…49c8)
2025-09-01 11:44:48 [Relaychain] ♻️ Reorg on #27578017,0x6e07…49c8 to #27578017,0x748e…e448, common ancestor #27578016,0xee25…9522
2025-09-01 11:44:48 [Relaychain] 🏆 Imported #27578017 (0xee25…9522 → 0x748e…e448)
2025-09-01 11:44:48 [🌗] 🏆 Imported #12414312 (0xa8ce…6154 → 0xb433…8d11)
2025-09-01 11:44:48 [🌗] 🆕 Imported #12414312 (0xa8ce…6154 → 0x0cf8…f2b1)
2025-09-01 11:44:49 [Relaychain] 💤 Idle (15 peers), best: #27578017 (0x748e…e448), finalized #27578014 (0x5715…331e), ⬇ 655.6kiB/s ⬆ 507.0kiB/s
2025-09-01 11:44:49 [🌗] 💤 Idle (65 peers), best: #12414312 (0xb433…8d11), finalized #12414306 (0xabaa…3fbe), ⬇ 376.1kiB/s ⬆ 659.3kiB/s
2025-09-01 11:44:54 [Relaychain] 💤 Idle (15 peers), best: #27578017 (0x748e…e448), finalized #27578014 (0x5715…331e), ⬇ 871.3kiB/s ⬆ 387.5kiB/s
2025-09-01 11:44:54 [🌗] 💤 Idle (65 peers), best: #12414312 (0xb433…8d11), finalized #12414306 (0xabaa…3fbe), ⬇ 19.1kiB/s ⬆ 25.6kiB/s
2025-09-01 11:44:54 [🌗] ♻️ Reorg on #12414312,0xb433…8d11 to #12414313,0x5c87…d712, common ancestor #12414311,0xa8ce…6154
2025-09-01 11:44:54 [🌗] 🏆 Imported #12414313 (0x0cf8…f2b1 → 0x5c87…d712)
2025-09-01 11:44:54 [Relaychain] 🏆 Imported #27578018 (0x748e…e448 → 0x2224…bb9b)
2025-09-01 11:44:59 [Relaychain] 💤 Idle (14 peers), best: #27578018 (0x2224…bb9b), finalized #27578014 (0x5715…331e), ⬇ 468.9kiB/s ⬆ 166.9kiB/s
2025-09-01 11:44:59 [🌗] 💤 Idle (65 peers), best: #12414313 (0x5c87…d712), finalized #12414306 (0xabaa…3fbe), ⬇ 35.4kiB/s ⬆ 232.5kiB/s
2025-09-01 11:45:00 [Relaychain] 🏆 Imported #27578019 (0x2224…bb9b → 0x71ac…126d)
2025-09-01 11:45:00 [Relaychain] 🆕 Imported #27578019 (0x2224…bb9b → 0xd90a…58bc)
Actual result
The newHeads subscription delivered #12414308 and then #12414313, skipping notifications for #12414309, #12414310, #12414311, and #12414312.
Expected result
eth_subscribe(newHeads) should emit a notification for every new canonical head as it becomes best, even if some of those heads are later reorged out.
Metadata
Metadata
Assignees
Labels
No labels