Receipt Root Mismatch Causing Chain Halt

[siladu]

We have had at least three occurrences of receipt root mismatch where restarting the clients fixes it.

Couple of recent examples from prd-elc-bu-ls-mainnet-canary

{"@timestamp":"2025-12-31T21:28:38,338","level":"WARN","thread":"vert.x-worker-thread-0","class":"MainnetBlockBodyValidator","message":"Invalid block 24135298 (0xb025a54e8499958a108475683f2f3ef2087d1b01280c5b9214a2182846836052): receipts root mismatch (expected=0x1c24b0b4b570877290421d29ce5cdc346fbfce03c55fc9a94c337410c3d83773, actual=0xb79b88ae4d726e2d05f2602666261536eeea44e6118764f5ca9badbd8456d615)","throwable":""}

{"@timestamp":"2025-12-17T06:47:13,996","level":"WARN","thread":"vert.x-worker-thread-0","class":"MainnetBlockBodyValidator","message":"Invalid block 24030636 (0xad6177d169efb88b82b67449f5e4e71486b19697e8ba574c76c5ae135a2061f5): receipts root mismatch (expected=0xdf7b20bb9686913928d1fa32de8f50666860e35587df1b8d50e737572a0c68b9, actual=0x14c076f349f763db702869013094c0b42b5ecfc08a2c8f04a312a9af36da8e6d)","throwable":""}
{"@timestamp":"2025-12-17T06:47:13,996","level":"INFO","thread":"vert.x-worker-thread-0","class":"MainnetBlockValidator","message":"Invalid block 24030636 (0xad6177d169efb88b82b67449f5e4e71486b19697e8ba574c76c5ae135a2061f5): Optional[failed to validate output of imported block]","throwable":""}
{"@timestamp":"2025-12-17T06:47:14,000","level":"WARN","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Invalid new payload: number: 24030636, hash: 0xad6177d169efb88b82b67449f5e4e71486b19697e8ba574c76c5ae135a2061f5, parentHash: 0x684849f75973280feb2d2bc00063d113e6934b9d0b372c469f44a95c4dc50d24, latestValidHash: 0x684849f75973280feb2d2bc00063d113e6934b9d0b372c469f44a95c4dc50d24, status: INVALID, validationError: failed to validate output of imported block","throwable":""}

and one from a teku user:

2025-12-19 02:00:38.005+0000 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #24,043,527  (dd54d.....91269)|  258 tx ( 58.9% parallel)| 16 ws|  9 blobs|  27.13 mwei bfee|  34,097,042 ( 56.8%) gas used| 404.6ms exec|  84.28 Mgas/s| 25 peers
2025-12-19 02:00:38.179+0000 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | FCU(VALID) | head: dd54d.....91269 | safe: 2dfd9.....8afa9 | finalized: f376f.....c7a3c
2025-12-19 02:00:49.765+0000 | vert.x-worker-thread-0 | WARN  | MainnetBlockBodyValidator | Invalid block 24043528 (0xe08987ddbc0a7e7b738e7f35c65b2803daf8d29b94724d0253cf549cd1cc7a93): receipts root mismatch (expected=0x1970d398e88bb11b822402d5766b710d43ec6163d046c840194a74b845c1b559, actual=0x3a97cc74b2835e42acfa2e2fddba48cbc04f34419316163d378c6be9a10e34c6)
2025-12-19 02:00:49.765+0000 | vert.x-worker-thread-0 | INFO  | MainnetBlockValidator | Invalid block 24043528 (0xe08987ddbc0a7e7b738e7f35c65b2803daf8d29b94724d0253cf549cd1cc7a93): Optional[failed to validate output of imported block]
2025-12-19 02:00:49.765+0000 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 24043528, hash: 0xe08987ddbc0a7e7b738e7f35c65b2803daf8d29b94724d0253cf549cd1cc7a93, parentHash: 0xdd54d5fb2221aa4af814631e5fedcfbeedc38117bd74d76ffec3820e7eb91269, latestValidHash: 0xdd54d5fb2221aa4af814631e5fedcfbeedc38117bd74d76ffec3820e7eb91269, status: INVALID, validationError: failed to validate output of imported block
2025-12-19 02:01:15.134+0000 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | FCU(VALID) | head: dd54d.....91269 | safe: 2dfd9.....8afa9 | finalized: f376f.....c7a3c
2025-12-19 02:01:51.115+0000 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | FCU(VALID) | head: dd54d.....91269 | safe: 2dfd9.....8afa9 | finalized: f376f.....c7a3c

---

Full days logs for the most recent prd-elc-bu-ls-mainnet-canary issue:
prd-elc-bu-ls-mainnet-canary_invalid-block-24135298_besu.log
prd-elc-bu-ls-mainnet-canary_invalid-block-24135298_lodestar-beacon.log

[siladu]

One solution to the chain halt we discussed was detecting a scenario like this and automatically removing the block from the bad blocks cache to allow the CL to retry (or just retrying only once might work).

In order to maintain bad block reporting, we might need to split out the current cache into "reportable bad blocks" and "operational bad blocks" and this fix would only work with the operational cache, whilst allowing us to still report via the debug_getBadBlocks API.

[siladu]

The solution discussed above is actually a redo of an older issue #5460 :)