archive node stalls at tip after shallow head reorg until reth restart

[aidazsolt-bot]

Describe the bug

On Ethereum mainnet, a reth archive node stopped advancing shortly after a shallow head reorg and then stayed frozen at the tip until both Lighthouse and reth were restarted.

Observed sequence:

1. A depth-1 head reorg happened at block 25562890 on 2026-07-18T23:31:49Z.
2. The archive node continued normally for ~192 seconds and then committed its last canonical block 25562906 at 2026-07-18T23:35:01Z.
3. After that, it received no further payloads, while peers stayed healthy (~87–101 peers).
4. reth began logging:

WARN Beacon client online, but no consensus updates received for a while. This may be because of a reth error, or an error in the beacon client! Please investigate reth and beacon client logs!

5. Lighthouse later started logging repeated Engine API timeouts against http://localhost:8551/:

CRIT Failed to update execution head
ExecutionForkChoiceUpdateFailed
HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out)

6. Restarting Lighthouse alone did not recover the node. Restarting reth recovered it and it resumed pipeline catch-up.

Important control case: a second mainnet node running in parallel as reth --minimal saw the same head reorg (reth_blockchain_tree_reorgs{commitment="head"} +1, latest_reorg_depth=1) but did not freeze and remained at lag=0.

This looks like an archive-node-specific CL↔EL / Engine API stall pattern after a shallow reorg. I am not claiming the reorg is proven root cause, only that it is the last verified chain event before the freeze.

Steps to reproduce

I do not have a deterministic reproducer yet, but this is the closest real-world sequence:

1. Run an Ethereum mainnet archive node with Lighthouse on the Engine API (localhost:8551).
2. In parallel, run a second control node on the same chain as reth --minimal (optional but useful for comparison).
3. Let the archive node follow the live tip under normal production load.
4. Observe a shallow head reorg (in this case, depth 1 at block 25562890).
5. Shortly after the reorg, the archive node may stop receiving new payloads and freeze at a canonical tip even though peer count remains healthy.
6. A few minutes later, reth starts emitting Beacon client online, but no consensus updates received..., and Lighthouse starts reporting Engine API timeouts to localhost:8551.
7. Restarting both Lighthouse and reth recovers the archive node; the parallel minimal node continues normally without freezing.

Sanitized runtime details:

• Archive node: standard reth node on mainnet, archive/default mode
• Parallel control node: reth node --minimal on mainnet
• Archive txpool tuning included:
  • --txpool.lifetime 10
  • --txpool.pending-max-count 33
  • --txpool.max-batch-size 50
• Both nodes exported Prometheus metrics on :6060
• Consensus client was Lighthouse via local Engine API on http://localhost:8551/

Node logs

Relevant sanitized excerpts from the incident:

# First canonical block at the reorg height
2026-07-18T23:31:39Z  INFO Received new payload ... number=25562890 hash=0x6639...
2026-07-18T23:31:39Z  INFO Block added to canonical chain number=25562890 hash=0x6639...
2026-07-18T23:31:45Z  INFO Canonical chain committed number=25562890 hash=0x6639...

# Competing payload at the same height
2026-07-18T23:31:49Z  INFO Received new payload ... number=25562890 hash=0xe14a...
2026-07-18T23:31:49Z  INFO Block added to fork chain number=25562890 hash=0xe14a...
2026-07-18T23:31:51Z  INFO Canonical chain committed number=25562890 hash=0xe14a...

# Last healthy tip
2026-07-18T23:35:01Z  INFO Block added to canonical chain number=25562906 hash=0xccfe...
2026-07-18T23:35:01Z  INFO Canonical chain committed number=25562906 hash=0xccfe...

# After that the tip stays frozen
2026-07-18T23:40:23Z  WARN Beacon client online, but no consensus updates received for a while. This may be because of a reth error, or an error in the beacon client! Please investigate reth and beacon client logs! period=322s
...
2026-07-19T01:51:06Z  CRIT Failed to update execution head
2026-07-19T01:51:06Z  CRIT ExecutionForkChoiceUpdateFailed
2026-07-19T01:51:06Z  CRIT HttpClient(url: http://localhost:8551/, kind: timeout, detail: operation timed out)

Additional verified observations:

• Archive tip froze at block 25562906 from 2026-07-18T23:35:01Z until restart.
• Peer count remained healthy during the freeze.
• The parallel minimal node observed the same depth-1 reorg but kept following the tip.
• Recovery required restarting reth; Lighthouse restart alone was insufficient.

I also have longer local journal/debug artifacts and Prometheus/Mimir evidence for:

• reth_blockchain_tree_reorgs{commitment="head"} +1
• reth_blockchain_tree_latest_reorg_depth = 1
• archive canonical tip plateau
• minimal node remaining healthy

Platform(s)

Linux (x86)

Container Type

LXC/LXD

What version/commit are you on?

Prometheus reth_info for both the archive node and the parallel minimal node reported:

version=2.3.0
git_sha=8b1198f6
build_profile=maxperf
build_timestamp=2026-06-11T20:39:54.466769077Z

The recovery restart log for the affected archive node also reported Start 2.3.0, so I am filing this against reth/v2.3.0-8b1198f6.

What database version are you on?

I did not capture the exact reth db version command output at incident time, but the affected archive node's restart log explicitly reports:

INFO reth::cli: Loaded storage settings settings=StorageSettings { storage_v2: true } pruning_mode="archive"

So this incident happened on a storage v2 archive node.

The same environment also logged RocksDB-backed healing/consistency checks during restart, e.g.:

INFO reth::providers::rocksdb: TransactionHashNumbers: healing via transaction ranges ...
INFO reth::providers::rocksdb: StoragesHistory: healing via changesets ...
INFO reth::providers::rocksdb: AccountsHistory: healing via changesets ...

If needed, I can follow up with an exact fresh reth db version capture from the same node.

Which chain / network are you on?

mainnet

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No custom prune config on the affected archive node.

The only known archive-specific runtime tuning relevant here was:

--txpool.lifetime 10
--txpool.pending-max-count 33
--txpool.max-batch-size 50

The parallel control node was reth --minimal.

If you've built Reth from source, provide the full command you used

Binary path was a local maxperf build of reth (/usr/src/Cargo/bin/reth), but I did not preserve the exact build invocation used for that binary.

Code of Conduct

• I agree to follow the Code of Conduct

[aidazsolt-bot]

A second occurrence happened on the same Ethereum archive node on reth v2.4.1 (git_sha=8eb21017, maxperf), on 2026-07-22. Mimir evidence:

• reth_blockchain_tree_reorgs{commitment="head"} increased 2 -> 3 at 14:47:05Z; latest_reorg_depth=1.
• The parallel minimal node observed the same depth-1 reorg (counter increment visible at 14:47:15Z due to scrape skew) and remained healthy.
• Both nodes were at canonical block 25588974 at 14:50:50Z.
• The archive node then remained frozen at 25588974 through 14:59:55Z (~9m05s), while the minimal node advanced to 25589019 (+45 blocks).
• Archive peers remained healthy during the plateau: 64-68 connected peers (65 at the last pre-restart sample).
• The archive exporter went down at 15:00:05Z and came back at 15:00:20Z, consistent with the operator restart. By 15:01:20Z the archive node had caught up to 25589018 and resumed following the chain. It is currently at lag=0.

This is the same observable sequence as the original report: shallow head reorg, archive continues briefly, archive tip stalls while peers and the minimal control remain healthy, restart restores progress. The temporal relationship is now reproducible across v2.3.0 and v2.4.1, but it still does not prove that the reorg causes the stall.