Peer drop still happening even after building image from: https://github.com/bnb-chain/reth-bsc/pull/297

[ashu1777]

Description:
We tried building a custom image on top of tag v0.0.8-beta which includes PR. And ran it for a days on one of our node.
And today we got the same peer drop issue:

026-03-27T08:26:27.771409Z  INFO Fork choice decision made by fast finality need_reorg=true incoming_number=89002486 incoming_hash=0xb81f93fc0ea023bb588259e8a4959abe8a4cf87623575f3f9ced85a900d8e939 current_number=89002485 current_hash=0x7e8fb0bbccc43cfb10207cd92acd2200abf1bc2c2645070e44c464f58284918c method="fast_finality"
2026-03-27T08:26:27.771778Z  INFO Canonical chain committed number=89002486 hash=0xb81f93fc0ea023bb588259e8a4959abe8a4cf87623575f3f9ced85a900d8e939 elapsed=187.631µs
2026-03-27T08:26:27.771792Z  INFO Forkchoice updated head_block_hash=0xb81f93fc0ea023bb588259e8a4959abe8a4cf87623575f3f9ced85a900d8e939 safe_block_hash=0x7e8fb0bbccc43cfb10207cd92acd2200abf1bc2c2645070e44c464f58284918c finalized_block_hash=0x782c719e5112ef48f643786380fc71cc14f1d69a474e3933a2ee959fe179650a
2026-03-27T08:26:41.072519Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:27:56.071741Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:29:11.071982Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:30:26.072612Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:31:41.071757Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:32:56.071745Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:34:11.072666Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:35:26.426294Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:36:41.426052Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:37:56.425773Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:39:11.425834Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:40:26.425665Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:41:41.425940Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:42:56.426065Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:44:11.425849Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:45:26.426138Z  INFO Status connected_peers=1 latest_block=89002486
2026-03-27T08:46:04.509366Z  INFO Wrote network peers to file peers_file="/var/lib/node/known-peers.json"

Seems like this is fix: #297, is not solving peer drop issue.

[ashu1777]

Tried building an image from the develop branch (upcoming release), but the issue still persists. Peers gradually drop to 1 or even 0, and the node falls out of sync with the chain tip.

[constwz]

#333
I will attempt to solve the problem of reduced peers in this PR. Please give me some time.

[0x19dG87]

I am constantly seeing this issue on my node as well. There is nothing strange in logs, just peer drop and node start lagging.

[0x19dG87]

This time it is not even recovered after peers drop. Running 0.0.9 version.

2026-04-30T05:36:34.358758Z  INFO Status connected_peers=0 latest_block=95456889
2026-04-30T05:37:54.662894Z  INFO Status connected_peers=0 latest_block=95456889
2026-04-30T05:39:09.664019Z  INFO Status connected_peers=0 latest_block=95456889
2026-04-30T05:40:32.266011Z  INFO Status connected_peers=0 latest_block=95456889
2026-04-30T05:40:32.266086Z  WARN Beacon client online, but no consensus updates received for a while. This may be because of a reth error, or an error in the beacon client! Please investigate reth and beacon client logs! period=19304.675626163s

[hub-il]

This time it is not even recovered after peers drop. Running 0.0.9 version.

2026-04-30T05:36:34.358758Z  INFO Status connected_peers=0 latest_block=95456889
2026-04-30T05:37:54.662894Z  INFO Status connected_peers=0 latest_block=95456889
2026-04-30T05:39:09.664019Z  INFO Status connected_peers=0 latest_block=95456889
2026-04-30T05:40:32.266011Z  INFO Status connected_peers=0 latest_block=95456889
2026-04-30T05:40:32.266086Z  WARN Beacon client online, but no consensus updates received for a while. This may be because of a reth error, or an error in the beacon client! Please investigate reth and beacon client logs! period=19304.675626163s

I see the same thing on our node updated to 0.0.9.

[freakz867]

Same on latest version, in one day gradually drops from starting 25-30 to 3-4, and then in next day to 1. Only a restart helps, but everything starts all over again, within a day it drops to 3-4 and, accordingly, delays in blocks and other things appear. Exactly same situation was on 0.0.8 with different hardware. Default startup config like:
reth-bsc node --chain=bsc --datadir=/data/reth-bsc/data_dir --authrpc.addr "0.0.0.0" --authrpc.port 8551 --engine.parallel-sparse-trie --http --http.addr "127.0.0.1" --http.port 8545 --ws --ws.addr "127.0.0.1" --http.api "admin, debug, eth, net, trace, txpool, web3, rpc, reth, ots" --db.max-size=12TB --db.page-size=8KB

If there is anything that can be done to diagnose it with logs or etc, please tell me... There is currently no alternative to this node; as for me Geth takes a very long time to perform debug_trace calls, this is ideal but having to reboot the node to restore peers and losing 5-10 minutes of online node every day is sad

[will-2012]

hi @freakz867
#344. This is a temporary version. You can compile the code using this branch which contains more diagnostic information, and add env RUST_LOG="debug,net::session=trace,net::peers=trace,reth_network::peers=trace,reth_network::state=trace" to your environment variables.

If you can reproduce the above case, please upload the more detailed logs for that period. Thanks! This will be very helpful in troubleshooting; by the way, the team is also trying to reproduce this issue internally.

Any feedback you provide is appreciated, and we will keep you updated on any progress. Thanks

[freakz867]

Hi @will-2012, Here are the logs according to your instructions. They are 19 GB uncompressed, 610MB xz compressed. If you need a smaller or more specific portion instead, please let me know exactly what you need.

The node was running for about 13–14 hours. I also attached the peers graph for that logs timeframe. This time the behavior was not the usual perfectly linear decline for some reason, but the peer count was still decreasing over time anyway. Startup config exactly as i wrote above + metrics and --max-peers=100

I have restarted the node again now, and I will probably provide another set of logs later if the typical clear linear peer decline appears more clear. Also small note that I also replaced my real IP address with a different one in the logs. That should not be a problem; I just did it so you would not think the issue is related to my actual IP/nat/etc. Uploaded it to google drive https://drive.google.com/file/d/1KYDAk_F5y3cKjq_-0JiWrd7dpstaqVNX/view?usp=sharing , sorry maybe there are better option but I don't know about it...

[freakz867]

@will-2012 Here is the second run. This time the behavior is much more illustrative and significantly faster than usual (4.5 hours to 2 peers from 20+). The logs are 6.2 GB uncompressed and 206 MB compressed. I am also attaching the peers graph.
https://drive.google.com/file/d/1llI5mQhw7AsuvyeExpAmXp9qmVPico2n/view?usp=sharing

[will-2012]

hi @freakz867 thank your feedback~

I note that

grep "Reth-BSC" reth_mod_2.log
2026-05-07T17:24:02.763636Z  INFO reth::cli: Starting Reth-BSC version="0.1.0 (52185aa)"

Your binary is compiled from a commit made two days ago.

Could you please try this latest commit: e7fa1f1?
By the way, the team is also trying to reproduce and fix this issue internally. We'll let you know if there's any concrete progress.

[freakz867]

@will-2012 I built the node from the specified commits completely. The first run behaved very badly, the second one was slightly better, but the peers are still dropping anyway so I decided to stop and grab logs too and start new run. I am attaching both logs and both graphs.
1st run: https://drive.google.com/file/d/14rsSXoSbAVRINbW-hyQk-ZjXucU4K2cA/view?usp=sharing

2nd run: https://drive.google.com/file/d/1M_gKHVuYb-O8xOcjtBJbIvnvpjQ-DJw4/view?usp=sharing

[freakz867]

This debug-p2p branch seems to be very unstable overall. In practice, I have to restart the node about every 4 hours because of the peer drop and the related issues. I have collected more logs as well - let me know if you need them. Likewise, if any additional changes or adjustments are needed, I will gladly make them. On the graph above - 3 restarts for last 12 hours, can upload logs for that if you need them.

[will-2012]

This debug-p2p branch seems to be very unstable overall. In practice, I have to restart the node about every 4 hours because of the peer drop and the related issues. I have collected more logs as well - let me know if you need them. Likewise, if any additional changes or adjustments are needed, I will gladly make them. On the graph above - 3 restarts for last 12 hours, can upload logs for that if you need them.

Got it, thanks.