[Bug] testnet aptos v1.48.1 halt because of error shown below

[bomaxchenbtse]

🐛 Bug

suddenly can't keep syncing and error happens shown as below

To reproduce

Stack trace/error message

ul 06 07:18:30 apt01-stg aptos-node[354676]: 2026-07-06T07:18:30.573167Z [network-Pub-5] WARN network/framework/src/protocols/rpc/mod.rs:670 [sampled] [full_node,Public,f3d37d80] Error making outbound RPC request to 0058220d (request_id 71). Error: Error: Rpc timed out
Jul 06 07:18:30 apt01-stg aptos-node[354676]: Stack backtrace:
Jul 06 07:18:30 apt01-stg aptos-node[354676]:    0: anyhow::kind::Adhoc::new
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14
Jul 06 07:18:30 apt01-stg aptos-node[354676]:    1: aptos_network::protocols::rpc::OutboundRpcs::handle_outbound_request::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /opt/aptos-core/network/framework/src/protocols/rpc/mod.rs:541:57
Jul 06 07:18:30 apt01-stg aptos-node[354676]:    2: aptos_network::protocols::rpc::OutboundRpcs::handle_outbound_request::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /opt/aptos-core/network/framework/src/protocols/rpc/mod.rs:554:38
Jul 06 07:18:30 apt01-stg aptos-node[354676]:    3: <core::pin::Pin<P> as core::future::future::Future>::poll
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/core/src/future/future.rs:133:9
Jul 06 07:18:30 apt01-stg aptos-node[354676]:    4: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/git/checkouts/futures-rs-bce5b78f2276d961/1845d7d/futures-util/src/stream/futures_unordered/mod.rs:518:24
Jul 06 07:18:30 apt01-stg aptos-node[354676]:    5: futures_util::stream::stream::StreamExt::poll_next_unpin
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/git/checkouts/futures-rs-bce5b78f2276d961/1845d7d/futures-util/src/stream/stream/mod.rs:1638:24
Jul 06 07:18:30 apt01-stg aptos-node[354676]:    6: <futures_util::stream::stream::select_next_some::SelectNextSome<St> as core::future::future::Future>::poll
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/git/checkouts/futures-rs-bce5b78f2276d961/1845d7d/futures-util/src/stream/stream/select_next_some.rs:34:48
Jul 06 07:18:30 apt01-stg aptos-node[354676]:    7: <core::pin::Pin<P> as core::future::future::Future>::poll
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/core/src/future/future.rs:133:9
Jul 06 07:18:30 apt01-stg aptos-node[354676]:    8: futures_util::future::future::FutureExt::poll_unpin
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/git/checkouts/futures-rs-bce5b78f2276d961/1845d7d/futures-util/src/future/future/mod.rs:558:24
Jul 06 07:18:30 apt01-stg aptos-node[354676]:    9: aptos_network::peer::Peer<TSocket>::start::{{closure}}::{{closure}}::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/git/checkouts/futures-rs-bce5b78f2276d961/1845d7d/futures-util/src/async_await/select_mod.rs:321:13
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   10: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/core/src/ops/function.rs:298:21
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   11: aptos_network::peer::Peer<TSocket>::start::{{closure}}::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/git/checkouts/futures-rs-bce5b78f2276d961/1845d7d/futures-util/src/async_await/select_mod.rs:321:13
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   12: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/git/checkouts/futures-rs-bce5b78f2276d961/1845d7d/futures-util/src/future/poll_fn.rs:56:9
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   13: aptos_network::peer::Peer<TSocket>::start::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /opt/aptos-core/network/framework/src/peer/mod.rs:260:13
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   14: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/core.rs:375:24
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   15: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/loom/std/unsafe_cell.rs:16:9
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   16: tokio::runtime::task::core::Core<T,S>::poll
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/core.rs:364:30
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   17: tokio::runtime::task::harness::poll_future::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/harness.rs:535:30
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   18: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/core/src/panic/unwind_safe.rs:274:9
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   19: std::panicking::catch_unwind::do_call
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/std/src/panicking.rs:581:40
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   20: std::panicking::catch_unwind
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/std/src/panicking.rs:544:19
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   21: std::panic::catch_unwind
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/std/src/panic.rs:359:14
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   22: tokio::runtime::task::harness::poll_future
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/harness.rs:523:18
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   23: tokio::runtime::task::harness::Harness<T,S>::poll_inner
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/harness.rs:210:27
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   24: tokio::runtime::task::harness::Harness<T,S>::poll
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/harness.rs:155:20
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   25: tokio::runtime::task::raw::RawTask::poll
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/raw.rs:267:18
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   26: tokio::runtime::task::LocalNotified<S>::run
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/mod.rs:510:13
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   27: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/multi_thread/worker.rs:647:18
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   28: tokio::task::coop::with_budget
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/task/coop/mod.rs:167:5
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   29: tokio::task::coop::budget
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/task/coop/mod.rs:133:5
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   30: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/multi_thread/worker.rs:638:9
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   31: tokio::runtime::scheduler::multi_thread::worker::Context::run
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   32: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/multi_thread/worker.rs:536:24
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   33: tokio::runtime::context::scoped::Scoped<T>::set
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/context/scoped.rs:40:9
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   34: tokio::runtime::context::set_scheduler::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/context.rs:176:38
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   35: std::thread::local::LocalKey<T>::try_with
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/std/src/thread/local.rs:513:12
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   36: std::thread::local::LocalKey<T>::with
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/std/src/thread/local.rs:477:20
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   37: tokio::runtime::context::set_scheduler
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/context.rs:176:17
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   38: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/multi_thread/worker.rs:531:9
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   39: tokio::runtime::context::runtime::enter_runtime
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/context/runtime.rs:65:16
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   40: tokio::runtime::scheduler::multi_thread::worker::run
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/multi_thread/worker.rs:523:5
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   41: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/scheduler/multi_thread/worker.rs:489:45
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   42: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/blocking/task.rs:42:21
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   43: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/core.rs:375:24
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   44: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/loom/std/unsafe_cell.rs:16:9
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   45: tokio::runtime::task::core::Core<T,S>::poll
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/core.rs:364:30
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   46: tokio::runtime::task::harness::poll_future::{{closure}}
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/harness.rs:535:30
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   47: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/core/src/panic/unwind_safe.rs:274:9
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   48: std::panicking::catch_unwind::do_call
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/std/src/panicking.rs:581:40
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   49: std::panicking::catch_unwind
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/std/src/panicking.rs:544:19
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   50: std::panic::catch_unwind
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/std/src/panic.rs:359:14
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   51: tokio::runtime::task::harness::poll_future
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/harness.rs:523:18
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   52: tokio::runtime::task::harness::Harness<T,S>::poll_inner
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/harness.rs:210:27
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   53: tokio::runtime::task::harness::Harness<T,S>::poll
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/harness.rs:155:20
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   54: tokio::runtime::task::raw::RawTask::poll
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/raw.rs:267:18
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   55: tokio::runtime::task::UnownedTask<S>::run
Jul 06 07:18:30 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/task/mod.rs:547:13
Jul 06 07:18:30 apt01-stg aptos-node[354676]:   56: tokio::runtime::blocking::pool::Task::run
Jul 06 07:18:31 apt01-stg aptos-node[354676]:              at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.50.0/src/runtime/blocking/pool.rs:161:1(truncated) {"network_context":{"role":"full_node","network_id":"Public","peer_id":"f3d37d80700bdfba148ded81eefa9ae7efdab05ce870f48ab158008262c19648"},"remote_peer":"0x0058220de6ba1af4c4a803e8727d9ed372104d2e60057cb16d6a99e646512826"}

Expected Behavior

A clear and concise description of what you expected to happen.

System information

Please complete the following information:

• Aptos Core Version v1.48.1
• Rust Version : rustup 1.29.0 (28d1352db 2026-03-05)
• Computer OS : ubuntu24

Additional context