Skip to content

Node reports SyncSatus::NoSync even when it is still syncing #16004

Open
#16005
Open
Node reports SyncSatus::NoSync even when it is still syncing#16004
#16005
Assignees
pugachAG

Description

@kevindeforth
kevindeforth
opened on Jul 1, 2026

Background

The MPC team recently observed an issue with the nearcore indexer. While the view client had a correct view of the chain and could serve finalized contract state, the indexer streamer was way behind the tip of the chain, close to genesis (c.f. mpc-#3623).

The MPC node spins up the nearcore indexer with SyncModeEnum::LatestSynced and await_for_node_synced: StreamWhileSyncing in the following order:

  1. MPC node spawns the neard node
  2. MPC node waits for the near Indexer client to report status.sync_info.syncing == false
  3. MPC node starts the streamer.

https://github.com/near/mpc/blob/5b2e62bf5b2c6d7763d6e167ce5e7b713ed86e53/crates/node/src/indexer/real.rs#L127-L170

https://github.com/near/mpc/blob/5b2e62bf5b2c6d7763d6e167ce5e7b713ed86e53/crates/node/src/indexer.rs#L424-L449

The issue is that the client may report status.sync_info.syncing = false despite not having caught up with the tip of the chain. This then leads to the indexer stream attempting to stream every message from genesis.

nearcore/chain/indexer/src/streamer/mod.rs

Lines 259 to 403 in 3cae67a

pub async fn start(
view_client: IndexerViewClientFetcher,
client: IndexerClientFetcher,
shard_tracker: ShardTracker,
indexer_config: IndexerConfig,
store_config: near_store::StoreConfig,
blocks_sink: mpsc::Sender<StreamerMessage>,
clock: Clock,
) {
tracing::info!(target: INDEXER, "starting streamer");
let indexer_db_path =
near_store::NodeStorage::opener(&indexer_config.home_dir, &store_config, None, None)
.path()
.join("indexer");
let db = match DB::open_default(indexer_db_path) {
Ok(db) => db,
Err(err) => panic!("Unable to open indexer db: {:?}", err),
};
let mut last_synced_block_height: Option<BlockHeight> = None;
// Consecutive failed attempts to build a streamer message; reset on success.
// In `WaitForFullSync` mode it is also reset while the node is syncing (see
// `MAX_BUILD_STREAMER_MESSAGE_ATTEMPTS`).
let mut build_streamer_message_attempts: u32 = 0;
'main: loop {
clock.sleep(INTERVAL).await;
match indexer_config.await_for_node_synced {
AwaitForNodeSyncedEnum::WaitForFullSync => {
let status = client.fetch_status().await;
let Ok(status) = status else {
tracing::error!(target: INDEXER, ?status, "failed to fetch node status, retrying");
continue;
};
if status.sync_info.syncing {
tracing::debug!(target: INDEXER, ?status, "the node is syncing, waiting");
continue;
}
}
AwaitForNodeSyncedEnum::StreamWhileSyncing => {}
};
tracing::debug!(target: INDEXER, "starting streaming the next block range");
let block = view_client.fetch_latest_block(indexer_config.finality.clone()).await;
let Ok(block) = block else {
tracing::error!(target: INDEXER, ?block, "failed to fetch latest block, retrying");
continue;
};
let latest_block_height = block.header.height;
let start_syncing_block_height = get_start_syncing_block_height(
&db,
&indexer_config,
last_synced_block_height,
latest_block_height,
);
tracing::debug!(
target: INDEXER,
%start_syncing_block_height,
%latest_block_height,
"streaming is about to start",
);
metrics::START_BLOCK_HEIGHT.set(start_syncing_block_height as i64);
metrics::LATEST_BLOCK_HEIGHT.set(latest_block_height as i64);
for block_height in start_syncing_block_height..=latest_block_height {
metrics::CURRENT_BLOCK_HEIGHT.set(block_height as i64);
let block = match view_client.fetch_block_by_height(block_height).await {
Ok(Some(block)) => block,
Ok(None) => {
tracing::debug!(target: INDEXER, ?block_height, "skip height - missing block");
continue;
}
Err(err) => {
tracing::error!(target: INDEXER, ?block_height, ?err, "skip height - failed to fetch block");
continue;
}
};
let streamer_message =
Box::pin(build_streamer_message(&view_client, block, &shard_tracker)).await;
let streamer_message = match streamer_message {
Ok(streamer_message) => {
build_streamer_message_attempts = 0;
streamer_message
}
Err(err) => {
// When waiting for full sync, a build failure while the node
// is not yet ready is expected (e.g. epoch data not available
// after a restart, see #15867): retry the same height forever
// without counting it against the budget. When streaming while
// syncing the node is ~always "syncing", so that gate would
// make the budget unreachable; there we count every failure so
// a genuinely stuck height eventually surfaces.
let transient_while_syncing = matches!(
indexer_config.await_for_node_synced,
AwaitForNodeSyncedEnum::WaitForFullSync
) && !node_is_ready(&client).await;
if transient_while_syncing {
build_streamer_message_attempts = 0;
tracing::warn!(target: INDEXER, ?block_height, ?err, "failed to build streamer message while the node is syncing, retrying the same height");
} else {
build_streamer_message_attempts += 1;
tracing::error!(target: INDEXER, ?block_height, ?err, attempts = build_streamer_message_attempts, "failed to build streamer message, retrying the same height");
assert!(
build_streamer_message_attempts < MAX_BUILD_STREAMER_MESSAGE_ATTEMPTS,
"failed to build streamer message at height {block_height} after {MAX_BUILD_STREAMER_MESSAGE_ATTEMPTS} attempts: {err:?}"
);
}
// Retry the same height on the next outer iteration instead of
// advancing `last_synced_block_height`.
break;
}
};
tracing::debug!(target: INDEXER, ?block_height, "sending streamer message to the listener");
let send_result = blocks_sink.send(streamer_message).await;
if send_result.is_err() {
tracing::error!(
target: INDEXER,
?block_height,
?send_result,
"unable to send streamer message to listener, listener doesn't listen, terminating",
);
break 'main;
};
metrics::NUM_STREAMER_MESSAGES_SENT.inc();
db.put(b"last_synced_block_height", &block_height.to_string()).unwrap();
last_synced_block_height = Some(block_height);
}
}
}
fn get_start_syncing_block_height(
db: &rocksdb::DB,
indexer_config: &IndexerConfig,
last_synced_block_height: Option<u64>,
latest_block_height: u64,
) -> u64 {
// If last synced is set, start from the next height
if let Some(last_synced_block_height) = last_synced_block_height {
return last_synced_block_height + 1;

For the MPC node, this has the following effects:

  • After syncing up with the head of the chain, the view client serves the finalized view of the MPC contract;
  • The indexer is still attempting to stream blocks from way behind, walking through every block starting from genesis. This means that the indexer is not serving any of the recent transactions, resulting in the MPC node missing signature requests on-chain.

Why the indexer reports status.sync_info.syncing = false after startup

status.sync_info.syncing = false gets set in

nearcore/chain/client/src/client_actor.rs

Line 905 in 3cae67a

syncing: self.client.sync_handler.sync_status.is_syncing(),

SyncStatus::is_syncing(&self) returns false if self == SyncStatus::NoSync:

nearcore/chain/client-primitives/src/types.rs

Lines 168 to 173 in 3cae67a

pub fn is_syncing(&self) -> bool {
match self {
SyncStatus::NoSync => false,
_ => true,
}
}

sync_status is updated in ClientActor::run_sync_step and is set to NoSync when the client has no eligible peers:

nearcore/chain/client/src/client_actor.rs

Lines 1817 to 1844 in 3cae67a

match sync {
SyncRequirement::AlreadyCaughtUp { .. }
| SyncRequirement::NoPeers
| SyncRequirement::AdvHeaderSyncDisabled => {
if currently_syncing {
// Initial transition out of "syncing" state.
tracing::debug!(target: "sync", prev_sync_status = ?self.client.sync_handler.sync_status, "disabling sync");
self.client.sync_handler.sync_status.update(SyncStatus::NoSync);
// Announce this client's account id if their epoch is coming up.
let head = match self.client.chain.head() {
Ok(v) => v,
Err(err) => {
tracing::error!(target: "sync", ?err, "sync: unexpected error");
return;
}
};
self.check_send_announce_account(head.prev_block_hash);
}
}
SyncRequirement::SyncNeeded { highest_height, .. } => {
if !currently_syncing {
tracing::info!(target: "client", ?sync, "enabling sync");
}
self.handle_sync_needed(highest_height);
}
}

The client reports SyncRequirement::NoPeers when it has no eligible peers (i.e. no peer whose block height is known):

nearcore/chain/client/src/client_actor.rs

Line 1695 in 3cae67a

return Ok(SyncRequirement::NoPeers);

It seems like this is possible in two scenarios:

  1. This is a validator node running in a one-node network. This is intended behavior — such a node has no peers to sync from and must declare itself synced in order to produce blocks.
  2. This is a node running in a network with other nodes that has connected to enough peers, but none of them has advertised a block height yet. A peer's height (ConnectedPeerState.block_info) is None at handshake and only set once that peer sends its first Block message, so highest_height_peers is empty despite the connected peers — causing syncing_info() to return NoPeers:

Scenario 2 is the one that affects the MPC node: it briefly reports syncing = false while still at genesis, the streamer starts, and its LatestSynced cursor pins at genesis.

Proposed Fix

The MPC node does not run a validator; it is a follower that always has to sync from peers. A follower that has not yet confirmed any peer's height has no way of knowing it is caught up, so it should keep reporting syncing = true rather than declaring itself synced.

Concretely: SyncRequirement::NoPeers should only result in NoSync when the node is a validator (which may legitimately bootstrap a fresh or one-node network). For a non-validator node, NoPeers should instead keep the node in AwaitingPeers (syncing = true) until it has confirmed a peer's height. This preserves the intended single-node-validator behavior (scenario 1) while fixing scenario 2.

edit:
So, this proposed fix isn't super robust. It only resolves the issue for non-validator nodes. A better approach might be to have a boolean to indicate if the node is supposed to be the only validator in the network.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions