fix: indexer wedging on startup (#3713)

[gilcu3]

cherry-pick of afb2c9b

[claude]

Pull request overview

Cherry-pick of #3713 into release/v3.13. Replaces the strict wait_for_full_sync gate in the streamer boot path with a SyncProgress state machine that requires REQUIRED_STABLE_POLLS = 4 consecutive non-syncing polls and strict head advancement before returning. This is the release-branch backport of the workaround for nearcore#16004, where neard transiently reports syncing == false at genesis on a fresh state-syncing node and pins the streamer's LatestSynced cursor to a stale head.

Changes:

• Introduces SyncProgress + REQUIRED_STABLE_POLLS in crates/node/src/indexer.rs.
• Adds ensure_head_follows_tip and factors sync_info out of wait_for_full_sync.
• real.rs streamer now gates on ensure_head_follows_tip; other callers (tee.rs, participants.rs, migrations.rs) still use wait_for_full_sync.
• 6 unit tests covering syncing / static-head / not-enough-polls / caught-up / late-advance / mid-run resync.

Reviewed changes

Per-file summary

File	Description
crates/node/src/indexer.rs	Adds SyncProgress, REQUIRED_STABLE_POLLS, sync_info, and ensure_head_follows_tip; keeps wait_for_full_sync for other consumers. Includes 6 unit tests.
crates/node/src/indexer/real.rs	Streamer boot now gates on ensure_head_follows_tip instead of wait_for_full_sync; comment trimmed to the essential rationale.

Findings

Blocking (must fix before merge): none. Code is a byte-clean cherry-pick of the already-approved #3713; unit tests exercise the predicate exhaustively; scoped to release/v3.13 as a hot-fix per @kevindeforth's note that this should be reverted once nearcore#16004 lands.

Non-blocking (nits, follow-ups, suggestions):

• crates/node/src/indexer.rs:495 — self.run_polls += 1 uses raw u32 arithmetic in production code. docs/engineering-standards.md §Use safe arithmetic methods asks for checked_add/saturating_add on primitives. Practically harmless (overflow only after ~68 years of 500 ms polls with a static head), but strictly the wedge case is the one where head stays static and polls accumulate — saturating_add(1) would be a one-line fix that preserves the semantics (the predicate saturates at REQUIRED_STABLE_POLLS anyway).
• crates/node/src/indexer.rs:451 and crates/node/src/indexer/tee.rs:64,138,188, crates/node/src/indexer/participants.rs:282, crates/node/src/indexer/migrations.rs:80 — the old wait_for_full_sync (single-poll flag check) remains the gate for TEE monitoring, participant-set reads, and migration reads. Those callers are still exposed to the same transient syncing == false at genesis that triggered the wedge (they'd just observe empty/stale contract state and, unlike the streamer, re-poll on their own cadence). Scoping to the streamer is defensible for a release cherry-pick, but worth a follow-up issue so the participant/migration/TEE readers migrate to ensure_head_follows_tip (or a shared predicate) once this pattern is validated in prod.
• crates/node/src/indexer.rs:463 — the loop calls tokio::time::sleep(INTERVAL).await before the first status probe, so first-boot always waits ~500 ms and the minimum "caught up" latency is ~2 s (REQUIRED_STABLE_POLLS × INTERVAL). Fine for a startup path, just noting since the fix is specifically about boot behavior.
• crates/node/src/indexer.rs:496 — head_height > start_head is a weak signal: a single-block advance across the whole 4-poll window (~2 s) satisfies it. Safe against the specific wedge (state-sync keeps head strictly static at genesis) but not defensive against a hypothetical neard that ticks a single block while otherwise wedged. A stricter head_height >= start_head + (REQUIRED_STABLE_POLLS as u64 - 1) would demand roughly one block per poll. Tradeoff, not a blocker.
• crates/node/src/indexer.rs:424,428-429,473-474 — the surviving doc comments describe what the code does (a rule of thumb engineering-standards.md §Write helpful code comments discourages) rather than the why (the neard bug and the specific wedge symptom). Consider trimming to a one-line pointer such as // Workaround for near/nearcore#16004; see #3623. on SyncProgress. Same nit was raised on fix: indexer wedging on startup #3713 by @kevindeforth.
• No E2E regression for the wedge itself. SyncProgress is exhaustively unit-tested but the wedge is a system-level failure — a regression here reverts to indefinite starvation. Worth a follow-up e2e test that forces a neard into the "transient syncing == false at genesis" state and asserts the streamer doesn't lock to genesis.

⚠️ Issues found — all non-blocking. Given this is a cherry-pick of already-approved code onto a release branch for a production hot-fix, none of the above should block the merge; they are follow-ups.