fix(debug-trace-server): stop rotating to fallbacks for not-yet-generated frontier witnesses

[flyq]

Summary

At the sync frontier the chain-sync prefetch treated the generator's "witness not found" as a provider failure and rotated to the public fallback. But a frontier witness the generator doesn't have yet cannot exist on the fallbacks either — they receive witnesses from the same generation pipeline. The fetcher now gives the generator a short exclusive grace for frontier-fresh blocks (retrying it alone with round backoff) and only then falls back to the full endpoint chain; deep catch-up blocks keep today's full failover from the first attempt.

Motivation

A 7.7 h production log (v2.0.14, 2026-07-21) shows the race is the norm, not the exception: 18,766 sync fetches hit "witness not yet generated" at the generator (~68% of blocks) and 18,499 of them rotated to the public fallback, which also missed (~600 ms per wasted round trip) before round 1 retried the generator successfully. Twice in that window the fallback did not answer its miss at all and hung until the 20 s per-attempt cap (blocks 21832224 / 21841054), stalling tip advance ~21 s each — with the witness almost certainly ready seconds in. The fallback won the frontier race in only 267 of 18,766 cases (1.4%). Routing fresh misses back to the generator removes both the daily waste and the routine exposure to fallback stalls; the ~21 s worst case becomes ~1–2 s (generation lag) with the fallback untouched.

Fix

• RpcClient::get_witness_light_first_provider_only (crates/stateless-common/src/rpc_client.rs) — the complement of the existing _from(skip) historical routing; the witness retry loop's provider selection generalizes from a skip prefix to an index range.
• TraceFetcher memoizes the last remote head observed by latest_block_number (which the pipeline already polls) as the freshness anchor, and fetch_witness routes (bin/debug-trace-server/src/chain_sync.rs): blocks within FRONTIER_FRESHNESS_BLOCKS (256) of that head give the generator a GENERATOR_WITNESS_GRACE (6 s) exclusive window, then fall through to the full chain; an unknown head or deeper blocks keep the full chain from the first attempt.
• Bounded regressions by construction: a downed generator costs at most the grace before failover (the per-attempt cap still bounds hangs), and a genuinely lost fresh witness reverts to exactly the pre-routing behavior after the grace. The grace requires the CLI-declared generator (--witness-generator-endpoint), mirroring the request path's can_skip_generator gate — single-endpoint and no-declared-generator deployments keep plain same-round failover untouched.

Testing

Six new chain_sync tests over real fixture witnesses served by scripted jsonrpsee mocks: a fresh miss retries the generator without touching the fallback; a stale (deep catch-up) miss fails over immediately with no grace; an unknown head classifies as non-fresh; a deployment without a declared generator keeps plain same-round failover; a downed generator falls back right after the grace; latest_block_number feeds the anchor. Plus an rpc_client test pinning that first_provider_only never rotates past provider 0 (mirror of the existing skip test). Full workspace: 301 tests, cargo fmt / clippy --workspace --all-targets --all-features / codespell all clean.

Notes

The public endpoint's 20 s hang-on-miss (vs. its usual ~600 ms -32603 error) is upstream behavior worth checking on the megaeth.com side for the two windows above; this PR removes the sync path's routine exposure to it, but a fallback that stalls when genuinely needed is still bounded only by the 20 s per-attempt cap. The two constants (256 blocks / 6 s) are policy values documented in code with qualitative rationale; their derivation from the log is above.

🤖 Generated with Claude Code

[mega-maxwell]

Claude review status

Living comment — rewritten in place. The review workflow keeps this single comment up to date instead of posting a new one each round, so it always describes the latest reviewed commit and the earlier text is intentionally gone. No reply is needed here; reply to a finding in its own review thread, and answer an open question in a reply on this PR. The next review round reconciles your answer.

🛠️ Review did not finish

Attempted head f3912403 · updated 2026-07-31T10:53:43+00:00

This round did not publish: MODEL_ACTION_FAILED in phase review_retry. Anything listed below is from the last round that did. Re-run the workflow or push a new commit to try again.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1a8d50f558

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 989b1137ac

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[vincent-k2026]

Approving 349c0d3. cargo test --workspace green here. The motivation is well-evidenced and the fix is fail-safe in every degraded direction I traced.

The failure mode I went looking for isn't reachable. I wanted to know whether an unverified eth_blockNumber anchor could be pushed in the dangerous direction — a head reported far in the future making everything look fresh and stalling sync on grace. It can't: the predicate is head − number <= FRONTIER_FRESHNESS_BLOCKS, so a too-high head makes the difference larger and the block classifies as not fresh, falling back to today's full chain. A lie in the harmful direction disables the grace rather than forcing it. Unknown head lands in the same place via the u64::MAX init and saturating_sub (chain_sync.rs:75, :94), and a reorg lowering the head just shrinks the fresh band — remote_head and TipTracker.tip are set from the same latest_block_number call, so they stay in lockstep and the fetcher never asks above the tip. Relaxed ordering is correct here since the value gates a heuristic only.

Your marginal-cost claim checks out arithmetically. With per_attempt_timeout = 20 s, a stalled generator costs grace (6 s, cancelled by the caller-side timeout) plus a full 20 s re-probe of provider 0 on the fall-through — ~26 s absolute. Pre-PR the full chain also probed provider 0 first and paid that same 20 s, so the increment really is exactly the grace. Cancellation at the timeout boundary is clean: the witness_concurrency permit drops as an RAII guard and the in-flight reqwest future is dropped rather than left poisoning a pool.

The shared-crate generalization is bit-for-bit compatible. _from(skip) now calls witness_round_robin(skip..len, ..); the old skip < len assert and the new !providers.is_empty() && providers.end <= len (rpc_client.rs:714-718) are equivalent for that range, and the slice is identical — so the validator binary and an external embedder see no change. first_provider_only uses 0..1 behind the witness_provider_count() > 1 gate, so it can't be empty, and an empty range would panic rather than silently no-op.

On the deferred "skip the stalled generator after the grace" — I agree with you, and I'd put the argument slightly stronger than you did. It isn't only a permanent skip that's unsafe: even a per-fetch ..._from(1, ..) on the fall-through is wrong, because the fallbacks are fed by the same pipeline. For a generator that is merely slow rather than down — witness landing at 7 s — skipping provider 0 means querying only endpoints that definitionally don't have it yet, trading generator lag for R2 upload lag. Keeping provider 0 in the fall-through is the correct call, and your livelock reasoning isn't a strawman.

There is one cheap middle ground worth a follow-up issue rather than a change here: cap the fall-through's provider-0 attempt with a shorter per-attempt timeout than the full 20 s. It just had its exclusive 6 s window, so a second full stall is pure waste — this would pull the stalled worst case from ~26 s down to ~6–8 s without the skip's livelock risk.

Tests: all seven discriminate, and two of them answer exactly the questions I'd have asked. fresh_witness_miss_retries_generator_without_touching_fallback proves non-contact with a request counter on the fallback mock (fb_hits == 0, gen_hits == 3), not just a happy-path return; fresh_fetch_falls_back_after_grace_when_generator_is_down uses a real counting RpcMetrics sink to assert deadline_exceeded == 0, which is the right way to pin the ca68d49 metric-semantics fix. no_declared_generator_keeps_plain_failover would fail if the generator_first gate were dropped, and test_witness_fetch_first_only_never_hits_later_providers would fail if 0..1 regressed to 0..len.

Two non-blocking notes:

A fully down generator now costs ~6 s per fresh block that it didn't before. Pre-PR a connection-refused generator was near-free — instant rotation. Post-PR the probe backs off and retries the dead endpoint for the whole grace before failing over, so throughput in that degraded state is roughly in_flight_window / 6 s; with window = cores·2 that needs ≥5 cores to hold ~1.6 blk/s. Comfortable on our boxes, tight on small ones. If you want to close it cheaply: break the grace early on a fast error from provider 0 (as opposed to a stall), since backing off against a refused connection buys nothing.

GENERATOR_WITNESS_GRACE's doc (chain_sync.rs:31-42) states the down/never-written failover delay but not the stalled worst case. An operator reading the constant will take 6 s as the ceiling when a stalled generator on a fresh block actually costs ~26 s. One sentence. Relatedly the 6 s isn't tied to a measured p99 generation lag in the comment — worth noting the assumption alongside the 1–2 s typical figure.

Merge-order note, no action needed here: git merge-tree against #161 and #162 shows conflicts in chain_sync.rs and rpc_client.rs, but they're all textual — the chain_sync test module's imports and helpers plus the TraceFetcher struct region. Neither of those PRs changes TraceFetcher::fetch's witness call shape, the witness deadline model, or the pipeline reorg/stale-reset hooks, so whichever lands second needs a mechanical rebase of the test module and nothing more.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7660f5bedb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".