fix(network-fabric): propagate ledger info failures instead of rescanning from genesis

[AkramBitar]

Fixes #2058

The issue

ScanBlock asks the peer for the current ledger height to know where to resume the block scan. If that call failed, the error was only logged and the height stayed 0, so the scan silently restarted from genesis — rescanning the whole chain and replaying finality notifications for every past transaction, with no error for the caller.

The fix

Retry the height lookup a few times, and if it still fails return the error instead of starting from block 0. A passing RPC hiccup is absorbed by the retry; a real outage is reported.

Before

ScanBlock(callback):
    start = 0
    if ledger exists:
        info, err = GetLedgerInfo()
        if err == nil: start = info.Height
        else:          log(err)          # error dropped, start stays 0
    return ScanBlockFrom(start, callback)

One failed RPC → scan restarts from genesis, caller sees no error.

After

ScanBlock(callback):
    start = 0
    if ledger exists:
        start, err = LedgerHeight()
        if err != nil: return err        # no scan at all
    return ScanBlockFrom(start, callback)

LedgerHeight():
    delay = LedgerInfoRetryDelay        # default 500ms
    for attempt = 1 .. LedgerInfoAttempts:   # default 7
        info, err = GetLedgerInfo()
        if err == nil and info != nil:
            return info.Height           # done
        log(err)
        if attempt == last: break
        wait delay or return if context cancelled
        delay = delay * 2
    return error "refusing to rescan from genesis"

What an attempt does

One attempt is one GetLedgerInfo RPC to the peer:

1. If it returns a height, use it and start the scan there. Height is the next block to be delivered, so nothing is replayed.
2. If it returns an error — or (nil, nil), which used to nil-deref — remember the error and log it as attempt N/M.
3. On the last attempt, stop looping. Otherwise wait delay, then double it. A cancelled context ends the wait immediately and returns.

Defaults: 7 attempts, 0.5s + 1s + 2s + 4s + 8s + 16s of waiting, ~31.5s worst case. Both bounds are configurable — see below.

If every attempt fails

• The error is returned, wrapping the last ledger error so errors.Is still reaches it: failed to get ledger info after 7 attempt(s), refusing to rescan from genesis.
• ScanBlockFrom is never called — no scan, so no genesis rescan and no duplicate finality notifications.
• The caller, FSC's ListenerManager.start(), logs the error, so block-delivery finality does not start for that manager. Finality is degraded, not lost: timed-out listeners are queried directly against the ledger (QueryByID → fetchTxs), and the recovery service re-registers listeners for pending transactions.
• Not a regression: if the peer cannot answer GetLedgerInfo across ~31.5s of retries, the ScanBlockFrom stream to that same peer would have failed too. The old code just failed later, after starting a full-chain rescan.

Block 0 is still used in one case only: no ledger configured at all.

Notes

• Retry, not just propagate: ListenerManager.start() calls ScanBlock once and only logs the error, so returning on the first hiccup would leave block-based finality dead until the process restarts. That is also why the budget has to outlast a peer restart rather than a dropped packet.
• *fabric.Delivery / *fabric.Ledger replaced by two small interfaces so the code is unit-testable. Field names unchanged, wiring untouched.

Tests

delivery_test.go, 11 cases: height resumption, error propagated with no scan started, transient failure retried, attempt budget honoured, nil info, cancelled context, nil ledger → block 0, scan error propagated, plus the cancellation/nil-info sentinel overlap, last-failure-only classification, and the default attempt budget.

config/config_test.go, 4 cases (the package had none): defaults, configured values, non-positive rejection, and the budget appearing in String().

Against the old code 4 fail and one panics. All pass with the fix under -count=2 -race; gofmt, go vet, golangci-lint, go build ./... clean.

Configurable retry budget

The 7-attempt default is a judgement call, not a measurement: it is the smallest count on the existing doubling schedule that clears ~30s, chosen as a rough proxy for how long a peer restart takes. How long a peer actually takes to come back is a deployment property, so both bounds are exposed alongside the five existing token.finality.delivery.* settings rather than baked in:

token:
  finality:
    delivery:
      ledgerInfoAttempts: 7        # attempts at reading the starting height
      ledgerInfoRetryDelay: 500ms  # first retry pause; doubles each attempt

Non-positive values fall back to the defaults: zero attempts would refuse every scan, and a non-positive delay would busy loop.

lookup is wired too, not just finality — it constructs the same Delivery, so leaving it out would have silently kept the old budget for lookup listeners. The exported provider constructors (NewDeliveryBasedFLMProvider, NewDeliveryBasedLLMProvider) take the values through a variadic option, so their existing signatures stay source-compatible for downstream callers.

Error classification

Failures are classifiable with errors.Is, so a caller never has to match on message text:

Sentinel	Meaning
ErrLedgerHeightUnavailable	the starting block could not be resolved, so no scan started
ErrNoLedgerInfo	the last attempt saw the ledger return neither info nor an error — a driver contract violation
context.Canceled / context.DeadlineExceeded	a wait between attempts was cut short, or the scan itself was cancelled

ErrLedgerHeightUnavailable accompanies every failure to resolve the starting block, including a cancelled one, which makes it the single test for "no scan started" — and makes "an error without it came from the scan" true.

The two refining sentinels are deliberately documented as not discriminators on their own:

• ErrNoLedgerInfo reports only the last observed failure, so a driver that violates the contract intermittently may surface an ordinary ledger error instead. Its absence is not proof the contract was kept.
• The same context governs the scan, so a context error alone does not say which phase ended. Pair it with ErrLedgerHeightUnavailable to tell a cancelled height read from a cancelled scan.

No in-tree caller inspects these sentinels yet — today the sole consumer logs the error and stops. Retry or restart around ScanBlock belongs to FSC's events.ListenerManager and is out of scope here: the wider budget shrinks that window but does not close it.

[github-actions]

📊 Token Validation Benchmark

Comparison of this PR against the base branch. 🟢 improvement · 🔴 regression · ➖ within ±1.0% noise.

Variant	Benchmark	Params	Workers	TPS (base → PR)	Δ TPS
csp	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	118 → 118	➖ +0.1%
csp	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	116 → 116	➖ +0.1%
ipa	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	78 → 78	➖ -0.3%
ipa	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	68 → 68	➖ -0.4%

[github-actions]

📊 Token Validation Benchmark

Comparison of this PR against the base branch. 🟢 improvement · 🔴 regression · ➖ within ±1.0% noise.

Variant	Benchmark	Params	Workers	TPS (base → PR)	Δ TPS
csp	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	106 → 106	➖ -0.1%
csp	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	121 → 121	➖ +0.0%
ipa	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	78 → 78	➖ -0.2%
ipa	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	76 → 76	➖ +0.1%

[github-actions]

📊 Token Validation Benchmark

Comparison of this PR against the base branch. 🟢 improvement · 🔴 regression · ➖ within ±1.0% noise.

Variant	Benchmark	Params	Workers	TPS (base → PR)	Δ TPS
csp	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	117 → 117	➖ -0.1%
csp	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	177 → 176	➖ -0.1%
ipa	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	78 → 78	➖ -0.1%
ipa	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	76 → 76	➖ +0.1%

[github-actions]

📊 Token Validation Benchmark

Comparison of this PR against the base branch. 🟢 improvement · 🔴 regression · ➖ within ±1.0% noise.

Variant	Benchmark	Params	Workers	TPS (base → PR)	Δ TPS
csp	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	137 → 137	➖ +0.1%
csp	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	115 → 116	➖ +0.3%
ipa	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	70 → 70	➖ +0.2%
ipa	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	76 → 76	➖ +0.1%

[github-actions]

📊 Token Validation Benchmark

Comparison of this PR against the base branch. 🟢 improvement · 🔴 regression · ➖ within ±1.0% noise.

Variant	Benchmark	Params	Workers	TPS (base → PR)	Δ TPS
csp	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	117 → 118	➖ +0.1%
csp	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	116 → 116	➖ -0.1%
ipa	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	78 → 78	➖ +0.0%
ipa	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	79 → 79	➖ -0.2%

[AkramBitar]

@adecaro
Thanks a lot for the review.

I flagging one change: the retry budget is now configurable instead of hardcoded.

Nothing retries ScanBlock — FSC's ListenerManager.start() calls it once and only logs the error — so an unreadable height costs the channel its block-based finality until restart. The old 3 attempts / ~1.5s didn't survive a peer restart, trading the genesis rescan for a silently dead listener.

Default is now 7 / ~31.5s, but that's a guess, not a measurement — hence:

token.finality.delivery.ledgerInfoAttempts: 7
token.finality.delivery.ledgerInfoRetryDelay: 500ms

Do you have a better sense of how long a peer usually takes to come back?
Please let me know your thoughts on that change?

Thanks a million,
Akram

[github-actions]

📊 Token Validation Benchmark

Comparison of this PR against the base branch. 🟢 improvement · 🔴 regression · ➖ within ±1.0% noise.

Variant	Benchmark	Params	Workers	TPS (base → PR)	Δ TPS
csp	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	147 → 147	➖ -0.4%
csp	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	115 → 115	➖ -0.0%
ipa	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	78 → 78	➖ +0.2%
ipa	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	76 → 76	➖ +0.1%

[SaidAltury-ibm]

there's some failing test

[SaidAltury-ibm]

LGTM