Recovery sweep has no per-transaction timeout, so one hung status query stalls every replica

[atharrva01]

Problem

The recovery manager builds its context once at startup, with no deadline
(token/services/storage/services/recovery/manager.go:114):

m.ctx, m.cancel = context.WithCancel(context.Background())

That context is passed straight through runSweep(m.ctx) into leadership acquisition,
ClaimPendingTransactions, and every per-transaction Recover call. Nothing on that path
narrows it, so the only thing that can ever cancel it is Stop().

Recover is a network call. TTXRecoveryHandler.Recover reaches
h.network.GetTransactionStatus(ctx, ...) at token/services/ttx/finality/recovery.go:76,
which queries the ledger for the transaction's status. If that query hangs rather than
failing, an unresponsive peer, a connection that dies without surfacing an error, an orderer
in catch-up, then the worker blocks for as long as the query does.

Why it matters

The blast radius is not one transaction, because leadership is held across the whole sweep.
runSweep (manager.go:216) acquires the advisory lock, defers its release, and only then
calls recoverTransactions, which both claims and processes the batch. So while the leader
is parked on a hung query:

• its own sweep makes no progress
• every other replica's AcquireRecoveryLeadership returns acquired=false and skips its tick
• no pending transaction on that TMS is recovered by anyone, for the duration of the hang

Recovery is the mechanism that repairs transactions whose finality listener was lost. One
unresponsive peer can therefore disable it across every replica of a TMS, quietly, with
nothing timing out to break it.

There is also nothing an operator can configure to bound this. validateConfig
(manager.go:199) only checks that TTL, ScanInterval, BatchSize, WorkerCount and
LeaseDuration are greater than zero. There is no per-transaction or per-sweep deadline.

Related, worth confirming at the same time

LeaseDuration (default 30s) can expire while the leader is still processing, since nothing
renews it. Today that is masked, because leadership serialises the sweep so no other replica
is claiming while one is at work. It stops being masked if sweeps ever run concurrently on a
path where leadership is not enforced. If a renewal mechanism is wanted, there is already a
precedent in auditdb/locker/postgres, whose heartbeatLoop
(token/services/storage/auditdb/locker/postgres/postgres.go:304) renews at roughly TTL/3.

[AkramBitar]

Triage: Priority set to High, Effort Medium. No severity was recorded in the description, so this notes the assessment rather than disagreeing with one.

High because the problem is live under default configuration and its blast radius is fleet-wide:

• Recovery is on by default — DefaultConfig() sets Enabled: true (token/services/storage/services/recovery/config.go:49) — and the manager is constructed and started in production for both the ttx and audit stores (token/services/network/fabric/network.go:461, :472).
• Nothing narrows the context between context.WithCancel(context.Background()) (manager.go:114) and the GetTransactionStatus network call in TTXRecoveryHandler.Recover (token/services/ttx/finality/recovery.go:76). Only Stop() can cancel it.
• Leadership is held across the entire sweep: runSweep acquires the advisory lock, defers leadership.Close(), and only then calls recoverTransactions (manager.go:216-229). So while the leader is parked on a hung query, every other replica gets acquired=false and skips its tick.

The result is that one unresponsive peer can disable recovery for every replica of a TMS, indefinitely, with nothing timing out and no operator-facing bound — validateConfig (manager.go:199-213) has no timeout knob to set. That is the largest real-world exposure among the issues filed in this batch.

Effort: Medium — a per-transaction deadline is a Config field, a validateConfig case, a default, and a context.WithTimeout at the per-transaction call site, plus config documentation. The LeaseDuration renewal raised under "Related, worth confirming at the same time" is separable and should not be folded in.

[AkramBitar]

@atharrva01
I am wondering if you can take this one?

If so, then I will assign it to you.

Thanks a lot for all your support we appreciate it a lot.

Regards,
Akram