TtlManager has no priority/budget-aware scheduling — TTL bumps can silently lose the race to archival at scale

[christabel888]

Summary

TtlManager (backend/src/contract_ops/ttl_manager.rs) polls every registered contract entry on every cycle with no prioritization and no accounting for a bounded per-cycle transaction/fee budget. The module's own failure-mode table documents "Contract archived (restore needed)" as a known outcome the manager "cannot auto-restore" — but doesn't reckon with the fact that at scale, the poll-everything design makes archival an inevitability, not just a possible failure mode, once the registry outgrows what one cycle can service.

Location

backend/src/contract_ops/ttl_manager.rs (679 lines) — the poll loop described in the module doc:

1. Iterates over every ContractEntry in the ContractRegistry.
2. Queries the Stellar node for the remaining TTL ... via RpcClient.
3. Compares the remaining TTL against the per-contract TtlPolicy::threshold.
4. If below threshold, issues an extendFootprintTtl transaction ...

backend/src/contract_ops/registry.rs (ContractEntry, ContractRegistry, TtlPolicy).

Current gap / Motivation

Soroban state archival is unforgiving: once an entry's TTL expires, it is archived and requires an explicit, separate restore operation before it's usable again — there is no grace period, and (per this module's own docs) the manager has no auto-restore path. The entire value of TtlManager rests on the guarantee that it will bump every entry before its TTL runs out.

That guarantee only holds today if a full poll-and-bump cycle over the entire registry — one query per instance, one query per known persistent storage key, and one extendFootprintTtl transaction per entry below threshold — completes within a single poll interval, for every entry, indefinitely. There is no such bound. As the registry grows (this is explicitly a system meant to track contract state across an analytics platform with potentially many indexed contracts and many persistent keys per contract), one of two things happens with no warning:

• Rate limits or fee-budget constraints on the RPC node/network mean not every "below threshold" entry can actually be bumped in a given cycle.
• The cycle itself simply takes longer than the poll interval, and the next cycle starts late (or overlaps, depending on how the poll loop is scheduled) — silently degrading the manager's actual bump cadence for the entries it hasn't reached yet.

Either way, an entry sitting right at the edge of TtlPolicy::threshold can lose the race and archive — a permanent, requires-manual-intervention outcome — purely because it happened to be near the end of the iteration order in a cycle that ran out of budget or time, with nothing in the system having noticed this was coming.

The hard part

This is a real-time scheduling problem under resource contention, not a "add pagination" fix:

1. Earliest-Deadline-First-correct prioritization. Entries closest to their TTL threshold must be serviced first whenever a cycle can't service everything — but "closest to threshold" isn't static; it needs to be recomputed relative to the current remaining TTL fetched each cycle, and the scheduler must be able to prove (not just hope) that under a defined maximum sustained registry growth rate and RPC throughput, no entry can be starved past its deadline.
2. True budget-awareness, not just a time limit. The real constraint is a composite of RPC request rate, transaction submission rate, and available fee balance — the manager needs a coherent model of "how much bump-work can I actually do this cycle" and must degrade gracefully (prioritizing correctly) rather than just doing as much as fits before an arbitrary timeout.
3. Batching where the protocol allows it. extendFootprintTtl operations can potentially cover multiple footprint entries in a single transaction — a scheduler-aware design should batch bumps for the same contract (or across contracts, if the transaction model allows) to get more entries serviced per unit of the fee/rate budget, rather than issuing one transaction per entry unconditionally.
4. Observable, provable safety margin. Given archival is irreversible-without-manual-restore, the system needs a way to know — via metrics, not by finding out an entry archived — when the registry has grown to the point that the current poll interval and budget can no longer guarantee every entry is serviced before its deadline, so operators get warned before the first archival, not after.
5. Correctness under partial-cycle failure. If the manager crashes or the RPC node becomes unreachable partway through a cycle (a scenario the existing failure-mode table already anticipates for the simple case), the prioritized scheduling state must survive and resume correctly — a naive restart that goes back to iterating "every entry in registry order" defeats the whole prioritization design under repeated partial failures.

Implementation

• Replace unconditional full-registry iteration with a priority structure (e.g. a min-heap keyed by remaining-TTL-relative-to-threshold) recomputed each cycle from fresh RPC data.
• Model the per-cycle budget explicitly (requests, transactions, fee balance) and define a deterministic, provably-EDF-respecting policy for what happens when budget runs out before the queue is empty.
• Batch extendFootprintTtl calls where the transaction/footprint model permits it.
• Add a metric (and, ideally, an alert-worthy signal) for "registry growth has outpaced the current interval/budget's ability to guarantee coverage" computed proactively, not reactively from an archival event.
• Add a test harness simulating registry growth and a constrained per-cycle budget, asserting no entry with a valid renewal path ever crosses its threshold unserviced.

Acceptance criteria

• A test demonstrates that under a bounded per-cycle budget insufficient to service every entry, the entries closest to expiry are still serviced first, cycle over cycle, with a documented and tested worst-case staleness bound for lower-priority entries.
• A metric exists that signals "current registry size + growth rate exceeds what the configured interval/budget can guarantee" before any entry actually archives.
• extendFootprintTtl calls are batched where the transaction/footprint model allows, with a documented rationale for the batching boundary chosen.
• The manager's behavior across a mid-cycle crash/restart preserves correct prioritization rather than reverting to unprioritized iteration.
• Existing cargo test suite stays green.

[GazzyLee]

gm gm, I’d be happy to work on this. I’ll redesign TtlManager around fresh TTL measurements and an explicit earliest-deadline-first priority queue, with bounded RPC, transaction, and fee budgets, batching extendFootprintTtl operations where the transaction model allows. I’ll add tests for constrained budgets, registry growth, worst-case staleness, and crash/restart recovery, plus a proactive capacity metric that warns when the registry can no longer be safely serviced within the configured interval. I’ll document the scheduling, batching, and safety guarantees and verify the existing cargo test suite remains green.
Please assign

[Akinyemi04]

i would like to work on this issue

[grantfox-oss]

🦊 GrantFox — @Akinyemi04 has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #382)
2. Your PR will be reviewed by the Stellar-Insightss maintainers

Good luck! Track your progress on GrantFox.