Export dc-auto-top leg liveness and DCA input gauges - #1263
Open
madninja wants to merge 4 commits into
Open
Conversation
A dc-auto-top leg is self-rescheduling: each run queues its own next task as its
last step, so a healthy leg always points at a task whose trigger is in the
future. Every way a leg breaks ends the same way, with no reschedule, after
which tuktuk retries until the task goes stale and is swept and nothing restarts
it without schedule_task_v0. Nothing observed that, so a stalled leg was only
visible once its HNT buffer drained days later.
solana_auto_top_off_task_trigger{name,leg} reports the trigger time of the task
each leg points at, and 0 when none is scheduled, so a single
`time() - value > interval` expression covers a leg that is failing to execute
and one whose task has been swept.
Also export the auto-top-off USDC balances and the USDC/USD pyth publish time,
the two inputs the HNT leg's DCA reads: it transfers a whole run's worth of USDC
up front and refuses to start on a feed older than five minutes, and a shortfall
or a stale feed reverts the run rather than shrinking it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tuktuk reuses task ids. Once a leg's task is swept, another program's task takes the same address, so the stored pointer still resolves and the trigger it carries is somebody else's. A dead leg then reported a healthy future trigger, which is the failure this gauge exists to catch. dc-auto-top stamps every task it queues with `topoff <leg> <truncated key>`, so the description separates its own task from a squatter at the same id. A task that does not carry the expected stamp now reports 0, alongside the sentinel and swept cases. Observed on mainnet: the helium mobile HNT leg pointed at task 9523, `queue entity_claim`, and read healthy while being dead. With the check it reports 0, and the three legs that do own their tasks still read healthy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A
dc-auto-topleg is self-rescheduling: each run queues its own next task as its last step. A healthy leg therefore always points at a task whose trigger is in the future.Every way a leg breaks ends the same way, with no reschedule: a stale or wrong-owner price oracle, an empty DCA input account, a leftover
dcaPDA from an undrained run, lamports below the rent-exempt plus crank-reward floor, or a reverting swap. tuktuk then retries the task until it goes stale, sweeps it, and nothing restarts the leg withoutschedule_task_v0.Nothing observed that. The carrier offload leg's HNT half sat broken for several days and the only alert that would eventually have noticed is a balance threshold that trips ~2 days later, once the HNT buffer has drained.
What
solana_auto_top_off_task_trigger{name,leg}reports the trigger time of the task each leg currently points at, and0when none is scheduled. One expression covers both failure shapes:A healthy leg has a future trigger so the difference is negative; a leg that is failing to execute has a past one; and
0makes the same expression catch a swept task rather than needing a second clause.Also exported, since they are the two inputs the HNT leg's DCA reads and neither was visible:
solana_account_balance{type="usdc"}). The DCA transfers a whole run's worth up front, so a shortfall reverts the run rather than shrinking it, which is the no-reschedule path above.solana_pyth_publish_time{name="usdc"}). The leg refuses to start on a feed older than five minutes. Only the HNT feed was exported.This finally uses the
USDC_MINTconstant that has been declared and unreferenced inindex.ts.Notes for review
helium_mobile_auto_topoff/hntreports 0, permanently. Helium Mobile is decommissioned as a separate payer: its traffic moved onto the normal carrier offload router (112c85vb…), which is why that router now carries thehelium_mobilevolume while the custom router112Kdmt…is a rump. SoBvge1…'s HNT leg is not going to be revived, and its 0 is correct rather than a defect. An alert onleg="hnt"must scope toname="carrier_auto_topoff";leg="dc"is still meaningful for both, since that account's DC leg is live. Flagging so the rule is not written against a permanently-firing series.watchretries every 5 minutes. The tradeoff is that a persistent partial failure holds a stale healthy-looking value, which is the same propertysolana_pyth_publish_timealready has.ws@8.18.0to8.18.2inside the@orpc/*peer-suffix strings. Any freshpnpm installproduces this, and hand-editing a lockfile to avoid it would be worse.Verification
pnpm turbo run build --filter=@helium/monitor-serviceis green, run against a tree using the same pre-built idls plustarget/typesplaceholder the Dockerfile itself sets up, which is what CI'stest-docker-buildsgate exercises.The compiled monitors were also run against mainnet and their emitted metrics read back, rather than relying on the build alone:
The
timestampdecode path is exercised by those non-zero readings, so thenow-trigger fallback is the only unexercised branch.