Skip to content

Export dc-auto-top leg liveness and DCA input gauges - #1263

Open
madninja wants to merge 4 commits into
developfrom
monitor-dc-auto-top-liveness
Open

Export dc-auto-top leg liveness and DCA input gauges#1263
madninja wants to merge 4 commits into
developfrom
monitor-dc-auto-top-liveness

Conversation

@madninja

@madninja madninja commented Aug 17, 2026

Copy link
Copy Markdown
Member

Why

A dc-auto-top leg is self-rescheduling: each run queues its own next task as its last step. A healthy leg therefore always points at a task whose trigger is in the future.

Every way a leg breaks ends the same way, with no reschedule: a stale or wrong-owner price oracle, an empty DCA input account, a leftover dca PDA from an undrained run, lamports below the rent-exempt plus crank-reward floor, or a reverting swap. tuktuk then retries the task until it goes stale, sweeps it, and nothing restarts the leg without schedule_task_v0.

Nothing observed that. The carrier offload leg's HNT half sat broken for several days and the only alert that would eventually have noticed is a balance threshold that trips ~2 days later, once the HNT buffer has drained.

What

solana_auto_top_off_task_trigger{name,leg} reports the trigger time of the task each leg currently points at, and 0 when none is scheduled. One expression covers both failure shapes:

time() - solana_auto_top_off_task_trigger{leg="hnt"} > 3600

A healthy leg has a future trigger so the difference is negative; a leg that is failing to execute has a past one; and 0 makes the same expression catch a swept task rather than needing a second clause.

Also exported, since they are the two inputs the HNT leg's DCA reads and neither was visible:

  • auto-top-off USDC balances (solana_account_balance{type="usdc"}). The DCA transfers a whole run's worth up front, so a shortfall reverts the run rather than shrinking it, which is the no-reschedule path above.
  • USDC/USD pyth publish time (solana_pyth_publish_time{name="usdc"}). The leg refuses to start on a feed older than five minutes. Only the HNT feed was exported.

This finally uses the USDC_MINT constant that has been declared and unreferenced in index.ts.

Notes for review

  • helium_mobile_auto_topoff/hnt reports 0, permanently. Helium Mobile is decommissioned as a separate payer: its traffic moved onto the normal carrier offload router (112c85vb…), which is why that router now carries the helium_mobile volume while the custom router 112Kdmt… is a rump. So Bvge1…'s HNT leg is not going to be revived, and its 0 is correct rather than a defect. An alert on leg="hnt" must scope to name="carrier_auto_topoff"; leg="dc" is still meaningful for both, since that account's DC leg is live. Flagging so the rule is not written against a permanently-firing series.
  • An RPC failure deliberately does not zero the gauge. The handler logs and leaves the previous reading, because reporting a transient read failure as a stalled leg would page on every blip; watch retries every 5 minutes. The tradeoff is that a persistent partial failure holds a stale healthy-looking value, which is the same property solana_pyth_publish_time already has.
  • The lockfile diff includes unrelated churn. Adding the two dependencies re-resolved ws@8.18.0 to 8.18.2 inside the @orpc/* peer-suffix strings. Any fresh pnpm install produces this, and hand-editing a lockfile to avoid it would be worse.

Verification

pnpm turbo run build --filter=@helium/monitor-service is green, run against a tree using the same pre-built idls plus target/types placeholder the Dockerfile itself sets up, which is what CI's test-docker-builds gate exercises.

The compiled monitors were also run against mainnet and their emitted metrics read back, rather than relying on the build alone:

solana_auto_top_off_task_trigger{name="carrier_auto_topoff",leg="dc"}          1787068800   (16:00Z, healthy)
solana_auto_top_off_task_trigger{name="carrier_auto_topoff",leg="hnt"}         1787069100   (16:05Z, healthy)
solana_auto_top_off_task_trigger{name="helium_mobile_auto_topoff",leg="dc"}    1787068800   (healthy)
solana_auto_top_off_task_trigger{name="helium_mobile_auto_topoff",leg="hnt"}            0   (no task)
solana_pyth_publish_time{name="usdc"}                                         1787000639
solana_account_balance{name="carrier_auto_topoff",type="usdc"}                         0

The timestamp decode path is exercised by those non-zero readings, so the now-trigger fallback is the only unexercised branch.

madninja and others added 4 commits August 17, 2026 16:33
A dc-auto-top leg is self-rescheduling: each run queues its own next task as its
last step, so a healthy leg always points at a task whose trigger is in the
future. Every way a leg breaks ends the same way, with no reschedule, after
which tuktuk retries until the task goes stale and is swept and nothing restarts
it without schedule_task_v0. Nothing observed that, so a stalled leg was only
visible once its HNT buffer drained days later.

solana_auto_top_off_task_trigger{name,leg} reports the trigger time of the task
each leg points at, and 0 when none is scheduled, so a single
`time() - value > interval` expression covers a leg that is failing to execute
and one whose task has been swept.

Also export the auto-top-off USDC balances and the USDC/USD pyth publish time,
the two inputs the HNT leg's DCA reads: it transfers a whole run's worth of USDC
up front and refuses to start on a feed older than five minutes, and a shortfall
or a stale feed reverts the run rather than shrinking it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tuktuk reuses task ids. Once a leg's task is swept, another program's task takes
the same address, so the stored pointer still resolves and the trigger it carries
is somebody else's. A dead leg then reported a healthy future trigger, which is
the failure this gauge exists to catch.

dc-auto-top stamps every task it queues with `topoff <leg> <truncated key>`, so
the description separates its own task from a squatter at the same id. A task
that does not carry the expected stamp now reports 0, alongside the sentinel and
swept cases.

Observed on mainnet: the helium mobile HNT leg pointed at task 9523,
`queue entity_claim`, and read healthy while being dead. With the check it
reports 0, and the three legs that do own their tasks still read healthy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant