feat(backend): connect the inactivity watchdog to on-chain Soroban payout execution

[Yunusabdul38]

Closes #1039

What was missing

InactivityWatchdogService swept expired plans, flipped their status to TRIGGERED in PostgreSQL and enqueued a plan.triggered webhook — but never touched the chain. No inheritance was unlocked and no off-ramp payout was initiated. StellarSubmitClient could only relay a pre-signed envelope to Horizon; it had no way to build or sign a Soroban invocation.

What this does

backend/src/stellar_submit.rs — Soroban invocation

StellarSubmitClient gains an optional Soroban context, configured from the environment:

Variable	Required	Default
SOROBAN_RPC_URL	yes	—
INHERITANCE_CONTRACT_ID	yes	—
STELLAR_SIGNER_SECRET	yes	—
STELLAR_NETWORK_PASSPHRASE	no	testnet
SOROBAN_POLL_INTERVAL_MS	no	1000
SOROBAN_POLL_TIMEOUT_SECS	no	60

with_soroban() validates the S… and C… strkeys immediately, so a misconfigured signer fails at startup rather than on the first expired plan (main.rs exits rather than running with a half-configured signer).

invoke_contract() runs the full flow: read the signer's sequence from Horizon → build an InvokeHostFunction transaction → simulateTransaction over JSON-RPC for the footprint, minResourceFee and auth entries → re-assemble, sign for the network → sendTransaction → poll getTransaction until it lands, fails, or times out. trigger_inheritance(plan_id) wraps it for this contract's trigger_inheritance(caller, plan_id).

Events are decoded out of TransactionMeta V3 and V4 (protocol 23 moved contract events onto the operation meta), with helpers to match on topics and read a u64 out of a struct event payload.

backend/src/inactivity_watchdog.rs — two-phase sweep

1. Expired plans are claimed into a new TRIGGERING status under the existing advisory lock, so a concurrent worker cannot pick them up.
2. Each is triggered on-chain outside the transaction. A plan reaches TRIGGERED only if the contract's INHERIT/TRIGGER event is present and carries the plan_id we asked for. Anything else — no event, wrong plan, submission rejected — lands in TRIGGER_FAILED with the reason stored in last_trigger_error.

Retry: exponential backoff via INACTIVITY_WATCHDOG_ONCHAIN_MAX_ATTEMPTS (3), _ONCHAIN_BACKOFF_MS (1000) and _ONCHAIN_MAX_BACKOFF_MS (30000). Only transport/inclusion failures are retried; a contract-level rejection will be rejected identically next time, so it fails fast.

Alerting: failures log at error, increment inheritx_watchdog_onchain_triggers_total{outcome="failure"}, and enqueue a new plan.trigger_failed webhook. plan.triggered now carries trigger_tx_hash and onchain_plan_id.

Crash recovery: a plan left in TRIGGERING by a worker that died mid-submission is re-claimed once it goes stale (INACTIVITY_WATCHDOG_TRIGGER_STALE_AFTER_SECS, default 900).

Backwards compatible: with no Soroban configuration the watchdog behaves exactly as before and warns loudly at startup that payouts are not being executed on-chain.

Supporting changes

• Migration 20260820000000 adds onchain_plan_id (the contract's u64 plan id, unique where present, so one DB plan cannot double-submit), trigger_tx_hash, trigger_attempts, trigger_started_at, last_trigger_error.
• POST /api/plans accepts an optional onchain_plan_id and returns it on plan responses — without a way to set it the integration would be inert. Existing clients are unaffected (the field is optional).
• main.rs builds one Stellar client and shares it between the API and the watchdog.

Baseline fix — please read (first commit)

While verifying the new migration I found the migration chain has never applied to an empty database. sqlx::migrate!() aborts on the first failure, and main.rs only logs a warning, so this was invisible:

1. 20260624000000_add_inactivity_watchdog_fields computed inactivity_deadline_at as last_ping + INTERVAL, but last_ping is a BIGINT of Unix seconds — operator does not exist: bigint + interval. This is the watchdog's own migration, so on a fresh database status and inactivity_deadline_at do not exist and every sweep errors.
2. 20260628000000 was used by two migrations (add_kyc_records and create_admins) — sqlx rejects the duplicate version with VersionMismatch.
3. 20260729000000_create_plans_beneficiaries_payouts re-creates plans and beneficiaries, which already exist — relation "plans" already exists.

The first commit repairs all three: derive the deadline with to_timestamp(), renumber create_admins to 20260628000001 (order preserved), and reduce the third migration to the only table that was actually new (payout_logs, now referencing the existing beneficiaries). Nothing in the codebase referenced the duplicate definitions' columns.

Since none of these migrations ever applied anywhere, editing them in place is safe — there are no recorded checksums to invalidate. Happy to split this commit into its own PR if you'd prefer to review it separately — but without it the column this feature needs can never be created.

Testing

• Unit: network-id/signature-hint derivation, event decoding from V3/V4 meta, topic + contract + plan_id matching, strkey config validation, retry classification, backoff doubling and capping.
• Database-backed (backend/tests/inactivity_watchdog_db_test.rs): the sweep's claim/finalise SQL and the full migration chain, covering the no-chain path, a plan with no onchain_plan_id, stale re-claim and fresh in-flight plans.
• CI's PostgreSQL service was declared but unused and had no health check. Added one, and set WATCHDOG_TEST_DATABASE_URL on the test step so the new tests actually run. Deliberately not DATABASE_URL — existing tests assert behaviour when the database is unreachable.

Locally, on the pinned 1.88 toolchain: cargo fmt --check, cargo clippy --all-targets --all-features -- -D warnings, cargo test (88 passing) and cargo build --release all clean.

Follow-ups (not in scope here)

• trigger_inheritance returns InheritanceAlreadyTriggered when the chain already triggered the plan. Today that is recorded as TRIGGER_FAILED; decoding the contract error code out of resultXdr and treating it as success would be a nice refinement.
• The contract's trigger_inheritance freezes loans and records trigger state; the anchor off-ramp in stellar_anchor.rs is still driven separately by POST /api/payouts.

[ONEONUORA]

Thank you for your contribution @Yunusabdul38