Skip to content

subservers: report remote sub-server disconnects at runtime - #1373

Open
Vandit1604 wants to merge 2 commits into
lightninglabs:masterfrom
Vandit1604:644-remote-subserver-disconnect-status
Open

subservers: report remote sub-server disconnects at runtime#1373
Vandit1604 wants to merge 2 commits into
lightninglabs:masterfrom
Vandit1604:644-remote-subserver-disconnect-status

Conversation

@Vandit1604

@Vandit1604 Vandit1604 commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Change

In remote mode, a sub-server that disconnected after startup was still reported as running by the status server, so litcli status kept showing it as healthy. Only failures during startup were tracked; a disconnect at runtime was not.

This adds a watcher on each remote sub-server's gRPC connection. When the connection drops (TransientFailure/Shutdown) the sub-server is marked errored, and when it recovers (Ready) it is marked running again, so the status server reflects the real runtime state.

Two smaller things came along with it:

  • Manager.Stop now also tears down remote sub-servers (it skipped them before), so the watcher and the remote connection are stopped on shutdown. The remote branch in stop() was effectively dead before, since remote servers never reached it.
  • status.SetErrored logged "could not start the X sub-server", which is wrong for a runtime disconnect, so the log is now neutral.

Note: the watcher nudges an idle connection with conn.Connect() so a disconnect is still observed when no requests are in flight. That keeps the remote connection warm rather than letting it idle out, which is the trade-off for observability here.

Scope

This is the first, smallest step towards #644, as suggested by @ViktorT-11: get the status reporting right before adding any reconnect/restart logic. It intentionally does not restart sub-servers or touch the lnd reconnect path.

Known gaps left as follow-ups:

  • Integrated pool/faraday expose no runtime error channel, so their disconnects are still not tracked.
  • Actually restarting the lnd-dependent sub-servers after a disconnect.
  • A remote server that is already down at connect time is briefly shown as running until the first failure transition (the connection is dialled lazily).

Testing

TestWatchRemoteConn starts a real gRPC backend, stops it, and asserts the disconnect is reported, then restarts it and asserts the recovery is reported.

Part of #644.

@Vandit1604
Vandit1604 force-pushed the 644-remote-subserver-disconnect-status branch from 9d4b8ed to af43e4c Compare August 13, 2026 18:49
@Vandit1604
Vandit1604 marked this pull request as ready for review August 13, 2026 23:54
@ViktorT-11
ViktorT-11 self-requested a review August 24, 2026 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant