subservers: report remote sub-server disconnects at runtime - #1373
Open
Vandit1604 wants to merge 2 commits into
Open
subservers: report remote sub-server disconnects at runtime#1373Vandit1604 wants to merge 2 commits into
Vandit1604 wants to merge 2 commits into
Conversation
Vandit1604
force-pushed
the
644-remote-subserver-disconnect-status
branch
from
August 13, 2026 18:49
9d4b8ed to
af43e4c
Compare
Vandit1604
marked this pull request as ready for review
August 13, 2026 23:54
ViktorT-11
self-requested a review
August 24, 2026 08:16
Vandit1604
force-pushed
the
644-remote-subserver-disconnect-status
branch
from
August 24, 2026 11:07
af43e4c to
f75779f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
In remote mode, a sub-server that disconnected after startup was still reported as
runningby the status server, solitcli statuskept showing it as healthy. Only failures during startup were tracked; a disconnect at runtime was not.This adds a watcher on each remote sub-server's gRPC connection. When the connection drops (
TransientFailure/Shutdown) the sub-server is marked errored, and when it recovers (Ready) it is marked running again, so the status server reflects the real runtime state.Two smaller things came along with it:
Manager.Stopnow also tears down remote sub-servers (it skipped them before), so the watcher and the remote connection are stopped on shutdown. The remote branch instop()was effectively dead before, since remote servers never reached it.status.SetErroredlogged "could not start the X sub-server", which is wrong for a runtime disconnect, so the log is now neutral.Note: the watcher nudges an idle connection with
conn.Connect()so a disconnect is still observed when no requests are in flight. That keeps the remote connection warm rather than letting it idle out, which is the trade-off for observability here.Scope
This is the first, smallest step towards #644, as suggested by @ViktorT-11: get the status reporting right before adding any reconnect/restart logic. It intentionally does not restart sub-servers or touch the lnd reconnect path.
Known gaps left as follow-ups:
pool/faradayexpose no runtime error channel, so their disconnects are still not tracked.Testing
TestWatchRemoteConnstarts a real gRPC backend, stops it, and asserts the disconnect is reported, then restarts it and asserts the recovery is reported.Part of #644.