More resilient startup behavior and feedback for the user

[da-kami]

If Electrs' RPC is not available then rocket will not start up, meaning that one does not get any feedback because there is no user interface.

This problem is especially problematic in environments where we cannot apply external health checks like on Umbrel. If the not start on Umbrel the user will have to query for the logs, but on Umbrel (at least to my knowledge) one cannot easily retrieve the logs for only one application unless SSHing into the machine and querying the docker logs, so this is super cumbersome.

On Start9 we can at least query the /alive endpoint and get feedback if the application is started and the UI available. But that does not help much with understanding what the problem is; the user will still have to go through the logs to understand the problem.

On Electron we could specify external monitoring of dependencies in the Electron context. At the moment the UX is horror, because there won't be a UI and the logs are hidden 😅

I am specifying proposals to deal with this problem as answers.

[da-kami]

Proposal 1: Ensure that rocket always starts and provide feedback in the UI

We make our startup more resilient and provide more feedback about the applications status to the user in the ItchySats UI.

First, we will have to make sure that there are no blocking errors that can prevent rocket (which serves the UI) from starting.
Second, the UI needs to get information about the application status.

I propose a /status endpoint where we can query the daemon for the status. We could also do this as part of the feed, but that is more tricky because that would be more of an alerting system (with "fire" and "resolve") like in Grafana. I think a simple status endpoint that can be queried initially and then periodically in an interval is a simpler solution. Internally we can have a SystemStatus actor that manages the system status state.

Given that we have some environments (notably Umbrel) where we cannot easily specify meaningful external health checks I lean towards this proposal.

[klochowicz]

I like this solution, as this would also work for the electron app 👍

[bonomat]

+1 for the /status endpoint and not putting it into the feed.

[holzeis]

I also do like this proposal better - however, what would be the difference between the status and the alive endpoint? I suggest we combine them and return a more verbose health response about the service.

[klochowicz]

The difference would be that /alive can be accessed unauthenticated, a more comprehensive health report meanwhile should only be accessible to authenticated users.

[holzeis]

hmm.. it is true that the alive check is currently unauthenticated, but that does not look to me the differentiating aspect of that endpoint. In its nature it is doing the same - telling you if the service is available (but now a bit more granular than yes or no). IMHO You could easily argue that the alive endpoint should also be authenticated resp. that the status endpoint does not have to be authenticated.

[da-kami]

Proposal 2: Health management outside of the application

This would allow us to ensure that dependencies work before even starting the application which would reduce the code / complexity of the implementation in ItchySats.

E.g. EmbassyOs (Start9) actually displays a warning for ItchySats if Electrs is having a problem. For Electron we could also perform dependency checks on a startup screen that is managed within the Electron context outside of ItchySats. For Umbrel this is a bit tricky - I don't think we can provide dependency information to the user outside of ItchySats.

One thing that is not solved by this proposal is the behavior if e.g. Electrs becomes unavailable while the application is already running. Even if we monitor this from the outside, we can hardly just stop ItchySats and tell the user to fix the problem before the user can use it again. I think the UX of Proposal 1 would be better.