Improve failure/recovery

[Stebalien]

Right now, various services can fail early and exit due to some internal error. We don't currently have any way to recover.

Ideally, at the very top, we'd listen for such failures and attempt to re-start (possibly with a backoff).

[Stebalien]

We're going to need #392 for this to be completely correct, but there's no reason to wait on this.

But we will likely need to figure out #368 (comment) first. That is, we need a way to detect if a sub-service has failed and either restart it or restart it (if possible) or exit so our parent can restart us.

E.g.:

1. If the certificate exchange fails, we can likely just restart that and leave everything else running.
2. If the GPBFT participant fails, we likely need to restart most of the F3 stack.

So, the first task here will be to gracefully handle early service aborts.

[Kubuxu]

Adding to M2.5 as a review and confirm item.

[rjan90]

What we need to figure out is active recovery (F3 is in a weird spot).

[BigLep]

@masih : is this needed for nv25, or can it be listed as potential improvement after f3 activation?

[masih]

I think we can take a pass at this after activation 👍