run restore on all stf periodically

[sapinb]

Description

Run stf::restore() periodically on all state machines based on configured time interval.

This is a catch-all to resolve any lost actions that prevent the stf from continuing execution.
Equivalent to periodically restarting the mosaic node.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature/Enhancement (non-breaking change which adds functionality or enhances an existing one)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactor
• New or updated tests
• Dependency update
• Security fix

Notes to Reviewers

Job scheduler does not currently dedupe actions. With periodic restores, this can potentially cause duplicate job runs, although it will not impact the correctness of the process, just some inefficiency.

Checklist

• I have performed a self-review of my code.
• I have commented my code where necessary.
• I have updated the documentation if needed.
• My changes do not introduce new warnings.
• I have added tests that prove my changes are effective or that my feature works.
• New and existing tests pass with my changes.

Related Issues

[AaronFeickert]

What's the worst-case scenario for the efficiency hit due to job duplication? If this ends up meaningfully affecting runtime, we should understand the extent.

[Zk2u]

If we are going to do this, we should clear the job queue then restore into it. This needs to ensure that the clearing operation is done properly, doesn't have race conditions or violate any of the other guarantees the job scheduler makes to the other components

[Zk2u]

Blocking issue in crates/sm-executor/src/lib.rs inside run_inner.

This PR still rebuilds self.job_handle.recv(), self.net_client.recv(), and self.command_rx.recv() directly inside monoio::select! on every loop iteration. That is the cancel-unsafe pattern we just fixed in #167: if another branch wins after a direct handoff has matched a waiting kanal receiver, the dropped receive future can drop the delivered message.

So as written, this PR reintroduces the same completion-loss class we traced for the GeneratingPolynomialCommitments wedge. Before this can merge, it needs to be rebased onto current main and the periodic restore tick integrated with the persistent pinned receive futures pattern from #167, rather than the old per-iteration recv() futures.

This is on the assumption this is actually a good idea, which i'm not sure it is and we should discuss on slack first