Allow graceful termination of ObjectDiffusion from the client side -> make sure it regains agency frequently enough

[tbagrel1]

There is currently a couple of issues with the current design of ObjectDiffusion w.r.t. miniprotocol technical requirements, as identified in the meeting with the network team on 2025/12/17 (cc @coot @crocodile-dentist):

1. There is no way for the client side to terminate gracefully if it is blocked on waiting an answer to a blocking request for ID. Indeed, the server as agency in this situation, but is not allowed to answer with an empty list of objects. The only way is to wait for a "long timeout"
- But it is a "requirement" for a node to have a way to terminate gracefully its client-side miniprotocols at will within ~30s, which would suggest "long timeout < 30s"
- But due to the parametrization of Peras, the duration of a round, etc, for certificate diffusion, with Peras in its "happy path", the server may not have any new object available within 90s, which suggests "long timeout > 90s" (otherwise a blocking request will almost always fail with timeout)

The design proposed in #144 would not help here, since the client would still be blocking (in state StObjIdsMustReplay) on receiving an answer from the server for a long (~ 90s) span of time.

2. If a Peras cooldown happens, then the ObjectDiffusion miniprotocols (for votes and certs) will fail ungracefully at some point since the consecutive Peras cooldown duration is not really bound and could last several minutes/hours. It would be better, when a cooldown is detected, to have a way to terminate the peras miniprotocols gracefully (and re-initiate them later)

Potential solutions for 1:

S1-A. Make the server answer with "keep-alive" objects every X seconds, so that we can use a relatively short timeout even for blocking requests. But this would consume extra network bandwith (this is my "naive" potential solution)

S1-B. Do not use a blocking ID request, but instead do a simple polling system based on non-blocking requests (pseudo blocking request = retry maximum $n$ times a non-blocking request, with a $smallDelay$ in between each, and if we still haven't received anything, sleep for $longDelay$ and repeat) (this has been proposed by the network team)

The goals of both solutions is to make sure the client regain agency frequently enough, as it is the only one who can trigger a graceful termination with MsgDone. Both solutions could also impact heavily the caught-up detection we want to implement in #144

Potential solutions for 2:
Easier for this one I think; at least in theory. The component in charge of deciding if we vote or not should have a way to inform the network layer that we need to terminate these protocols because of a cooldown. I'm not exactly sure how the restart would work though

In any case, this seems to require changes that depart significantly from the Peras design document. Typically, making a non-blocking request for IDs when we don't currently have unacknowledged ones is currently a protocol violation.

Note

This has been detected while trying to unblock IntersectMBO/ouroboros-network#5267, but it isn't clear if solving this issue will completely unblock IntersectMBO/ouroboros-network#5267

[ch1bo]

There is no way for the client side to terminate gracefully if it is blocked on waiting an answer to a blocking request for ID

What's the backstory here? In which situations is this problematic? i.e. Why is this a requirement?

[crocodile-dentist]

There is no way for the client side to terminate gracefully if it is blocked on waiting an answer to a blocking request for ID

What's the backstory here? In which situations is this problematic? i.e. Why is this a requirement?

The practical diffusion layer consideration is that peer selection mechanism periodically decides to demote a hot peer down to warm, and this has to complete within a specific period or the connection is dropped. Furthermore, if everything goes well, this peer has a chance to be promoted back to hot eventually. A client which is blocked indefinitely, or just for 'too long' - which is satisfied by the certificate round duration - will adversely affect peer selection, since the connection will be dropped and it will first have to sampled to warm to have a shot at being promoted back to hot/active again.

[ch1bo]

@crocodile-dentist Thanks for the context! This helps me understand this.

Demoting from hot to warm then means putting all mini-protocols into an idle, client-side agency state? I could not find anything about this policy and had a quick look at https://ouroboros-network.cardano.intersectmbo.org/ouroboros-network/Ouroboros-Network-PeerSelection-Governor.html

[coot]

hot -> warm demotion stops all hot mini-protocols, e.g. chain-sync, block-fetch, tx-submission; We also have established mini-protocols, which continue to be operational (keep-alive and peer-sharing).

I am not sure if this is documented somewhere...

[coot]

There could be another way of fixing this, but it requires a bit experimentation. It is more principled, but much more complex. I outline it here if one will want to experiment with it in the future. Crucially, this approach will not require polling, but using STM instead to resolve race conditions (for which STM is pretty good).

Driver typed-protocols for the client

On the client side we'd need an STM action which returns next message from the protocol. I had something similar for one of our experiments with typed-protocols, (e.g. CollectSTM which is combined with recvMessageSTM). Then we can combine the recvMessageSTM with a signal from the outbound governor to terminate the mini-protocol (e.g. first-to-finish synchronisation).

Client changes in typed-protocols

When the termination signal wins the race, the client could pipeline MsgDone. This potentially requires some changes in the the Client API.

Server changes in typed-protocols

The server would need to be able to wait on the MsgDone while it's blocked on getting more objects to send. The challenge here is that when we do so, we haven't yet sent the response, so normal progression of Server which is done thanks to its recursive nature is not possible.

But we could construct a continuation as an STM transaction, which either returns the message with object ids if they become available, or if MsgDone was received, we construct an empty reply follow by handling MsgDone (e.g. termination of the mini-protocol). This would require a new constructor in the Server API, maybe that's the same change that's needed for the client which is outlined above in the Driver section.

[tbagrel1]

@coot would this force us to introduce a new constructor AgencyBoth :: Agency?

[coot]

No, the idea is to avoid that. The protocol would still uphold to the same state machine.