Unexpected interaction between different congestion control policies

[ciprianf]

Describe the bug

This is a fork from the Discord discussion: https://discord.com/channels/914168414178779197/1466777377358086281.

All messages at the same priority share the network pipeline which makes the interaction between messages with different congestion control policies unexpected and potentially hard to reason about. A Block message can block the pipeline up to wait_before_close (default 5 seconds), which makes a Drop message stall far longer than one would expect by default (50 milis).

To reproduce

Described on the discord channel.

System info

Irrelevant.

[ciprianf]

Specifically, the pipeline lock is acquired here https://github.com/eclipse-zenoh/zenoh/blob/main/io/zenoh-transport/src/common/pipeline.rs#L898 with no timeout. A Drop message can be blocked for a long time waiting for a Block one.