Restore decoupling

[joaoantoniocardoso]

As a follow-up to some problems found in #593's introduced strategy of removing the decoupling between the sinks, this patch:

• Finds the missing piece that previously made the WebRTCSink queue accumulate buffers: WeBRTCBin's inner clocksync elements were sync=true. Now we finally have a truly sync=false WebRTC, which means it always runs with empty queues in steady state.
• Adds the decoupling back to each sink, so disturbances on one sink can't propagate upstream and/or to other sinks.
• Removes unneeded queues from the source pipelines so oscillations that amplify freeze-and-burst events can't happen.
• Refactors WebRTCSink so it resembles the other sinks more closely.
• The added measured latency sits between 2 and 5ms total (Pi 4), which seems a good price for the robustness.

Solves:

• bug: mavlink-camera-manager loses frames on 1.5.0-beta.36 bluerobotics/BlueOS#3927

Evaluation:

• Tested against behavioral regressions (using [next] [6] Add integration test suite with dynamic ports and parallel execution #594).
• Tested against performance regressions using hypothesis tests.
• Tested against high CPU use and network impairment.

[patrickelectric]

just to be sure, we are failing here, but the insert will happen still ?

[joaoantoniocardoso]

Great question. In short, this is keeping the same behavior, which contains a fragility we need to improve. See below.

Each Pipeline has a PipelineRunner, which runs independently. The "Streams" are groups of one Source Pipeline + multiple Sink Pipelines.

When the Source Pipeline of a Stream fails (e.g., if v4l2src's device is closed by Linux), there's a watcher that tears down and recreates the whole Stream (Source Pipeline + each Sink Pipeline).

When a Sink Pipeline of a Stream fails (e.g., if something happens to the socket), it just dies forever.

The lack of communication upstream (from Sink Pipelines to Source Pipelines) was intentional: we don't want the Source or other Sinks to be disturbed by a Sink. What it is lacking is the same recreation strategy used by the Source Pipeline.

Also, in the future, the API should surface the status of each Pipeline Sink of a Stream, but for that, we must semantically separate the Stream concept from the Source Pipeline concept, which today is very liquid/mixed because that was the original design.

[patrickelectric]

Ok, that's a pretty fair point.