[LiveAudioMixer] Can't work with pauses in RTP streams #1034
Description
Issue Summary
When using LiveAudioMixer if one of the streams has pauses (periods of silence or no audio, mic muting, call hold, switching devices/network etc etc), the mixer fails to handle these pauses correctly. This results in ignoring streams after a pause or overfilling buffer with silence data in case of Opus DTX.
To demonstrate the issue, i put IO.inspect({pts, end_pts, queue_ts}, label: "PTS/END_PTS/QUEUE_TS") in live_queue.ex:124
Case 1, pts always old:
...
PTS/END_PTS/QUEUE_TS: {1880000000, 1900000000, 1880000000}
PTS/END_PTS/QUEUE_TS: {1900000000, 1920000000, 1900000000}
PTS/END_PTS/QUEUE_TS: {1920000000, 1940000000, 1920000000}
// Pause
// Take some time
// Resume
PTS/END_PTS/QUEUE_TS: {1940000000, 1960000000, 3460000000}
PTS/END_PTS/QUEUE_TS: {1960000000, 1980000000, 3480000000}
PTS/END_PTS/QUEUE_TS: {1980000000, 2000000000, 3480000000}
...
Here we can see that after pause QUEUE_TS is much higher than PTS, so mixer ignores these buffers, and will never mix audio from this stream again.
Case 2, pts fresh after pause (Opus DTX case):
...
PTS/END_PTS/QUEUE_TS: {1880000000, 1900000000, 1880000000}
PTS/END_PTS/QUEUE_TS: {1900000000, 1920000000, 1900000000}
PTS/END_PTS/QUEUE_TS: {1920000000, 1940000000, 1920000000}
// Silence
// Take some time
// Resume
PTS/END_PTS/QUEUE_TS: {3460000000, 3480000000, 3460000000}
PTS/END_PTS/QUEUE_TS: {3480000000, 3500000000, 3480000000}
PTS/END_PTS/QUEUE_TS: {3500000000, 3520000000, 3500000000}
...
Here we can see that after pause PTS is much higher than QUEUE_TS, and this code:
silence_duration = pts - queue_ts
silence = RawAudio.silence(stream_format, silence_duration)
will add a lot of silence buffers to the queue, that causes overflowing toilet_capacity of next element, or just unnecessary delay in audio if toilet_capacity is big enough.