Video / Audio playback disruption

[DougShuffield]

I am using OMXplayer on a Pi 3B+ running Rasbian stretch to display a PC HDMI output broadcast via UDP multicast from an HDMI extender in my system. I am having an issue where the video / audio playback gets disrupted when I start a UE4 based game on the PC. It seems that the dts / pts gets reset during the transition when the UE4 game starts which then causes OMXplayer to not be able to play the audio and sometimes not be able to play the video. Here is my command:

omxplayer --dbus_name "org.mpris.MediaPlayer2.omxplayer.gameplay" --with-info --live --threshold 0.5 --fps 30 --crop 320,0,1600,1080 --win 640,0,1920,1080 --layer 10 $gpIP

Here is the log when the issue occurs somewhere between 21:28:20 and 21:28:23:

21:28:20 T:18446744072662452240 DEBUG: Normal M:189392402 (A:188754000 V:188610000) P:0 A:-0.64 V:-0.78/T:0.50 (1,1,0,0) A:0% V:0% (-0.61,6.34)
21:28:20 T:18446744072662452343 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:0
21:28:20 T:18446744072662452373 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:1
21:28:20 T:18446744072662452642 DEBUG: Live: 0.66 (-0.64) S:1.001 T:0.50
21:28:20 T:18446744072662453001 INFO: CDVDPlayerAudio::Decode dts:188806000 pts:188806000 size:627
21:28:20 T:18446744072662453454 INFO: CDVDPlayerAudio::Decode dts:188832000 pts:188832000 size:627
21:28:20 T:18446744072662465283 INFO: COMXAudio::ApplyVolume - Volume=1.00 (* 1.00 * 1.00)
21:28:20 T:18446744072662495931 INFO: CDVDPlayerAudio::Decode dts:188858000 pts:188858000 size:627
21:28:20 T:18446744072662496227 DEBUG: Normal M:189436515 (A:188832000 V:188610000) P:0 A:-0.60 V:-0.83/T:0.50 (1,1,0,0) A:0% V:0% (-0.66,6.34)
21:28:20 T:18446744072662496323 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:0
21:28:20 T:18446744072662496350 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:1
21:28:20 T:18446744072662496863 DEBUG: Live: 0.65 (-0.60) S:1.001 T:0.50
21:28:21 T:18446744072662931688 INFO: CDVDPlayerAudio::Decode dts:188676000 pts:188676000 size:626
21:28:21 T:18446744072662932096 ERROR: COMXAudioCodecOMX::GetData Unexpected change of size (9216->4608)
21:28:21 T:18446744072662935281 INFO: COMXAudio::ApplyVolume - Volume=1.00 (* 1.00 * 1.00)
21:28:21 T:18446744072662935571 DEBUG: Normal M:189874233 (A:188676000 V:188610000) P:0 A:-1.20 V:-1.26/T:0.50 (1,1,0,0) A:0% V:0% (-1.02,6.34)
21:28:21 T:18446744072662935716 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:0
21:28:21 T:18446744072662935777 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:1
21:28:21 T:18446744072662936236 DEBUG: Live: 0.63 (-1.20) S:1.001 T:0.50
21:28:21 T:18446744072662936573 INFO: CDVDPlayerAudio::Decode dts:188702000 pts:188702000 size:627
21:28:21 T:18446744072662937599 INFO: CDVDPlayerAudio::Decode dts:188728000 pts:188728000 size:627
21:28:21 T:18446744072662938028 INFO: CDVDPlayerAudio::Decode dts:188754000 pts:188754000 size:627
21:28:21 T:18446744072662959971 INFO: CDVDPlayerAudio::Decode dts:188780000 pts:188780000 size:627
21:28:21 T:18446744072662960429 DEBUG: Normal M:189900978 (A:188754000 V:188610000) P:0 A:-1.15 V:-1.29/T:0.50 (1,1,0,0) A:0% V:0% (-1.04,6.34)
21:28:21 T:18446744072662960563 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:0
21:28:21 T:18446744072662960622 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:1
21:28:21 T:18446744072662960971 DEBUG: Live: 0.61 (-1.15) S:1.001 T:0.50
21:28:21 T:18446744072662961218 INFO: CDVDPlayerAudio::Decode dts:188806000 pts:188806000 size:627
21:28:21 T:18446744072662961673 INFO: CDVDPlayerAudio::Decode dts:188832000 pts:188832000 size:627
21:28:21 T:18446744072663051943 INFO: CDVDPlayerAudio::Decode dts:188858000 pts:188858000 size:627
21:28:21 T:18446744072663052406 DEBUG: Normal M:189993083 (A:188832000 V:188610000) P:0 A:-1.16 V:-1.38/T:0.50 (1,1,0,0) A:0% V:0% (-1.14,6.34)
21:28:21 T:18446744072663052542 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:0
21:28:21 T:18446744072663052605 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:1
21:28:21 T:18446744072663052954 DEBUG: Live: 0.59 (-1.16) S:1.001 T:0.50
21:28:23 T:18446744072664819247 INFO: CDVDPlayerAudio::Decode dts:330000 pts:330000 size:627
21:28:23 T:18446744072664819580 DEBUG: Normal M:191762084 (A:188858000 V:188610000) P:0 A:-2.90 V:-3.15/T:0.50 (1,1,0,0) A:0% V:0% (-2.90,6.34)
21:28:23 T:18446744072664819716 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:0
21:28:23 T:18446744072664819777 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:1
21:28:23 T:18446744072664820231 DEBUG: Live: 0.56 (-2.90) S:1.001 T:0.50
21:28:23 T:18446744072664820367 ERROR: COMXAudioCodecOMX::GetData Unexpected change of size (4608->9216)
21:28:23 T:18446744072664822839 INFO: COMXAudio::ApplyVolume - Volume=1.00 (* 1.00 * 1.00)
21:28:23 T:18446744072664822988 INFO: CDVDPlayerAudio::Decode dts:356000 pts:356000 size:627
21:28:23 T:18446744072664824004 INFO: CDVDPlayerAudio::Decode dts:382000 pts:382000 size:627
21:28:23 T:18446744072664824759 INFO: CDVDPlayerAudio::Decode dts:408000 pts:408000 size:627
21:28:23 T:18446744072664848176 INFO: CDVDPlayerAudio::Decode dts:435000 pts:435000 size:627
21:28:23 T:18446744072664848467 DEBUG: Normal M:191791151 (A:435000 V:188610000) P:0 A:-191.36 V:-3.18/T:0.50 (1,1,0,0) A:0% V:0% (-2.93,6.34)
21:28:23 T:18446744072664848583 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:0
21:28:23 T:18446744072664848643 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:1
21:28:23 T:18446744072664849201 DEBUG: Live: -1.36 (-191.36) S:0.990 T:0.50
21:28:23 T:18446744072664921256 INFO: CDVDPlayerAudio::Decode dts:461000 pts:461000 size:627
21:28:23 T:18446744072664922742 DEBUG: Normal M:191864631 (A:461000 V:188610000) P:0 A:-191.40 V:-3.25/T:0.50 (1,1,0,0) A:0% V:0% (-3.01,6.34)
21:28:23 T:18446744072664922953 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:0
21:28:23 T:18446744072664923017 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:1
21:28:23 T:18446744072664923485 DEBUG: Live: -3.26 (-191.40) S:0.990 T:0.50
21:28:23 T:18446744072664923787 INFO: CDVDPlayerAudio::Decode dts:487000 pts:487000 size:627
21:28:23 T:18446744072664950956 INFO: CDVDPlayerAudio::Decode dts:513000 pts:513000 size:627
21:28:23 T:18446744072664954353 INFO: COMXAudio::ApplyVolume - Volume=1.00 (* 1.00 * 1.00)
21:28:23 T:18446744072664954579 DEBUG: Normal M:191892783 (A:513000 V:188610000) P:0 A:-191.38 V:-3.28/T:0.50 (1,1,0,0) A:0% V:0% (-3.04,6.34)
21:28:23 T:18446744072664954658 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:0
21:28:23 T:18446744072664954684 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:1
21:28:23 T:18446744072664954924 DEBUG: Live: -5.14 (-191.38) S:0.990 T:0.50
21:28:23 T:18446744072664955291 INFO: CDVDPlayerAudio::Decode dts:539000 pts:539000 size:627
21:28:23 T:18446744072665010146 INFO: CDVDPlayerAudio::Decode dts:565000 pts:565000 size:627
21:28:23 T:18446744072665010768 DEBUG: Normal M:191951381 (A:565000 V:188610000) P:0 A:-191.39 V:-3.34/T:0.50 (1,1,0,0) A:0% V:0% (-3.09,6.34)
21:28:23 T:18446744072665010866 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:0
21:28:23 T:18446744072665010895 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:1
21:28:23 T:18446744072665011197 DEBUG: Live: -7.01 (-191.39) S:0.990 T:0.50

Any help resolving this would be much appreciated. Let me know if there is any thing else I can provide to help troubleshoot.

Doug
omxplayer.log

[popcornmix]

omxplayer is being deprecated in favour of vlc. Are you able to try with vlc?

[DougShuffield]

I saw that in another thread and am trying it out as we speak.

Currently looking for the command line options to specify dispmanx layer, window size, and window position with cvlc...

In my particular instance I am using two instances of omxplayer for a split screen camera / gameply and using kivy for an overlay. I am a little concerned that vlc may be too resource heavy to do it on the 3B+, but I will try it out and report back. My only other thought was to compile omxplayer with the latest ffmpeg to see if that might have more robust handling of the stream transition.

[DougShuffield]

Any help in converting these calls to omxplayer into their equivalent calls to cvlc would be much appreciated. The info on realtime crop, window position, and dispmanx layer is lacking for cvlc. I continue to look, but not very clear.

Camera left hand split

omxplayer -n -1 --threshold 0.01 --fps 30 --crop 547,0,1094,922 --win 0,0,640,1080 --layer 20 $camrxIP &

Gameplay right hand split

omxplayer --live --threshold 0.5 --fps 30 --crop 320,0,1600,1080 --win 640,0,1920,1080 --layer 10 $gpIP &

[popcornmix]

@jc-kynesim can you advise?
I believe we can use window_size and window_x/window_y but google isn't too helpful.
Did you add display_layer and fullscreen?

[jc-kynesim]

Current vlc certainly allows positioning & sizing of the video playback window in "fullscreen". The --mmal-layer and -f options should do the trick for layer & fullscreen, crop may be unimplemented unless it turns up automagically (I'm sure I haven't tested it). A possible problem as it stands is that I think that the video layer disables all layers beneath it and there is no option for stopping that.

[popcornmix]

vlc certainly allows positioning & sizing of the video playback window in "fullscreen"

Do you know what command line options are for this (running cvlc)?
--mmallayer and -f are accepted. --window_size and --window_x/--window_y don't seem to be.

[KenT2]

The existence of 'fullscreen commands is really great.

A possible problem as it stands is that I think that the video layer disables all layers beneath it and there is no option for stopping that.

Is it a case of adding an option or is this disabling a permanent drawback. I would like to move Pi Presents from omxplayer to cvlc. In PP I overlay omx videos over x windows text and images using Tkinter but this would stop all of that.

Are all the 'fullscreen' options documented anywhere. Is the code viewable?

[popcornmix]

You should be able to get the code from apt source.
I think this is it:
https://github.com/RPi-Distro/vlc/tree/buster-rpt/debian/patches
(upstream vlc in top level directory plus patches in this directory)

Adding an option to not disable the layers beneath the video should be straightforward in a future update.

[jc-kynesim]

If you are brave then vlc -H will list everything along with at least some description. Beware of the dusty corners of the --mmal-xxx options - some of those were invented for debug purposes only and may have rotted.

[popcornmix]

@jc-kynesim I've tried:

cvlc -f --video-x 64 --video-y 64 --width 256 --height 320 <video>

but it has no obvious effect. Do you know how to alter size and positioning of the area video is rendered to?

[jc-kynesim]

I'll check tomorrow - I've run out of time right now. I know I did manage to move stuff around when I was testing but I've forgotten - sorry.

[DougShuffield]

Thanks for the efforts guys. I will continue to look into the cvlc path as well.
I am also going to try out the omxplayer pull request: Added option --force-fps #678 to see if that can possibly help with the reset of the dts / pts.

Doug

[jc-kynesim]

OK - looks like I am misremembering doing positioning in fullscreen. It clearly doesn't work currently. I'll try and make sure it gets into the next patch.

[DougShuffield]

No dice with the force-fps branch as it seems the fake pts is only generated for video, not audio. When the audio pts/dts disruption occurs, still craps out the audio output and all I hear is static.

@popcornmix : Is there a fairly simple way to modify the code to detect the condition where the audio pts / dts resets like this (large negative delta between last pts and current pts) and then flush the buffer and start again with the new dts/pts. Maybe adding a call to OMXPlayerAudio::Flush() in OMXPlayerAudio::Decode of OMXPlayerAudio.cpp when the condition is met? I am investigating the code, but still not certain on the appropriate place to add, or if it will work at all...

Thanks again for all your help!

[DougShuffield]

@popcornmix Any thoughts?

What I would like to do:

1. Detect reset of PTS (current pts < last pts)
2. Disregard previous values of PTS in calculation of m_latency or reset calculation.
3. Reset or flush audio buffers (maybe not necessary?)
4. Start audio decoding and calculation of latency for --live fresh starting with new PTS

I've tried a few things, but there are many moving parts, so any insight here would be appreciated.