[BUG] Audio choppy on Bluetooth Speaker

[YozoraWolf]

Describe the bug
So I've been refining the GSR Script I use to replay (They launch on system startup), and I noticed that when saving replays they would save with some choppiness to them (latency perhaps?). Note that this is no longer a sync issue, as you will see in the video it is completely in sync. The issue is the chop.

To Reproduce

Just ran:

flatpak run --command=gpu-screen-recorder com.dec05eba.gpu_screen_recorder -w screen -fm cfr -f 60 -a "bluez_sink.C0_91_B9_DE_7C_AE.a2dp_sink.monitor" -ac aac -k hevc -c mp4 -r 60 -o /home/wolf/Videos

Expected behavior
It should sound normal, no choppiness at all.

Screenshots
If applicable, add screenshots to help explain your problem.

Replay_2024-04-13_14-06-14.mp4

Desktop (please complete the following information):

• X11 or Wayland: x11
• Desktop environment/Window Manager: GNOME3
• Distro: Linux Mint 21.3 Cinnamon
• GPU: NVIDIA Corporation GA106M [GeForce RTX 3060 Mobile / Max-Q]
• Version: flatpak (latest test flatpak provided by @dec05eba)

Additional context

wolf@wolf-mint:~/Videos/Sound$ mpv --no-config Replay_2024-04-13_14-06-14.mp4 
 (+) Video --vid=1 (*) (hevc 1920x1080 60.000fps)
 (+) Audio --aid=1 (*) (aac 2ch 48000Hz)
AO: [pulse] 48000Hz stereo 2ch float
VO: [gpu] 1920x1080 yuv420p
AV: 00:00:17 / 00:00:32 (55%) A-V:  0.296 ct:  4.253
Invalid audio PTS: 18.225604 -> 18.705146
Invalid audio PTS: 18.726479 -> 19.207104

Audio/Video desynchronisation detected! Possible reasons include too slow
hardware, temporary CPU spikes, broken drivers, and broken files. Audio
position will not match to the video (see A-V status field).

AV: 00:00:17 / 00:00:32 (55%) A-V:  1.248 ct:  4.262
Invalid audio PTS: 19.228438 -> 19.709167
AV: 00:00:17 / 00:00:32 (55%) A-V:  1.715 ct:  4.275
Invalid audio PTS: 19.730500 -> 20.210083
AV: 00:00:17 / 00:00:32 (55%) A-V:  2.181 ct:  4.288
Invalid audio PTS: 20.231417 -> 20.710729
AV: 00:00:17 / 00:00:32 (55%) A-V:  2.641 ct:  4.308
Invalid audio PTS: 20.732063 -> 21.210854
AV: 00:00:17 / 00:00:32 (55%) A-V:  3.106 ct:  4.322
Invalid audio PTS: 21.232188 -> 21.711688
AV: 00:00:17 / 00:00:32 (55%) A-V:  3.572 ct:  4.335
Invalid audio PTS: 21.733021 -> 21.989146
AV: 00:00:20 / 00:00:32 (64%) A-V:  0.000
Invalid audio PTS: 19.730500 -> 20.210083
Invalid audio PTS: 20.231417 -> 20.710729
Invalid audio PTS: 20.732063 -> 21.210854
Invalid audio PTS: 21.232188 -> 21.711688
Invalid audio PTS: 21.733021 -> 21.989146
AV: 00:00:20 / 00:00:32 (63%) A-V:  0.000
Invalid audio PTS: 19.730500 -> 20.210083
Invalid audio PTS: 20.231417 -> 20.710729
Invalid audio PTS: 20.732063 -> 21.210854
Invalid audio PTS: 21.232188 -> 21.711688
Invalid audio PTS: 21.733021 -> 21.989146
AV: 00:00:19 / 00:00:32 (62%) A-V:  0.000
Invalid audio PTS: 18.225604 -> 18.705146
Invalid audio PTS: 18.726479 -> 19.207104
Invalid audio PTS: 19.228438 -> 19.709167
AV: 00:00:19 / 00:00:32 (62%) A-V:  0.000
Invalid audio PTS: 18.225604 -> 18.705146
Invalid audio PTS: 18.726479 -> 19.207104
Invalid audio PTS: 19.228438 -> 19.709167
AV: 00:00:19 / 00:00:32 (61%) A-V:  0.000
Invalid audio PTS: 18.225604 -> 18.705146
Invalid audio PTS: 18.726479 -> 19.207104
Invalid audio PTS: 19.228438 -> 19.709167
AV: 00:00:19 / 00:00:32 (60%) A-V:  0.000
Invalid audio PTS: 19.730500 -> 20.210083
Invalid audio PTS: 20.231417 -> 20.710729
Invalid audio PTS: 20.732063 -> 21.210854
Invalid audio PTS: 21.232188 -> 21.711688
Invalid audio PTS: 21.733021 -> 21.989146
AV: 00:00:19 / 00:00:32 (60%) A-V:  0.000

Audio/Video desynchronisation detected! Possible reasons include too slow
hardware, temporary CPU spikes, broken drivers, and broken files. Audio
position will not match to the video (see A-V status field).

AV: 00:00:17 / 00:00:32 (54%) A-V:  0.296 ct:  0.263
Invalid audio PTS: 18.225604 -> 18.705146

Audio/Video desynchronisation detected! Possible reasons include too slow
hardware, temporary CPU spikes, broken drivers, and broken files. Audio
position will not match to the video (see A-V status field).

AV: 00:00:17 / 00:00:32 (55%) A-V:  0.767 ct:  0.272
Invalid audio PTS: 18.726479 -> 19.207104
AV: 00:00:17 / 00:00:32 (55%) A-V:  1.235 ct:  0.285
Invalid audio PTS: 19.228438 -> 19.709167
AV: 00:00:17 / 00:00:32 (55%) A-V:  1.702 ct:  0.298
Invalid audio PTS: 19.730500 -> 20.210083
AV: 00:00:17 / 00:00:32 (55%) A-V:  2.168 ct:  0.312
Invalid audio PTS: 20.231417 -> 20.710729
AV: 00:00:17 / 00:00:32 (55%) A-V:  2.628 ct:  0.332
Invalid audio PTS: 20.732063 -> 21.210854
AV: 00:00:17 / 00:00:32 (55%) A-V:  3.093 ct:  0.345
Invalid audio PTS: 21.232188 -> 21.711688
AV: 00:00:17 / 00:00:32 (55%) A-V:  3.559 ct:  0.358
Invalid audio PTS: 21.733021 -> 21.989146
AV: 00:00:17 / 00:00:32 (55%) A-V:  0.294 ct:  2.262
Invalid audio PTS: 18.225604 -> 18.705146

Audio/Video desynchronisation detected! Possible reasons include too slow
hardware, temporary CPU spikes, broken drivers, and broken files. Audio
position will not match to the video (see A-V status field).

AV: 00:00:17 / 00:00:32 (55%) A-V:  0.765 ct:  2.270
Invalid audio PTS: 18.726479 -> 19.207104
AV: 00:00:17 / 00:00:32 (55%) A-V:  1.239 ct:  2.277
Invalid audio PTS: 19.228438 -> 19.709167
AV: 00:00:17 / 00:00:32 (55%) A-V:  1.699 ct:  2.297
Invalid audio PTS: 19.730500 -> 20.210083
AV: 00:00:17 / 00:00:32 (55%) A-V:  2.166 ct:  2.310
Invalid audio PTS: 20.231417 -> 20.710729
AV: 00:00:17 / 00:00:32 (55%) A-V:  2.632 ct:  2.323
Invalid audio PTS: 20.732063 -> 21.210854
Invalid audio PTS: 21.232188 -> 21.711688
AV: 00:00:17 / 00:00:32 (55%) A-V:  3.570 ct:  2.343
Invalid audio PTS: 21.733021 -> 21.989146
AV: 00:00:27 / 00:00:32 (87%) A-V:  1.795 ct:  6.397

Although it seems mpv --no-config seems to be throwing sync issues. VLC replays it with no real desync.

No abnormal output from gsr, just updates on fps.

• I played the video with the command mpv --no-config video.mp4 (if applicable)
• I use a laptop with an integrated GPU and a dedicated GPU

[YozoraWolf]

Update:

https://drive.google.com/file/d/1ZpyyoYZbTh6kMP98kiA_Ne4I6YNUFuUy/view?usp=sharing

(Uploaded to GDrive due to Github's file limit)

The issue fixed itself. But it always happens randomly. Which is a problem for reliability.

This is the mpv --no-config for this video

wolf@wolf-mint:~/Videos/GenshinImpact$ mpv --no-config chopfixed.mp4 
 (+) Video --vid=1 (*) (hevc 1920x1080 60.000fps)
 (+) Audio --aid=1 (*) (aac 2ch 48000Hz)
AO: [pulse] 48000Hz stereo 2ch float
VO: [gpu] 1920x1080 yuv420p
AV: 00:00:17 / 00:00:30 (57%) A-V: -0.001

Exiting... (Quit)

[dec05eba]

This is caused by the same thing as in #10 . It happens when the device doesn't produce any audio for 0.5 seconds, which is a lot since most devices produce audio in a few milliseconds. The latest flatpak version fixes something related to this but it likely doesn't fix this particular issue. I can test some things but there isn't any perfect way to deal with it.

[YozoraWolf]

This is caused by the same thing as in #10 . It happens when the device doesn't produce any audio for 0.5 seconds, which is a lot since most devices produce audio in a few milliseconds. The latest flatpak version fixes something related to this but it likely doesn't fix this particular issue. I can test some things but there isn't any perfect way to deal with it.

Agreed. It is worth noting that this does not happen with the Built-in Speakers. I do have small hunch it may have something to do with devices going to sleep when inactive... A bluetooth issue perhaps? I'm not sure if there's any real solution to be honest, but...

I'll see if I can find anything on it.

Update: Had to restart laptop, error is not present, but it could happen in any of the restart intervals. (I have a script running in the background that restarts gsr when the default sink is switched.)

[dec05eba]

I could test some things if you could manage to make your ffmpeg libraries work so you can build gpu screen recorder from source. Does reinstalling the ffmpeg libraries (libavcodec-dev, libavformat-dev, libavutil-dev, libswresample-dev, libavfilter-dev) not fix the issue you had?

[YozoraWolf]

I could test some things if you could manage to make your ffmpeg libraries work so you can build gpu screen recorder from source. Does reinstalling the ffmpeg libraries (libavcodec-dev, libavformat-dev, libavutil-dev, libswresample-dev, libavfilter-dev) not fix the issue you had?

Yeah I fully uninstalled any remnants of the previous install and am using the version from the Package Manager

ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers

I also have gpu-screen-recorder repo locally, so I can pull anytime and build. (Though I am using the flatpak version you provided)

[dec05eba]

thats the ffmpeg binary version, not the version of the libraries. You get the version of the libraries with pkg-config --modversion libavcodec for example. But does it work if you build and run it now?

[YozoraWolf]

thats the ffmpeg binary version, not the version of the libraries. You get the version of the libraries with pkg-config --modversion libavcodec for example. But does it work if you build and run it now?

61.1.101

Let me reinstall gpu-screen-recorder from source

Update: Somehow my libraries (libavcodec, libavfilter, and others) completely broke on linker while compiling and I'm searching for a solution. Will update once I get that sorted out.

[YozoraWolf]

Update

Finally had sometime to tackle the issue with ffmpeg. I'm running 6.0 now and can build with no issues.

However I came across another issue:

Selected audio sinks: bluez_sink.C0_91_B9_DE_7C_AE.a2dp_sink.monitor
Starting recording with the following arguments: -w screen -fm cfr -f 60 -a bluez_sink.C0_91_B9_DE_7C_AE.a2dp_sink.monitor -ac aac -k h264 -c mp4 -r 60 -o /home/wolf/Videos
Warning: selected video codec h264 is not supported, trying hevc instead
Error: your gpu does not support 'h265' video codec. If you are sure that your gpu does support 'h265' video encoding and you are using an AMD/Intel GPU,
  then make sure you have installed the GPU specific vaapi packages.
  It's also possible that your distro has disabled hardware accelerated video encoding for 'h265' video codec.
  This may be the case on corporate distros such as Manjaro, Fedora or OpenSUSE.
  You can test this by running 'vainfo | grep VAEntrypointEncSlice' to see if it matches any H264/HEVC profile.
  On such distros, you need to manually install mesa from source to enable H264/HEVC hardware acceleration, or use a more user friendly distro. Alternatively record with AV1 if supported by your GPU.
  You can alternatively use the flatpak version of GPU Screen Recorder (https://flathub.org/apps/com.dec05eba.gpu_screen_recorder) which bypasses system issues with patented H264/HEVC codecs.
  Make sure you have mesa-extra freedesktop runtime installed when using the flatpak (this should be the default), which can be installed with this command:
  flatpak install --system org.freedesktop.Platform.GL.default//23.08-extra

I tried running vainfo

libva info: VA-API version 1.20.0
libva error: vaGetDriverNames() failed with unknown libva error
vaInitialize failed with error code -1 (unknown libva error),exit

But I keep getting issues. (I got my drivers up to date already, tried reinstalling vinfo, but doesn't work)

If you have any other ideas, do let me know, I'll continue to see if I can tackle that tomorrow.

[dec05eba]

You are likely missing the gpu specific vaapi driver package. For intel on mint it should be either intel-vaapi-driver or intel-media-va-driver (try installing both). But since you have ffmpeg 6.0 did you install it from source? in that case you need to build it with vaapi support.

[YozoraWolf]

You are likely missing the gpu specific vaapi driver package. For intel on mint it should be either intel-vaapi-driver or intel-media-va-driver (try installing both). But since you have ffmpeg 6.0 did you install it from source? in that case you need to build it with vaapi support.

Built ffmpeg 6.0 with VAAPI support:

wolf@wolf-mint:~$ ffmpeg
ffmpeg version 6.0 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --enable-vaapi --enable-vdpau
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
Hyper fast Audio and Video encoder

Yet for some reason it only supports nvdec? (Which is weird because I could use nvenc before)

wolf@wolf-mint:~$ vainfo
libva info: VA-API version 1.22.0
libva error: vaGetDriverNames() failed with unknown libva error
libva info: User environment variable requested driver 'nvidia'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.22 (libva 2.12.0)
vainfo: Driver version: VA-API NVDEC driver [egl backend]
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileVP8Version0_3          :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain12             :	VAEntrypointVLD

Still looking into this

[dec05eba]

oh you may have the LIBVA_DRIVER_NAME environment variable set to nvidia which you shouldn't do when running vainfo. It should use your intel device. but what happens if you run gpu screen recorder now? does it give the same error? gpu screen recorder should automatically use the correct vaapi device, ignoring the nvidia device.

[YozoraWolf]

oh you may have the LIBVA_DRIVER_NAME environment variable set to nvidia which you shouldn't do when running vainfo. It should use your intel device. but what happens if you run gpu screen recorder now? does it give the same error? gpu screen recorder should automatically use the correct vaapi device, ignoring the nvidia device.

You are absolutely right, I did set that env variable.

But I'm not running intel, mine's an AMD though.

wolf@wolf-mint:~/.gsr$ vainfo
libva info: VA-API version 1.22.0
libva error: vaGetDriverNames() failed with unknown libva error
libva info: User environment variable requested driver 'nvidia'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.22 (libva 2.12.0)
vainfo: Driver version: VA-API NVDEC driver [egl backend]
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileVP8Version0_3          :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain12             :	VAEntrypointVLD

wolf@wolf-mint:~/.gsr$ ./gsr-start-replay.sh  // This still runs gpu-screen-recorder in the back
Selected audio sinks: bluez_sink.C0_91_B9_DE_7C_AE.a2dp_sink.monitor
Starting recording with the following arguments: -w screen -fm cfr -f 60 -a bluez_sink.C0_91_B9_DE_7C_AE.a2dp_sink.monitor -ac aac -k h264 -c mp4 -r 60 -o /home/wolf/Videos
Warning: selected video codec h264 is not supported, trying hevc instead
Error: your gpu does not support 'h265' video codec. If you are sure that your gpu does support 'h265' video encoding and you are using an AMD/Intel GPU,
  then make sure you have installed the GPU specific vaapi packages.
  It's also possible that your distro has disabled hardware accelerated video encoding for 'h265' video codec.
  This may be the case on corporate distros such as Manjaro, Fedora or OpenSUSE.
  You can test this by running 'vainfo | grep VAEntrypointEncSlice' to see if it matches any H264/HEVC profile.
  On such distros, you need to manually install mesa from source to enable H264/HEVC hardware acceleration, or use a more user friendly distro. Alternatively record with AV1 if supported by your GPU.
  You can alternatively use the flatpak version of GPU Screen Recorder (https://flathub.org/apps/com.dec05eba.gpu_screen_recorder) which bypasses system issues with patented H264/HEVC codecs.
  Make sure you have mesa-extra freedesktop runtime installed when using the flatpak (this should be the default), which can be installed with this command:

And yeah, I get the same issues, I heard that you'd have to build ffmpeg enabling nvenc, so I'm working on that too as we speak!

[dec05eba]

Oh, amd. For amd you should install mesa-va-drivers. im not sure if you are using nvenc, what does it say when you run gpu screen recorder flatpak? in the title it should mention if its running on your amd igpu or nvidia gpu.