HDR capture completely skewed in certain games

[arazadaz]

Describe the bug
Before anything, thanks for your effort on this project, but in any case, attempting to capture HDR content from certain games outputs wrong colors/saturation, wrong exposure, wrong contrast, and wrong gamma. Basically everything displays wrong when trying to capture HDR content in certain games. It's rather hit or miss. I have a few HDR-enabled games, and so far it only seems to work properly in Halo Infinite. I've also tried setting OBS_VKCAPTURE_COLOR_SPACE=2, but unfortunately that didn't fix anything. I'll use the game Hunt Showdown to demonstrate the issue:

How it should look:

How it looks with HDR-enabled game-capture: (Not the result of color correction)

The over-exposure is especially problematic.

System (please complete the following information):

• OS: Cachy-OS(Arch)
• GPU: 7800 XT
• Mesa version: 24.3.1-6
• obs-vkcapture version: 1.5.1-1

OBS log:

debug: Found portal inhibitor
debug: Attempted path: /usr/bin/../share/obs/obs-studio/locale/en-US.ini
debug: Attempted path: /usr/bin/../share/obs/obs-studio/locale.ini
debug: Attempted path: /usr/bin/../share/obs/obs-studio/themes
debug: Attempted path: /usr/bin/../share/obs/obs-studio/themes/
warning: ReadOne on org.freedesktop.portal.Settings returned an invalid reply
info: Platform: Wayland
info: CPU Name: AMD Ryzen 7 7800X3D 8-Core Processor
info: CPU Speed: 4758.231MHz
info: Physical Cores: 8, Logical Cores: 16
info: Physical Memory: 31165MB Total, 4259MB Free
info: Kernel Version: Linux 6.12.5-2-cachyos
info: Distribution: "CachyOS Linux" Unknown
info: Desktop Environment: KDE (KDE)
info: Session Type: wayland
info: Qt Version: 6.8.1 (runtime), 6.8.1 (compiled)
info: Portable mode: false
info: OBS 31.0.0-1.1 (linux)
info: ---------------------------------
info: ---------------------------------
info: audio settings reset:
        samples per sec: 48000
        speakers:        2
        max buffering:   960 milliseconds
        buffering type:  dynamically increasing
info: ---------------------------------
info: Initializing OpenGL...
info: Using EGL/Wayland
info: Initialized EGL 1.5
info: Loading up OpenGL on adapter AMD AMD Radeon RX 7800 XT (radeonsi, navi32, LLVM 18.1.8, DRM 3.59, 6.12.5-2-cachyos)
info: OpenGL loaded successfully, version 4.6 (Core Profile) Mesa 24.3.1-cachyos1.6, shading language 4.60
info: ---------------------------------
info: video settings reset:
        base resolution:   2560x1440
        output resolution: 1920x1080
        downscale filter:  Bicubic
        fps:               60/1
        format:            NV12
        YUV mode:          Rec. 709/Partial
info: NV12 texture support enabled
info: P010 texture support not available
info: Audio monitoring device:
        name: Default
        id: default
info: ---------------------------------
warning: Failed to load 'en-US' text for module: 'decklink-captions.so'
warning: Failed to load 'en-US' text for module: 'decklink-output-ui.so'
libDeckLinkAPI.so: cannot open shared object file: No such file or directory
warning: A DeckLink iterator could not be created.  The DeckLink drivers may not be installed
warning: Failed to initialize module 'decklink.so'
info: [pipewire] Available capture sources:
info: [pipewire]     - Monitor source
info: [pipewire]     - Window source
info: [linux-vkcapture] plugin loaded successfully (version 1.5.1)
info: [linux-vkcapture] Client 1 connected (pid=35041)
info: VAAPI: API version 1.22
info: FFmpeg VAAPI H264 encoding supported
info: FFmpeg VAAPI AV1 encoding supported
info: FFmpeg VAAPI HEVC encoding supported
error: os_dlopen(libnvidia-encode.so.1->libnvidia-encode.so.1): libnvidia-encode.so.1: cannot open shared object file: No such file or directory

info: NVENC not supported
warning: Failed to initialize module 'obs-nvenc.so'
info: ---------------------------------
info:   Loaded Modules:
info:     text-freetype2.so
info:     rtmp-services.so
info:     obs-x264.so
info:     obs-webrtc.so
info:     obs-vst.so
info:     obs-transitions.so
info:     obs-qsv11.so
info:     obs-outputs.so
info:     obs-libfdk.so
info:     obs-filters.so
info:     obs-ffmpeg.so
info:     linux-vkcapture.so
info:     linux-v4l2.so
info:     linux-pulseaudio.so
info:     linux-pipewire.so
info:     linux-jack.so
info:     linux-capture.so
info:     linux-alsa.so
info:     image-source.so
info:     frontend-tools.so
info:     decklink-output-ui.so
info:     decklink-captions.so
info: ---------------------------------
info: ---------------------------------
info: Available Encoders:
info:   Video Encoders:
info:   - ffmpeg_svt_av1 (SVT-AV1)
info:   - ffmpeg_aom_av1 (AOM AV1)
info:   - ffmpeg_vaapi_tex (FFmpeg VAAPI H.264)
info:   - av1_ffmpeg_vaapi_tex (FFmpeg VAAPI AV1)
info:   - hevc_ffmpeg_vaapi_tex (FFmpeg VAAPI HEVC)
info:   - obs_x264 (x264)
info:   Audio Encoders:
info:   - ffmpeg_aac (FFmpeg AAC)
info:   - ffmpeg_opus (FFmpeg Opus)
info:   - ffmpeg_pcm_s16le (FFmpeg PCM (16-bit))
info:   - ffmpeg_pcm_s24le (FFmpeg PCM (24-bit))
info:   - ffmpeg_pcm_f32le (FFmpeg PCM (32-bit float))
info:   - ffmpeg_alac (FFmpeg ALAC (24-bit))
info:   - ffmpeg_flac (FFmpeg FLAC (16-bit))
info:   - libfdk_aac (libfdk AAC)
info: ==== Startup complete ===============================================
info: All scene data cleared
info: ------------------------------------------------
info: pulse-input: Server name: 'PulseAudio (on PipeWire 1.2.7) 15.0.0'
info: pulse-input: Audio format: s24le, 48000 Hz, 2 channels
info: pulse-input: Sample format s24le not supported by OBS,using float32le instead for recording
info: pulse-input: Started recording from 'alsa_output.usb-MediaTek_Inc_JLab_Epic_Lab_Edition_Dongle_0000000000000000-00.analog-stereo.monitor' (default)
info: [Loaded global audio device]: 'Desktop Audio'
info: pulse-input: Server name: 'PulseAudio (on PipeWire 1.2.7) 15.0.0'
info: pulse-input: Audio format: s16le, 48000 Hz, 2 channels
info: pulse-input: Started recording from 'alsa_input.usb-Generic_Blue_Microphones_LT_2409071214579D0100D7_111000-00.analog-stereo' (default)
info: [Loaded global audio device]: 'Mic/Aux'
info:     - filter: 'Noise Suppression 2' (noise_suppress_filter_v2)
info:     - filter: 'Noise Gate' (noise_gate_filter)
info:     - filter: 'Gain' (gain_filter)
info:     - filter: 'Compressor' (compressor_filter)
info: PipeWire initialized
error: ext_screencopy_manager_v1 not available
info: Switched to scene 'Scene'
info: ------------------------------------------------
info: Loaded scenes:
info: - scene 'Scene':
info:     - source: 'Game Capture' (vkcapture-source)
info:         - filter: 'Color Correction' (color_filter_v2)
info:     - source: 'Screen Capture (PipeWire)' (pipewire-screen-capture-source)
info:         - filter: 'Color Correction' (color_filter_v2)
info: ------------------------------------------------
info: [pipewire] Screencast session created
info: [pipewire] Asking for monitor and window
info: adding 21 milliseconds of audio buffering, total audio buffering is now 21 milliseconds (source: Desktop Audio)

info: [linux-vkcapture] Creating texture from dmabuf 2560x1440 modifier:144115188621556484
info: [linux-vkcapture]  [0] fd:39 stride:20480 offset:0
info: [linux-vkcapture]  [1] fd:40 stride:2560 offset:30146560
warning: [pipewire] Failed to start screencast, denied or cancelled by user
info: ==== Shutting down ==================================================
info: pulse-input: Stopped recording from 'alsa_output.usb-MediaTek_Inc_JLab_Epic_Lab_Edition_Dongle_0000000000000000-00.analog-stereo.monitor'
info: pulse-input: Got 19677 packets with 23612400 frames
info: pulse-input: Stopped recording from 'alsa_input.usb-Generic_Blue_Microphones_LT_2409071214579D0100D7_111000-00.analog-stereo'
info: pulse-input: Got 0 packets with 0 frames
info: [linux-vkcapture] destroy
info: All scene data cleared
info: ------------------------------------------------
info: [linux-vkcapture] Client 1 disconnected
info: [linux-vkcapture] plugin unloaded
info: [Scripting] Total detached callbacks: 0
info: Freeing OBS context data
info: == Profiler Results =============================
info: run_program_init: 516.291 ms
info:  ┣OBSApp::AppInit: 10.819 ms
info:  ┃ ┗OBSApp::InitLocale: 2.29 ms
info:  ┗OBSApp::OBSInit: 425.417 ms
info:    ┣obs_startup: 1.593 ms
info:    ┗OBSBasic::OBSInit: 402.21 ms
info:      ┣OBSBasic::InitBasicConfig: 0.431 ms
info:      ┣OBSBasic::ResetAudio: 0.151 ms
info:      ┣OBSBasic::ResetVideo: 74.379 ms
info:      ┃ ┗obs_init_graphics: 56.102 ms
info:      ┃   ┗shader compilation: 39.227 ms
info:      ┣OBSBasic::InitOBSCallbacks: 0.004 ms
info:      ┣OBSBasic::InitHotkeys: 0.019 ms
info:      ┣obs_load_all_modules2: 260.357 ms
info:      ┃ ┣obs_init_module(decklink-captions.so): 0 ms
info:      ┃ ┣obs_init_module(decklink-output-ui.so): 0 ms
info:      ┃ ┣obs_init_module(decklink.so): 0.064 ms
info:      ┃ ┣obs_init_module(frontend-tools.so): 48.493 ms
info:      ┃ ┣obs_init_module(image-source.so): 0.006 ms
info:      ┃ ┣obs_init_module(linux-alsa.so): 0.001 ms
info:      ┃ ┣obs_init_module(linux-capture.so): 0 ms
info:      ┃ ┣obs_init_module(linux-jack.so): 0.002 ms
info:      ┃ ┣obs_init_module(linux-pipewire.so): 8.923 ms
info:      ┃ ┣obs_init_module(linux-pulseaudio.so): 0.005 ms
info:      ┃ ┣obs_init_module(linux-v4l2.so): 6.117 ms
info:      ┃ ┣obs_init_module(linux-vkcapture.so): 0.069 ms
info:      ┃ ┣obs_init_module(obs-ffmpeg.so): 13.943 ms
info:      ┃ ┣obs_init_module(obs-filters.so): 0.02 ms
info:      ┃ ┣obs_init_module(obs-libfdk.so): 0.001 ms
info:      ┃ ┣obs_init_module(obs-nvenc.so): 0.065 ms
info:      ┃ ┃ ┗nvenc_check: 0.052 ms
info:      ┃ ┣obs_init_module(obs-outputs.so): 0.003 ms
info:      ┃ ┣obs_init_module(obs-qsv11.so): 12.938 ms
info:      ┃ ┣obs_init_module(obs-transitions.so): 0.004 ms
info:      ┃ ┣obs_init_module(obs-vst.so): 0.003 ms
info:      ┃ ┣obs_init_module(obs-webrtc.so): 0.002 ms
info:      ┃ ┣obs_init_module(obs-x264.so): 0.004 ms
info:      ┃ ┣obs_init_module(rtmp-services.so): 0.625 ms
info:      ┃ ┗obs_init_module(text-freetype2.so): 0.005 ms
info:      ┣OBSBasic::InitService: 2.057 ms
info:      ┣OBSBasic::ResetOutputs: 6.888 ms
info:      ┣OBSBasic::CreateHotkeys: 0.021 ms
info:      ┣OBSBasic::InitPrimitives: 0.032 ms
info:      ┗OBSBasic::Load: 31.004 ms
info: obs_hotkey_thread(25 ms): min=0 ms, median=0 ms, max=0.124 ms, 99th percentile=0.001 ms, 100% below 25 ms
info: audio_thread(Audio): min=0.005 ms, median=0.019 ms, max=5.231 ms, 99th percentile=2.061 ms
info: obs_graphics_thread(16.6667 ms): min=0.06 ms, median=0.297 ms, max=37.477 ms, 99th percentile=4.285 ms, 99.9763% below 16.667 ms
info:  ┣tick_sources: min=0 ms, median=0.006 ms, max=20.813 ms, 99th percentile=0.019 ms
info:  ┣output_frame: min=0.049 ms, median=0.165 ms, max=8.581 ms, 99th percentile=4.082 ms
info:  ┃ ┗gs_context(video->graphics): min=0.049 ms, median=0.165 ms, max=8.58 ms, 99th percentile=4.082 ms
info:  ┃   ┣render_video: min=0.007 ms, median=0.047 ms, max=2.75 ms, 99th percentile=0.089 ms
info:  ┃   ┃ ┗render_main_texture: min=0.005 ms, median=0.045 ms, max=2.749 ms, 99th percentile=0.085 ms
info:  ┃   ┗gs_flush: min=0.001 ms, median=0.102 ms, max=8.497 ms, 99th percentile=4.021 ms
info:  ┗render_displays: min=0.001 ms, median=0.084 ms, max=37.373 ms, 99th percentile=0.4 ms
info: =================================================
info: == Profiler Time Between Calls ==================
info: obs_hotkey_thread(25 ms): min=25.006 ms, median=25.058 ms, max=26.658 ms, 99.8422% within ±2% of 25 ms (0% lower, 0.157785% higher)
info: obs_graphics_thread(16.6667 ms): min=12.13 ms, median=16.667 ms, max=37.48 ms, 99.3567% within ±2% of 16.667 ms (0.321663% lower, 0.321663% higher)
info: =================================================
info: Number of memory leaks: 0

Game log: Lots here
Hunt Showdown

[fililip]

I can confirm the issue with Quake II RTX.

Capturing a Rec.2020 + PQ video from mpv (waylandvk, Vulkan video decode) results in identical color representation, while games like Q2RTX look 'overclipped' as seen on the screenshots. No color space value for OBS_VKCAPTURE_COLOR_SPACE seems to be working, with 0 or 1 it looks better than with 2 or default but the colors are still unacceptable.

OBS is set to P010 and Rec.2100 (PQ).

Update: It appears that Q2RTX uses VK_COLOR_SPACE_SRGB_NONLINEAR_KHR (while the aforementioned mpv example uses VK_COLOR_SPACE_HDR10_ST2084_EXT) and yet it does HDR by (probably) simply boosting the max luma parameter in VkHdrMetadataEXT. Then, when OBS captures that data, perhaps it incorrectly interprets it as Rec.2100?

@nowrep Is this actionable on vkcapture's end or is there more that needs to be done in OBS for instance?

[adromel]

Can confirm the issue as well with Star Citizen (using wine wayland).
The reported swapchain is as follows :

info:    Format:       VK_FORMAT_R16G16B16A16_SFLOAT
info:    Color space:  VK_COLOR_SPACE_EXTENDED_SRGB_LINEAR_EXT

But no matter the OBS_VKCAPTURE_COLOR_SPACE env, OBS tonemapping tentatives, I can't seem to preview/output decent colors.

Is this addressable by obs-vkcapture? Otherwise do we have an idea on what end the blocker resides?

@fililip have you found any fix or answer related to this ?

[fililip]

Unfortunately, I haven't been able to find any workaround or fix for this issue.

[adromel]

If devs want us to help debug this, or give us some insights whether it's only compositor or wayland related, that'd be grand!

[nowrep]

Anything other than VK_COLOR_SPACE_HDR10_ST2084_EXT probably doesn't work at all.

But also currently 10000 white level for PQ is used, but this needs to use the value from HDR metadata.

[adromel]

Thanks for your prompt answer.

Correct me if I'm wrong, but to my narrow understanding, there's a mismatch between VK_COLOR_SPACE_EXTENDED_SRGB_LINEAR_EXT and every single OBS target color spaces (GS_CS_SRGB, GS_CS_SRGB_16F, GS_CS_709_EXTENDED and GS_CS_709_SCRGB -that can be set via OBS_VKCAPTURE_COLOR_SPACE), right?
If this is the case, as per this OBS wiki page, shouldn't the Game Capture process the color space difference?

Would this be subject to be a feature looking forward, as I'm guessing not all games/programs will use Rec2100? Or does this solely rely on OBS? Or are we doomed to be left unable to record HDR properly for such swapchains?

[nowrep]

VK_COLOR_SPACE_HDR10_ST2084_EXT uses PQ transfer function, so on OBS plugin side it needs DrawPQ effect. This works fine currently.

VK_COLOR_SPACE_EXTENDED_SRGB_LINEAR_EXT uses linear transfer function, so OBS needs to interpret the values as is. This seems to be broken, as you reported. GS_CS_SRGB_16F should be correct here (as long as the Vk format is float16).

I think what is also missing is the white level from HDR metadata.

I will take a look at it over this weekend, hopefully.

[adromel]

That's great news!
Feel free to drop a patch if you need me to test this out

[adromel]

I took a quick look at how OBS handles it for win, and regarding the color space it seems to be using GS_CS_709_SCRGB if GS_RGBA16F color format is detected, which is the case on my use case:
https://github.com/obsproject/obs-studio/blob/e73662f6c14b1510861e6a6bdeadde36bdfdc8a7/plugins/win-capture/game-capture.c#L1878-L1883

Regardless, wouldn't all these conversions/transfer functions processings be applicable to this project, or is this more complex than that concerning multipliers/white level/metadata?
https://github.com/obsproject/obs-studio/blob/e73662f6c14b1510861e6a6bdeadde36bdfdc8a7/plugins/win-capture/game-capture.c#L1958-L2034

[nowrep]

You can try #280 Seems better, but probably not quite there yet?

[adromel]

It's better for sure! Colors are way more accurate, what goes on my side is that the new preview image and output are dimmer compared to the game.

Edit: confirmed with QIIRTX as well, while a record of a mpv Rec2020 PQ playback is accurate in output (but not in preview, same dim issue)

[fililip]

You can try #280 Seems better, but probably not quite there yet?

Thanks for fixing it! This branch seems to address the issue in q2rtx for me too, the recordings look neat now.

[fililip]

Oh, actually, this PR seems to break SDR content now? vkcube looks much brighter in OBS, which it shouldn't be:

It also seems to do that in recordings.