metal: add support for plane color pipelines

[mahkoh]

Closes #1068
Closes #1098
Closes #1000

Disabled by default. Can be enabled per device.

Disabled on AMD if a hardware cursor is active on the output.

[mahkoh]

@njdom24, @witchlliee please test this.

[witchlliee]

@njdom24, @witchlliee please test this.

To clarify, I built Jay with this PR and replaced the binary. It still shows compositing in games tho. How should I properly debug this for you, if needed?

[mahkoh]

Have you enabled the feature in the GPU pane in the control center?

[witchlliee]

Have you enabled the feature in the GPU pane in the control center?

Ah, sorry. Enabled it here now but still not working unfortunately.

[mahkoh]

Post the output of drm_info.

[hamburg3rz]

This is working well in the games that I've tried so far! seems to be no issues with the color or HDR. interestingly for Overwatch I had to try entering and exiting fullscreen a few times but it worked eventually. the visualize compositing square goes away when no cursor is on screen. however I noticed that regardless of the color pipelines setting being on or off that any games that have an in-game toggle for HDR will crash if HDR is toggled off. proton has this error:

wl_display#1: error 3: An error occurred while processing a request: Could not process a `wp_color_manager_v1#60.get_surface` request: wl_surface already has a color-management extension
835.707:0154:0174:err:waylanddrv:waylanddrv_unix_read_events Protocol error on wl_display#1 with code 3
835.707:0154:0174:err:waylanddrv:wayland_read_events_thread Failed to read events from the compositor, terminating process

I don't get this same crash with kwin and I don't believe I got it with Jay until now. let me know if I can post other info or logs

[witchlliee]

Post the output of drm_info.

drm_info.log

[mahkoh]

@hamburg3rz reproduce the crash with WAYLAND_DEBUG logging.

[mahkoh]

@witchlliee Post the WAYLAND_DEBUG output for the game.

[witchlliee]

@witchlliee Post the WAYLAND_DEBUG output for the game.

steam-230410.tar.gz

[hamburg3rz]

@hamburg3rz reproduce the crash with WAYLAND_DEBUG logging.

steam-2357570.log
here you go, sorry reupload since I think I might have uploaded from the wrong game. edit: third time's the charm this one should actually have debug on for you

[mahkoh]

@witchlliee Post the WAYLAND_DEBUG output for the game.

steam-230410.tar.gz

It doesn't set any image description so I'd expect direct scanout to work. Please restart Jay, set the log level to debug, start the game, and then send the jay log file.

[mahkoh]

@hamburg3rz reproduce the crash with WAYLAND_DEBUG logging.

steam-2357570.log here you go, sorry reupload since I think I might have uploaded from the wrong game. edit: third time's the charm this one should actually have debug on for you

[1413688.425] {overwatch.exe}  -> wp_color_manager_v1#4.create_windows_scrgb(new id wp_image_description_v1#64)
[1413688.465] {overwatch.exe}  -> wp_color_manager_v1#4.get_surface(new id wp_color_management_surface_v1#58, wl_surface#46)
[1413688.468] {overwatch.exe}  -> wp_color_management_surface_v1#58.set_image_description(wp_image_description_v1#64, 0)
[1428841.250] {mesa vk display queue}  -> wp_color_manager_v1#65.get_surface(new id wp_color_management_surface_v1#76, wl_surface#46)

So whoever creates the overwatch.exe queue, probably winewayland, creates a color management surface for wl_surface#46. Then mesa creates another.

This looks like a bug in winewayland. Mesa guards this request behind

   return colorspace != VK_COLOR_SPACE_PASS_THROUGH_EXT;

and the vulkan spec says that the vulkan driver may create a color management surface is any other color space is used.

I.e., you should inspect every place in which a vulkan swapchain is created and make sure that only VK_COLOR_SPACE_PASS_THROUGH_EXT is used if winewayland is managing the color management surface itself.

[witchlliee]

@witchlliee Post the WAYLAND_DEBUG output for the game.

steam-230410.tar.gz

It doesn't set any image description so I'd expect direct scanout to work. Please restart Jay, set the log level to debug, start the game, and then send the jay log file.

jay.log

[mahkoh]

I've added additional debug logging. Please retry the test.

[hamburg3rz]

the crashes when turning off HDR seem to be proton/winewayland related like you said. the creator of proton-em was able to identify the problem. so with that the color pipelines are working great for me, thank you

[witchlliee]

here is another log:

jay.log

So I was using proton-cachyos like @hamburg3rz and it has some issues, so I will be retesting with another more reliable build for winewayland

[mahkoh]

Are you sure you don't have direct scanout disabled for the device?

[witchlliee]

Are you sure you don't have direct scanout disabled for the device?

yes it works for re7 for example. will do some more tests

[mahkoh]

Then send logs where you first use re7 and then switch to the game that doesn't work.

[witchlliee]

Then send logs where you first use re7 and then switch to the game that doesn't work.

here:

jay.log

[witchlliee]

Okay did a re-test with another proton build and everything seems to be working, even Warframe that wasn't working at all for some reason.

[njdom24]

Attempt 2: dtn_logs_2.zip

[mahkoh]

Are you sure you are capturing /sys/kernel/debug/dri/1/amdgpu_dm_dtn_log and not /sys/kernel/debug/dri/0/amdgpu_dm_dtn_log?

[njdom24]

Positive. I ran sudo cat /sys/kernel/debug/dri/1/amdgpu_dm_dtn_log > dtn_working_pq.log pulled from my shell history, and the outputs under /sys/kernel/debug/dri/1/ match up with my connected outputs :/

[mahkoh]

I think the issue is that the kernel doesn't have logging of the color settings enabled for DCN 4.0.1 cards such as your 9070. For DCN 3.0.1 it sets

	.log_hw_state = dcn10_log_hw_state,
	.log_color_state = dcn30_log_color_state,

and dcn10_log_hw_state calls into ->log_color_state. But for DCN 4.0.1 it only sets

	.log_hw_state = dcn10_log_hw_state,

which then falls back to the generic dcn10_log_color_state which doesn't log anything that would be helpful.

Not sure what I can do. The AMD driver has bugs where it just doesn't program the hardware sometimes. On my system this occurs somewhat regularly if I switch to a different TTY and then back to Jay.

[njdom24]

Gotcha. I'm aware of this RDNA4 issue and am running kernel 7.1 with the attached patch FWIW. But if there's nothing to do on the Jay side, it's not an invasive issue anyway. I'm content waiting for it to be fixed on the driver side eventually.

[mahkoh]

I wouldn't hold my breath. That might be the issue or it might not be.

I asked claude if there was any other way to get this information and it told me two things:

• I could read the register values directly. I'm not going to bother with that but if anyone is adventurous they can probably get AI to write a small tool that does exactly this.
• It pointed me to https://www.mail-archive.com/amd-gfx@lists.freedesktop.org/msg122014.html which was posted over a year ago and would have added the required logging.

Adding color pipelines has been the most painful thing I've done in a long time. There is no solid foundation to build on since the kernel drivers are full of bugs and unspecified behavior. For example, here are two LUTs in the AMD color pipeline:

    │               ├───Color Operation 392
    │               │   └───Properties
    │               │       ├───"TYPE" (immutable): enum {1D Curve, 1D LUT, 3x4 Matrix, Multiplier, 3D LUT} = 1D LUT
    │               │       ├───"BYPASS" (atomic): range [0, 1] = 1
    │               │       ├───"NEXT" (atomic, immutable): object colorop = 399
    │               │       ├───"SIZE" (atomic, immutable): range [0, UINT32_MAX] = 4096
    │               │       ├───"LUT1D_INTERPOLATION": enum {Linear} = Linear
    │               │       └───"DATA" (atomic): blob = 0
    │               └───Color Operation 411
    │                   └───Properties
    │                       ├───"TYPE" (immutable): enum {1D Curve, 1D LUT, 3x4 Matrix, Multiplier, 3D LUT} = 1D LUT
    │                       ├───"BYPASS" (atomic): range [0, 1] = 1
    │                       ├───"NEXT" (atomic, immutable): object colorop = 0
    │                       ├───"SIZE" (atomic, immutable): range [0, UINT32_MAX] = 4096
    │                       ├───"LUT1D_INTERPOLATION": enum {Linear} = Linear
    │                       └───"DATA" (atomic): blob = 0

They look the same so they should act the same, right? No! Why would you think that? Despite claiming to have 4096 points with linear interpolation, the first LUT really only has ~200 points and will band like crazy if programmed in a way that cannot be represented in those ~200 points. So the kernel will reject such programmings, right? No! You should know better by now. It will helpfully log to syslog that this shit won't work:

amdgpu 0000:03:00.0: [drm] Losing delta precision while programming shaper LUT.

And then it will continue as if there was nothing wrong.

Then, we also have matrices which are needed to convert between color spaces. For example

    │               ├───Color Operation 382
    │               │   └───Properties
    │               │       ├───"TYPE" (immutable): enum {1D Curve, 1D LUT, 3x4 Matrix, Multiplier, 3D LUT} = 3x4 Matrix
    │               │       ├───"BYPASS" (atomic): range [0, 1] = 1
    │               │       ├───"NEXT" (atomic, immutable): object colorop = 387
    │               │       └───"DATA" (atomic): blob = 0

So we can just store out affine transformations in that matrix, right? No! If you store values smaller than -3 or larger than 3 in such matrices, the driver will silently clamp them to -3 or 3. And then your image will look fucked, of course.

So I had to hard code that the first LUT in the AMD pipeline is off limits and that I also cannot store values outside the blessed range in matrices. Luckily the AMD pipeline also has a Multiplier operation right before the matrix, and in most cases I can shift some of the magnitude from the matrix to the multiplier.

And according to the reports above LUTs on Nvidia are fucked as well despite claiming to have 1024 points. So I'll just have to hard disable the use of LUTs on Nvidia.

It is hard to imagine how any of this is possible in what is supposed to be a serious product. This API was developed by AMD employees. Someone, at least one person, must have been aware that the two LUTs in the pipeline behave very differently and then they looked at them having the exact same userspace API and decided that that's ok.

[njdom24]

Eesh, I don't envy you. Beats me how it's intended to function.
As for Nvidia, supposedly, KDE has a plan for Plasma 6.8, but this is the only related MR I see. If they can get something to work, I guess it'd be the path of least resistance to look at how when the time comes?

[visixx]

Possibly, but would that be the same LUT/op that also transforms sRGB into BT2020/wide color?

The source color space is sRGB/gamma22 ...

I appreciate you taking the time to explain the color pipeline and break down why the hardware cursor doesn't work, thanks!

I played around a bit more with this and I noticed that the banding actually is only visibly obvious in dark areas of the image. If the scene is bright (e.g. daytime in Minecraft) the direct scanned out image appears identical to the composited image. Personally, for games where the latency benefits of direct scanout matters, I'm totally cool with a bit of banding in darker spots, and I can just disable scanout for games where I'd prefer it to look perfect.

This definitely sounds like it's been extremely painful, so I totally understand the answer being (for Nvidia, especially) "the drivers are fucked and there's nothing to do." This already makes Jay the fastest compositor there is for games while being able to keep using a wide gamut or HDR which is awesome.

[mahkoh]

I've removed use of 1D LUTs on Nvidia by default. This means that the pipeline on Nvidia can only be used for scRGB -> HDR10 conversions.

I've also added environment variables to override all driver-specific code. See the changes to the book.

See also #1121

[visixx]

I've removed use of 1D LUTs on Nvidia by default. This means that the pipeline on Nvidia can only be used for scRGB -> HDR10 conversions.

I've also added environment variables to override all driver-specific code. See the changes to the book.

See also #1121

Thanks for providing an option for those of us with an NVIDIA card to still continue to use this feature even if there are issues.

I do have one last question (and, to be fair, I have no idea if there's any merit to it): I've noticed that drm_info whilst direct scanning out sRGB content with color pipelines enabled (so sRGB -> HDR) shows the corresponding Planes format as XRGB8888 whereas when that same game is composited it's using my configured format of XBGR2101010. Would that be a potential explanation behind the color banding?

Looking at the drm_info output I previously provided, I believe now that I mistakenly provided output from when I did not have a game running in direct scanout; when color pipelines are enabled and I run drm_info, the corresponding Plane does show the COLOR_PIPELINE as NVIDIA Lite instead of Bypass and the buffer's Format as XRGB8888.

Regardless, I imagine that it's the driver choosing an incorrect buffer format, so even if that's the problem it's still an upstream issue.

[mahkoh]

I've noticed that drm_info whilst direct scanning out sRGB content with color pipelines enabled (so sRGB -> HDR) shows the corresponding Planes format as XRGB8888 whereas when that same game is composited it's using my configured format of XBGR2101010. Would that be a potential explanation behind the color banding?

That sounds like a likely explanation. You can confirm it by setting the output format to xrgb8888, e.g. in the control center, and then comparing direct scanout and compositing.

[visixx]

I've noticed that drm_info whilst direct scanning out sRGB content with color pipelines enabled (so sRGB -> HDR) shows the corresponding Planes format as XRGB8888 whereas when that same game is composited it's using my configured format of XBGR2101010. Would that be a potential explanation behind the color banding?

That sounds like a likely explanation. You can confirm it by setting the output format to xrgb8888, e.g. in the control center, and then comparing direct scanout and compositing.

I gave this a shot and I was not able to obviously tell a difference between composited xrgb8888 and even abgr161616f. The format that most closely resembles the banding I'm noticing was rgb565, but even that isn't really the same since the banding/off-colors for composited rgb565 are very obvious regardless of the brightness of the image which is not the case with DS+color pipelines.

Thanks for humoring me.

[mahkoh]

As for Nvidia, supposedly, KDE has a plan for Plasma 6.8, but this is the only related MR I see.

The issue mentioned in the title of the MR is already solved in Jay.

[Copilot]

⚠️ Not ready to approve

It contains at least one correctness issue in new CM programming (fixed-point encoding) plus edge-case division-by-zero hazards in new LUT helpers that should be addressed before merging.

Pull request overview

This PR extends the Metal (DRM) backend’s color-management path to support programming scanout plane color pipelines (and related hardware color ops), enabling more accurate HDR / colorspace handling and allowing direct scanout to remain viable in more cases. The feature is disabled by default, can be enabled per device via config/IPC/CLI, and includes vendor-specific safety guards (e.g., AMD + hardware cursor).

Changes:

• Add a new Metal color-management programming layer (plane pipelines + CRTC gamma LUT path selection, LUT/matrix blob caching, and matcher/codegen for viable hardware paths).
• Expose a per-DRM-device “plane color pipelines” toggle via config (TOML spec + parser), IPC, jay_randr protocol (v38), CLI, and Control Center UI.
• Improve direct-scanout diagnostics by surfacing structured reasons for why scanout is rejected, and by integrating color-management programming into scanout decisions.

File summaries

File	Description
wire/jay_randr.txt	Adds jay_randr v38 request/event for plane color pipelines toggle and capability reporting.
toml-spec/spec/spec.yaml	Documents new plane-color-pipelines DRM device option.
toml-spec/spec/spec.generated.md	Generated docs update for plane-color-pipelines.
toml-spec/spec/spec.generated.json	Generated JSON schema update for plane-color-pipelines.
toml-config/src/lib.rs	Applies parsed plane_color_pipelines_enabled to DRM device config.
toml-config/src/config/parsers/drm_device.rs	Parses plane-color-pipelines option from config.
toml-config/src/config.rs	Stores plane_color_pipelines_enabled in ConfigDrmDevice.
src/virtual_output.rs	Refactors direct scanout preparation to return structured errors.
src/video/drm/sys.rs	Exposes new DRM object type constant for color ops and Range min field.
src/video/drm.rs	Adds DrmColorop object type and Range min field rename.
src/utils/object_registry.rs	Removes dead_code expectations on registry type aliases / key accessor.
src/utils/float_ext/tests.rs	Adds tests for MAX_SAFE_INT behavior for f32/f64.
src/utils/float_ext.rs	Introduces FloatExt and MAX_SAFE_INT for float->int safety bounds.
src/utils/const_slice.rs	Removes dead_code expectation from const slice helper.
src/utils/bool_ext.rs	Removes dead_code expectation from BoolExt.
src/utils.rs	Exposes new float_ext module.
src/tools/tool_client.rs	Bumps advertised jay_compositor interface version cap to 38.
src/state.rs	Adds state-level setter for per-device plane color pipeline usage.
src/ifs/jay_randr.rs	Sends plane pipeline capability/status event and handles set request (v38).
src/ifs/jay_compositor.rs	Bumps global jay_compositor version to 38.
src/gfx_api.rs	Introduces DirectScanoutError and returns Result for scanout preparation.
src/env.rs	Adds hardware color management environment variable toggles.
src/control_center/cc_gpus.rs	Adds GPU pane toggle for “Color Pipelines” when supported.
src/config/handler.rs	Adds IPC handlers for get/set plane color pipelines enabled.
src/cmm/cmm_transform.rs	Makes format_matrix public and marks ColorMatrix as JayHash.
src/cmm/cmm_render_intent.rs	Switches RenderIntent hashing to #[jay_hash] derive path.
src/cmm/cmm_eotf.rs	Adds conversion from compositor Eotf to algorithms AlgoEotf.
src/cmm/cmm_description.rs	Removes dead_code expectation on a color-description helper.
src/cli/randr.rs	Adds CLI subcommand + output/json fields for plane color pipelines status.
src/cli/json.rs	Extends JSON output with plane pipeline enablement + support fields.
src/backends/metal/video/metal_cm/metal_cm_plane/metal_cm_plane_matcher.rs	New: matches required CM operations to available plane pipeline ops and prepares programming.
src/backends/metal/video/metal_cm/metal_cm_plane.rs	New: parses plane color pipeline DRM properties and models plane ops.
src/backends/metal/video/metal_cm/metal_cm_paths/types.rs	New: typed representation of plane/CRTC CM operation paths and criteria.
src/backends/metal/video/metal_cm/metal_cm_paths/matcher.rs	New: generic matcher implementation to filter viable paths efficiently.
src/backends/metal/video/metal_cm/metal_cm_paths.rs	New: path/filter plumbing and accessors for generated path data.
src/backends/metal/video/metal_cm/metal_cm_matcher.rs	New: finds viable plane+CRTC programming combinations based on requirements.
src/backends/metal/video/metal_cm/metal_cm_lut.rs	New: LUT curve/LUT generation utilities used by hardware CM programming.
src/backends/metal/video/metal_cm/metal_cm_crtc/metal_cm_crtc_matcher.rs	New: matches CRTC gamma LUT requirements and prepares programming blobs.
src/backends/metal/video/metal_cm/metal_cm_crtc.rs	New: models CRTC color pipeline capabilities (gamma LUT op).
src/backends/metal/video/metal_cm.rs	New: central CM device/connector cache + programming application hooks.
src/backends/metal/video.rs	Integrates CM device/connector/plane/CRTC objects and pipeline capability detection.
src/backends/metal/transaction.rs	Programs CM state through transactions and filters default plane properties by plane type.
src/backends/metal/present.rs	Integrates CM programming into present/direct-scanout and improves scanout error reporting.
src/backends/metal.rs	Adds error variant for missing viable CM programming.
src/backend/transaction.rs	Adds transaction error for missing CM programming.
src/backend.rs	Extends BackendDrmDevice trait with plane color pipeline capability and toggle.
jay-config/src/video.rs	Exposes plane pipeline toggle API on DrmDevice config surface.
jay-config/src/_private/ipc.rs	Adds IPC messages/responses for get/set plane pipeline enabled.
jay-config/src/_private/client.rs	Adds client helpers for get/set plane pipeline enabled.
codegen/src/main.rs	Adds CM paths + LUT generation steps; improves rustfmt temp handling.
codegen/src/gen_lut.rs	New: generates LUT fill dispatch code into algorithms crate.
codegen/src/gen_cm_paths.rs	New: generates CM path tables/matchers into Metal CM paths module.
codegen/Cargo.toml	Adds dependency for built-in macros used by codegen.
Cargo.toml	Adjusts dev profile settings for jay-algorithms build behavior.
Cargo.lock	Locks new with_builtin_macros dependency.
book/src/environment-variables.md	Documents new hardware color management env vars.
book/src/control-center.md	Documents Control Center “Color Pipelines” toggle.
book/src/configuration/gpu.md	Documents config option and CLI usage for plane color pipelines.
book/src/cli.md	Documents CLI commands for plane color pipelines toggling.
algorithms/src/tf.rs	Adds AlgoEotf enum for transfer-function selection in algorithms crate.
algorithms/src/lut/generated.rs	New generated LUT dispatch implementations.
algorithms/src/lut.rs	New LUT utilities module (generated dispatch + 3D LUT fill).
algorithms/src/lib.rs	Exposes new lut module.

Review details

• Files reviewed: 62/64 changed files
• Comments generated: 3
• Review effort level: Low

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.