Universal dark mode shader filter: a feature request/discussion.

[unphased]

I got to thinking about why browser dark mode plugins/extensions are lame, and I realized that they all suck because they throw complexity at the problem when all that does is leave a long tail of issues (aka bright screen elements) falling through the cracks while adding more processing for the browser to do, slowing down the experience. The best browser extensions out there do an admirable job of preventing infuriating white flashes from temporarily being displayed by pages, but the price that you have to pay is reduced fluidity from the added processing done by the extension furiously intercepting the rendering, and you'll never be able to avoid fiddling with it when it invariably makes things impossible to read on some specific website.

I noticed that BetterDisplay is an indispensable part of using a mac at least for me, and it has a lot of interesting display mirroring type of functionality, so I wanted to see if there is any feasibility or interest from maintainers to add a feature that I'm imagining. I think it could potentially bring a large influx of users and paying customers. I think that it can be a stone that kills many birds, the sorry state of affairs of browser dark mode only being but one of these birds.

Let me explain.

The use case is that we're in a dark room, the prototypical example is doing browsing or getting work done on the computer while accompanying a young child to sleep. But the use cases are many. Sometimes we are up late in the night and need to use the computer and being exposed for even one second to a bright screen will ruin our night vision and possibly the rest of the night's rest, which has a real impact on physical health. Another example would be for the use of computers by astronomers in the field during a moonless night.

priority 1 is to be able to temporarily place a hard limit on visible flux out of the computer display.

Yes, we can manually reduce the brightness level, and in fact BetterDisplay already is really helpful by being able to on my MBP's XDR display crank the brightness down to below 1 nit by limiting the output brightness level even while the display is physically set to the lowest brightness. As I understand how this works, if the minimum level of brightness the monitor hardware allows us to set is 5 nits, we can set a "fake 1 nit" brightness level by scaling down the display buffer's intensities by a factor of 5. we lose 80% of our contrast ratio if that is applicable (aka non OLED) but we have a dimmer picture.

In challenging use cases at both ends of the brightness spectrum (when trying to make content legible at 1 nit in pitch dark or 1600 nits under direct sunlight) it tends to help to crank up the contrast slider.

I suppose one way of trying to capture the gestalt of dark mode is to say that we want to make the most of what contrast ratio might be available to us from the hardware at a given brightness level, by exploring ways of remapping/transforming/filtering the signal rather than merely scaling that signal.

I suggest a further enhancement to the low brightness scale technique, which is to implement different types of simple shaders to help support the stringent needs of dark mode, that is, preventing accidental bright exposure, by exploring some slightly more interesting but still simple shaders.

In terms of app UX:

• We can have a dark mode on keybind, which is not a toggle, so that users can mash that key blindly and ensure it will enable dark mode no matter how many times the key is pressed. What this would do is peg the brightness immediately to a low value like 3 nits, and enable dark mode filtering.
• User can configure which filtering techniques are applied for dark mode, as well as optionally use hotkeys to switch between them, to handle when content becomes impossible to consume under certain filters. When enough of these exist, the user will be able to confidently use their computer and view all kinds of content without worrying about accidentally getting blasted with unwanted light.

A brainstorming list of possible dark mode filter implementations:

1. An astronomer may already be familiar with the use of a red filter which guarantees no blue or green light can ever be emitted so it keeps your retinal rods safe. This is a good baseline but could be rather limiting or hardcore for many typical computer uses outside of emergencies. One way to think of red filter is of course that it represents Night Shift/Night Light/f.lux taken to its extreme logical conclusion.
2. An idea i had was the display buffer could be quickly metered for brightness and if the content is over 50% brightness, we could simply invert the image. This guarantees the image is never more than 50% brightness at any time. Thinking about it more, though, this is too rudimentary to be useful, and would cause nauseating flickering back and forth from inverted to regular picture when dynamic content straddles the 50% brightness mark. So forget this idea, it stinks.
3. a sobel or edge detection filter. Screenshot from wikipedia:
   
   Notice how the bright parts of the image got turned dark but the structure and detail remain intelligible. The drawback with this is that it will not necessarily mangle all text, but it would likely need some tuning to prevent certain text sizes from being difficult to read due to straddling a size between coming out as single edge or outline. It also does not guarantee low brightness, since content with high frequency patterns would result in this thing producing more bright output. There's also some unclear choices around how to deal with colors without potentially completely destroying the visual intent of the content.

Unfortunately I have hit a wall in terms of implementation ideas rather earlier than I would like, because I am starting to realize that it really will be difficult to actually guarantee many things together that I would want:

• guarantee the curtailing of overly bright content wherever it appears
• Attempt to keep all content legible with priority to textual content
• (difficult to outdo overbuilt browser plugins which have a metadata advantage for) preventing images from being completely ruined

Still, I think that there is something worth pursuing here even if we can only hit the first few, and I am convinced that it would greatly push the usability frontier of computing forward if we could make a really computationally efficient filter that applies to all content displayed on the screen that we can pull up instantly no matter what software we're using.

I think I have a decent illustrative real world example from this thought experiment on dark mode, and it's this screenshot from my gmail page:

The problem here is I already set a dark theme through the app, but it does have some glaring gaps; the big compose button and many emails still have white backgrounds. The browser extension Dark Reader is able to solve the problem but at the cost of bogging down the browser, and besides we're tempted to not engage the extension for a site that we already enabled dark mode on, but in this case it's merely a half baked dark mode and so it has fallen through the cracks.

Ideally I was imagining that a lightweight shader based filter may exist that could "detect regions of brightness" and "selectively invert them" but I now I wonder if that may be difficult to implement sufficiently efficiently. I do think detecting and nearly instantly responding by inverting something like 32x32 square pixel regions (and then refining and computing boundaries) if we meter them as "too bright" could actually be workable.

[waydabber]

Great idea. I was thinking about some improved night/dark mode feature for the app as well. Although for most users dark mode + night shift, combined with some dark mode browser extension might be fine, there could indeed be room for further improvements. I was thinking about some further contrast and color temperature/white point adjustments only though.

(I moved this to discussions so more ideas could be added).

[unphased]

When I get to playing around with render buffer shaders I will play with some experiments and report back if I find any useful results. I'm kinda hoping there might be a computationally efficient way to make a filter that makes a kernel change gradually based on local metering, somehow. I think making control flow dependent on texture read can be bad for performance but it could be mitigated if the branching could be sorted or dispatched coarsely

[waydabber]

Somewhat related:

#4155

I'll probably also add an option to stream filtered content to the origin screen (by excluding the stream window), which should more-or-less allow any kind of effects to be applied to a screen.

Now this implementation uses CI filters for various effects (with varying results), but we could use custom shaders (that should be even more efficient as CI has some inherent overhead).

[waydabber]

Ok, this is maybe bit crazy. But there are now 4 pages of image adjustment, effect and blend options. That should probably cover most of the use cases. :)

[unphased]

holy wow. is this released? I want to play with this so bad. Let me know which version it will appear in. on 3.4.1 now

(opted in to prerelease looks like i can get 3.4.2)

[unphased]

looks like I have it. this stream display thing seems insanely powerful.

[unphased]

It's really cool to be able to do this for streaming but I must say the stream display is very power intensive. The Image Adjustments is more efficient but i can only make an "astronomy mode" dark mode with that by cranking down non red gains.

Is stream display the only approach to be able to have custom shaders on the display output? If so it may be a non starter for performance reasons (for leisurely getting fancy shader dark mode in the evenings)

anyhow its cool to see that text does indeed show up outlined and hard to read with the edge modes, although it certainly works as expected to produce a black field with only features showing bright. Something more complex would be needed to make text not appear ringed.

[waydabber]

Yes, I know of no other way currently. Things could be made more lighter in the future if I stop using CoreImage rendering pipeline and use fully custom Metal shaders instead, as some of the overhead is coming from CI. However replicating all these shaders would be a huge amount of work. I'll probably work on some of the more basic ones.

Sadly directly applying filters to the desktop compositor without using streaming is (afaik) not possible with any recent macOS versions as the rendering pipeline is protected - once can't really read, mess with the desktop composition process or low level rendered framebuffer data, Apple blocked every known way for obvious security and privacy reasons.