Decouple Ambient Occlusion from Shadows

[nauta-turbidus]

Problem

Currently, Ambient Occlusion is subtracted from the calculated light value for a vertex, and light data directly influences vertex color. It causes data loss in shader, such that shadowed nodes, especially those in complete darkness, have no hardware coloring color data passed into the shader, or the data is severely scaled down and loses precision. Shader can't differentiate between shadow proper and ambient occlusion either. Of course this doesn't matter currently, but #14343 allows (that's its point) to additionally and uniformly light up everything.

The issue encountered there is that:

1. Hardware coloring is often lost when applying Ambient Light, and it doesn't work at all for nodes that had been in complete darkness (this one can be solved by changing one step of light encoding).
2. There's no Ambient Occlusion with bright enough Ambient Light (how bright exactly depends on how the colors are clamped), in particular none at all for nodes that had been in complete darkness.

See the discussion under the PR for screenshots showing both issues.

Solutions

The way the data is processed before passing to the shader has to change. Potentially, the way the data is passed to the shader may have to change too.

First of all, Ambient Occlusion is calculated in the function getSmoothLightCombined() at https://github.com/minetest/minetest/blob/58ea11c2b3101667e7a5ed118bf1cb2204f63586/src/client/mapblock_mesh.cpp#L148-L250 and right then and there it's baked into light_day and light_night, which are 8bit uints, which are then combined into a single u16 return value. A few other functions call it, but in the end its return value ends up either at https://github.com/minetest/minetest/blob/58ea11c2b3101667e7a5ed118bf1cb2204f63586/src/client/content_mapblock.cpp#L262 or at https://github.com/minetest/minetest/blob/58ea11c2b3101667e7a5ed118bf1cb2204f63586/src/client/content_mapblock.cpp#L472 and in both cases is unpacked from the u16 and put into a struct for further processing.

Stopping here for a moment, to decouple AO from shadows we'd need to either move the calculations out from that function, or just make it not bake the AO into its return value and return it separately. A few possible ways to do it:

1. Make the return value an u32. This should give us plenty of space to pack whatever is needed, possibly at the cost of not using all the bits and increasing memory usage.
2. Considering there are only 4 possible values of ambient occlusion in this function (no AO or one of the 3 values from the lookup table: https://github.com/minetest/minetest/blob/58ea11c2b3101667e7a5ed118bf1cb2204f63586/src/client/mapblock_mesh.cpp#L232-L236), another option would be to devote just 7 bits to light_day and light_night each (instead of 8) and have remaining 2 bits encode the ambient occlusion.
3. Pass a pointer to the function to place the AO calculation outcome directly somewhere, where it'll possibly be used later.

Now, at a later point the processed light data ends up in the encode_light() function at https://github.com/minetest/minetest/blob/58ea11c2b3101667e7a5ed118bf1cb2204f63586/src/client/mapblock_mesh.cpp#L950-L976 where it gets made into a color (which later is multiplied with hardware coloring color). Currently, it's essentially preapplying shadows onto the hardware coloring color, while storing data about the artificial/natural color ratio in the alpha channel (the latter is later used by the shader to determine the final lighting color – artificial and natural light have subtly different hues). What we need is storing the maximum color value for that vertex in the color channels, that is, applying only ambient occlusion (and "directional" face shading) there, while storing the rest in the alpha channel. Ways to do the latter:

1. Use 4 bits for the light value and 4 bits for the ratio.
2. Use 4 bits for the day and 4 bits for the night light.
3. Use 4 bits for the natural and 4 bits for the artificial light (requires significant changes in the previous parts of the pipeline).
4. Expand the amount to full 32bit floats instead of just bytes.

So why are we using bytes here if the GPU is working with floats anyway? Well, OpenGL can take color data both in a 4-byte float per channel format, and 1-byte int per channel format. We're using the latter. 4th point right above means that we'd change it. Possible side effects? Not sure at which point the bytes get converted to floats, but it's on the OpenGL side. Some incredibly old hardware may fail doing this completely (not sure if we support such). Newer hardware probably won't have issues, because bottleneck lies elsewhere anyway.

Alternatives

1. Do nothing at all about this. We're missing out on some standard graphical options.
2. Refactor all this code and somehow make a cleaner way to implement it all along the way.
3. Pass a second vertex color to the shader, as in Use a new two-color vertex type for mapblock meshes #14789 – not if this is needed, I think we'd have plenty of space to put all we need if we had the 4 floats available, instead of just 4 bytes.

Additional context

I'm willing to work on a PR for this, standalone or as part of #14343.

[Andrey2470T]

Thanks for looking into these two issues appearing with the ambient light. TBH I can not get why the more intensive ambient light applied the more AO gets vanished on the nodes. If I correctly understand, AO is a just darkening the light color aka decreasing both day and night banks and AO as the remain light is encoded into the u16 value.

How would it be more correct to bake the hardware color into the vertices to solve the issue with fading out the colors? I have an idea to squash it into the single u32 component of video::SColor, e.g. the blue one instead of multiplying with all rgb components. However, since the hardware color has u32 components and compressing them into u8 would lead to the data loss I assume. Then in the nodes vertex shader to decode this value back to the vec3 and pass it to the fragment one for summing it with the texture color. I've already tried to do so, however like GLSL doesn't support bitwise operators and requires some extension for that:

0(216) : error C0000: ... or #extension GL_EXT_gpu_shader4 : enable
0(218) : error C7548: '>>' requires "#extension GL_EXT_gpu_shader4 : enable" before use
0(219) : error C7548: '>>' requires "#extension GL_EXT_gpu_shader4 : enable" before use
0(220) : error C7011: implicit cast from "int" to "uint"
0(220) : error C7548: '&' requires "#extension GL_EXT_gpu_shader4 : enable" before use

static void applyTileColor(PreMeshBuffer &pmb)
{
	video::SColor tc = pmb.layer.color;
	if (tc == video::SColor(0xFFFFFFFF))
		return;
	for (video::S3DVertex &vertex : pmb.vertices) {
		video::SColor *c = &vertex.Color;

		// Encode the tile color into the single u32 component
		// saving that as the blue one in the vertex color
		u32 red_u8 = (tc.getRed() / 255  & 0xff) << 16;
		u32 green_u8 = (tc.getGreen() / 255 & 0xff) << 8;
		u32 blue_u8 = tc.getBlue() / 255 & 0xff;

		u32 hw = red_u8 | green_u8 | blue_u8;

		c->setBlue(hw);
		/*c->set(c->getAlpha(),
			c->getRed() * tc.getRed() / 255,
			c->getGreen() * tc.getGreen() / 255,
			c->getBlue() * tc.getBlue() / 255);*/
	}
}

// In the nodes vertex shader:
// Calculate color.
vec4 color = inVertexColor.argg;
int hw = int(inVertexColor.b);

int hw_red_f = hw >> 16;
int hw_green_f = hw >> 8;
int hw_blue_f = hw & 0xff;

// 'hardwareColor' is sent then in the fragment shader for
// summing with 'base.rgb'
hardwareColor.r = float(hw_red_f);
hardwareColor.g = float(hw_green_f);
hardwareColor.b = float(hw_blue_f);

[nauta-turbidus]

@Andrey2470T First of all, you don't need bitwise operations, you use arithmetic operations for that. My branch linked somewhere on your PR does just that for the alpha channel.

Secondly, currently it's u8 per channel that is passed to OpenGL. Passing f32 instead with more data packed in extra bytes is fine and won't lose any precision compared to current master (you only have the mantissa+1 bytes though).

Last but not least, the first thing you mentioned. Let me explain it again:

Currently, in master, AO is baked into light data, which also describes normally dark/shadowed places, and then it's multiplied with the hardware coloring color (baked into it), and that is the vertex color rgb. Vertex color alpha theoretically describes natural/artificial light ratio, which allows applying a different tint for each. The result of this is that if you are adding ambient light value to the vertex color (which you are doing), you are easily going over 1 in vertex color channels, either only for non-shadowed vertices or for all of them. Anyway, if you clamp it, you're cutting off any differences: both between the channels (leading to desaturation) and between different vertices (leading to loss of shadows and AO). Now, loss of shadows is what we actually want, but not loss of AO. So the issue is that we are treating them the same.

What I did in my branch is splitting the shadows+AO from the hardware coloring, and packing that into the alpha channel alongside the ratio instead. This allows me to cap the color value for each channel at what's in them, i.e. not allowing higher values than that. So, I unpack the data from the alpha channel, then make a copy of the color, then apply shadowing obtained from alpha onto the copy, then apply ambient light onto that, then I clamp each channel at the original vertex color. This works somewhat nicely, but one major issue is AO. And so I'd need to split AO from the light data, and modify the hardware coloring (i.e. color that shouldn't be affected by Ambient Light) with it, and then the same approach as outlined above preserves AO.

[sfan5]

Stopping here for a moment, to decouple AO from shadows we'd need to either move the calculations out from that function, or just make it not bake the AO into its return value and return it separately. A few possible ways to do it: [...]

It doesn't matter if we use a u32, clever bit packing or a struct inside C++. Not worth discussing this part.

What we need is storing the maximum color value for that vertex in the color channels, that is, applying only ambient occlusion there, while storing the rest in the alpha channel. Ways to do the latter: [...]

Using the alpha channel is a cool hack you can do when there's just one value to store.
Working with bit packed values isn't very ergonomic inside shaders however.

4. Expand the amount to full 32bit floats instead of just bytes.

There's no reason to do this.
It would blow up the vertex size by 96 bits when you essentially only want to store less than 8 extra ones.
The cleanest way would be to introduce a new vertex format for map blocks and just add the extra data we want there.

[nauta-turbidus]

There's no reason to do this.
It would blow up the vertex size by 96 bits when you essentially only want to store less than 8 extra ones.

Are you sure about it? I.e. does OpenGL expand to f32 before sending to GPU, or does it indeed send u8 to GPU and GPU expands it to f32? Only in the latter case would the inflation actually matter.

[sfan5]

Yes. Our vertex data on CPU has a 32-bit integer color (S3DVertex) and is uploaded to the GPU as-is.

[nauta-turbidus]

So what data does the shader need per vertex? Color, AO, light...?

Two colors? Rgb/rgba? One color and a byte for something more?

[sfan5]

If I understand the description correctly all you need is one byte that encodes the light level so we don't have to pre-multiply that onto the color.

[Andrey2470T]

@nauta-turbidus I've modified the changes in your branch and it seems there are no more those artifacts which I showed in the PR when testing that. I left the encoding in the color unchanged (where you mix the day/night ratio and the light in the alpha channel separating it from the rgb), but I've done passing the hardware color to the fragment shader and multiplying that with the texture one. Here that how the nodes_shader looks like now:

// vertex shader

// 'hardwareColor' is passed as vec3 to the fragment shader
hardwareColor.rgb = inVertexColor.rgb;

float ratio = floor(inVertexColor.a*16)/16;
float lightBase = floor((inVertexColor.a - ratio)*256)/16;

nightRatio = 1.0 - ratio;

// 'light' is almost the same 'varColor', just renamed
vec3 light = vec3(lightBase, lightBase, lightBase);
light.rgb = light.rgb * (ratio * dayLight.rgb +
	nightRatio * artificialLight.rgb) * 2.0;

float brightness = (light.r + light.g + light.b) / 3.0;
light.b += max(0.0, 0.021 - abs(0.2 * brightness - 0.021) +
	0.07 * brightness);

light.rgb += ambientLight;

lightColor = clamp(light, 0.0, 1.0);

// fragment shader

color = base.rgb;
vec4 col = vec4((color.rgb * hardwareColor.rgb) * lightColor.rgb, 1.0);

Results (note the armchair at the right is hardware-colored)

[Andrey2470T]

On the ground:

[nauta-turbidus]

On the ground:

This looks surprisingly well. I'll have to test it.