Add texel anti-aliasing

[HybridDog]

Nearest neighbour sampling causes aliasing on texel boundaries. With bilinear filtering and a nonlinear transformation of UV coordinates, we can render textures such that it looks like nearest neighbour sampling with anti-aliased texel edges.

This is an implementation of what has been linked at: #13108 (comment)

Example image with enabled texel_antialias and bilinear filtering, and strong undersampling:

(taken at 2025-12-31)

Limitations

These limitations are not something newly introduced by this PR but have an impact on the anti-aliasing quality nonetheless.

Seams due to tiling

Like with current bilinear filtering, texel colours reach the opposite side of the node.
Example, where a stripe of blue from the right side of the A appears on the left side of the texture, among other such stripes:

Darkening due to gamma-incorrect rendering

Like with mip mapping, since rendering happens without gamma correction, pixels can be darkened. In this example, green and red mix to dark pixels because of the non-linearity of sRGB:

Inventory item and wielded wieldmesh

Inventory item rendering apparently doesn't use bilinear filtering. Wielded wieldmesh rendering also uses nearest neighbour sampling for some reason whereas wielded nodes rendering uses the object shader. Texel anti-aliasing doesn't work in these cases.

How to test

Enable bilinear or trilinear filtering. Enabling undersampling can help to investigate the effect.

Quadratic kernel

In the video and additional notes at https://www.patreon.com/posts/83276362, t3ssel8r has shown a different way to smooth the edge.
To me the result with my code looked almost the same, so I have kept the simpler code.
Here's my code which should use the quadratic kernel if I didn't make a mistake:

vec2 smoothstep_impl(vec2 edge0, vec2 edge1, vec2 x) {
	vec2 t = clamp((x - edge0) / (edge1 - edge0), 0.0, 1.0);
	return t * t * (3.0 - 2.0 * t);
}

vec2 uv_texel_antialias()
{
	vec2 box_size = clamp(fwidth(varTexCoord.st) / texelSize0, 1e-5, 1.0);
	vec2 tx = varTexCoord.st / texelSize0 - 0.5 * box_size;
	vec2 tx_off = smoothstep_impl(1.0 - box_size, vec2(1.0), fract(tx));
	return (floor(tx) + 0.5 + tx_off) * texelSize0;
}

Effect of textureGrad()

I couldn't find an example where using textureGrad() instead of texture() in the shader fixes artifacts. It'd be nice to have an example to justify the change.

[HybridDog]

Here are some more screenshots with strong undersampling.

No mip mapping, no trilinear filter, no anisotropic filter

Nearest neighbour:

bilinear with texel_antialias (same screenshot as in the first comment):

bilinear without texel_antialias (thus textures upscaled to the min size):

mip mapping, trilinear filter, anisotropic filter

texel_antialias enabled:

texel_antialias disabled (thus textures upscaled to the min size):

[Desour]

Thanks!

I don't really like how the settings work.
It would be nice if we could get rid of the current bilinear_filter, trilinear_filter, and texture_min_size settings, and instead just offer texel_antialiasing (and make it imply trilinear filtering).
After all, IIRC, the main reason why we kept the settings is that users can get linear minification: #14007 (comment)
(I guess we also need to do something like a test rendering to check if anisotropic filtering implies bilinear filter, and disallow it then with disabled texel_antialiasing.)

Redefining the settings can also go into another PR, to avoid drama here.

Seams due to tiling

Also already happens with bilinear filter and texture_min_size.
The correct fix for this is to set tileable_vertical and tileable_horizontal to false in these cases.
But as these aren't even documented yet, I guess we shouldn't enable texel aa by default for now.

Effect of textureGrad()

It's hard to see a difference. And when there is one, I can't tell that one would look more correct than the other.

Inventory item and wielded wieldmesh

We should also use this for particles, btw..
(Can also happen later, i.e. after we got #include in shaders. It's probably not a big issue if this feature is not supported everywhere.)

[Desour]

Works. 👍

(With msaa ("fsaa") and this, only alpha tested edges are still jagged, right (as can be seen with undersampling)? One could look into using the alpha to coverage for those.)

[sfan5]

so won't this visibly change behavior for people who use bi/tri?

[HybridDog]

yes

[sfan5]

then we shouldn't enable it by default

[sfan5]

nobody is forcing us to use that. we can use textureGrad just fine (as long as the GLSL version is high enough)

[sfan5]

no textureGrad here?

[sfan5]

are you sure this even shows up in object_shader?

[HybridDog]

It works for me when I test it, so I assume that it works in general.

[Desour]

I somehow forgot testing with objects.

It appears to work with dropped items, but not for mtg boat or mtg cart.

[sfan5]

please update the comment to explain how texel AA makes this redundant.

[sfan5]

We should also use this for particles, btw..

Particles don't even use their own shader right now. That's another reason why blindly enabling this by default is not good (since it won't cover all cases currently covered by texture_min_size).

Also has anyone performance tested this on low-end HW?