Implement vertex shading

[ywmaa]

Implements Vertex Shading
resolves #43093

• OpenGL
• Vulkan Mobile
• Vulkan Forward+

Bugs :

• Vertex Shading receives shadows from one light source in GLES3 No Shadows will be casted in Vertex Lighting
• When using Forward+, vertex lighting doesn't render unless you get very close to the light node.
• When using lightmaps, DirectionalLights with the Static bake mode still affect lightmapped surfaces.
• This PR completely removes shadows in the compatibility renderer regardless of whether vertex shading is enabled
• The "force vertex shading" project setting should prompt users to restart the engine
• The "force vertex shading" project setting appears to only work for StandardMaterials (it should work regardless of the material type and should apply to all materials in the scene)
• Use simplified light model for vertex shading like 3.x
• Directional Light does nothing if shadows is enabled both in Vertex Lighting or Pixel Lighting (GLES3 only Issue)
• Shadows for vertex shading in Vulkan/Forward Renderers.
• In Forward+, the vertex lighting will pop in on the edges of the viewport. (Disable Light clustering with vertex lighting)
• When you enable shadows in Compatibility, the light is drawn twice: one with per-pixel shading, one with per-vertex shading
• when adding a varying in gdshader, it errors out

Note: This PR implements Vertex-Lighting without shadows.
Note: This PR implements Vertex-Lighting with Pixel-Shadows.
Production edit: Pixel shadows are only available for the first DirectionalLight3D in the scene. Omni and spot lights cannot cast shadows on vertex-shaded materials (like in Godot 3.x), except in the Compatibility rendering method where lights with shadows are rendered with a multipass approach.

[ywmaa]

For some reason, only the material preview shows the vertex shading, in the viewport it looks like it doesn't use vertex shading.
Any Idea why ?

Per-Pixel

Per-Vertex

[AnalogFeelings]

For some reason, only the material preview shows the vertex shading, in the viewport it looks like it doesn't use vertex shading.
Any Idea why ?

Per-Pixel

Per-Vertex

That does not look like vertex shading at the bottom material preview

[ywmaa]

are you sure this is not vertex-lighting ?

this is normal pixel lighting :

[ywmaa]

mmm I guess I did vertex shadow not vertex lighting, fixing it.

Edit : lol, it is actually the opposite, shadow calculation should be added in vertex.

[ywmaa]

I guess I need an advice from the Rendering Team, currently I am almost duplicating functions for the vertex-shading, should I just use Preprocessor to move every thing to vertex shading, or should I keep duplicating the stuff, because we may remove features from the vertex shading ?

[ywmaa]

Progress Vertex lighting (Lighting + Shadows) in GLES3/Compatibilty renderer:

currently known bug : Vertex Shading receives shadows from one light source in GLES3, and if you enable shadow casting on the second light, it won't cast light at all to the mesh.

Bug images :

Second light no shadows :

Second light with shadows :

Edit :

Well I just tried Godot 3.5, and vertex_lighting doesn't cast shadows at all

was it intended ?

[Calinou]

Regarding why the second light with shadows disappear, remember that Godot 4's GLES3 performs multipass lighting if you use multiple lights with shadows. This is different from Godot 3 GLES3 (and more comparable to Godot 3 GLES2, although it used multipass for non-shadowing lights too).

Well I just tried Godot 3.5, and vertex_lighting doesn't cast shadows at all

Yes, real-time shadows can't be received by vertex-lit surfaces. Only baked lightmap shadows can be seen on them.

In GLES3 (unlike GLES2), it should be possible to sample the shadowmap texture in vertex() if you want real-time shadows to be visible on vertex-lit surfaces, but it's not trivial. The shadow detail will also be limited by the receiving object's mesh complexity. (If this behavior needs to be disabled, the Do Not Receive Shadows material flag can be used.)

[ywmaa]

In GLES3 (unlike GLES2), it should be possible to sample the shadowmap texture in vertex() if you want real-time shadows to be visible on vertex-lit surfaces, but it's not trivial. The shadow detail will also be limited by the receiving object's mesh complexity.

so do you think I should remove shadows from vertex-lighting ?

[Calinou]

so do you think I should remove shadows from vertex-lighting ?

I think you can remove them for now. Support for receiving shadows in vertex-lit materials can be implemented in a future PR (if it's feasible at all) 🙂

[ywmaa]

Now Compatibilty/GLES3/OpenGL Rendering should be ready for Vertex-Lighting testing and regression testing.

I will continue work on Vulkan Mobile + Vulkan Forward+

[ywmaa]

SOOOOO Vertex Shading now works in Vulkan

Unfortunately I had to do Mobile + Forward in one commit, as both depend on scene_forward_lights_inc.glsl

Now TIME for review,
but remember :

[ywmaa]

My comment on shadows for vertex lighting :

I think it is possible to cast shadows using vertex_shading.

But I think vertex shading is mainly useful for performance reasons, and probably realtime shadows are costly.

So it doesn't make a lot of sense to have realtime shadows in vertex lighting.

I may implement shadows in another PR if there is a demand for it.

Personally I don't need shadows in my game for vertex lighting.

[Calinou]

Tested locally, it mostly works as expected. This is a great start 🙂

Testing projects:

• test_pr_83360.zip
• test_lightmap_preview_bake_4.x.zip

There are some rendering issues:

• When using Forward+, vertex lighting doesn't render unless you get very close to the light node (the objects far away use per-pixel shading for comparison):

simplescreenrecorder-2023-10-19_20.03.01_forward_plus.mp4

This may be related to the light clustering system.

• When using Mobile, vertex lighting occasionally flickers in and out on specific objects when moving the camera (this may be hard to reproduce):

simplescreenrecorder-2023-10-19_20.04.02_mobile.mp4

In the testing project, the plane is subdivided so it can receive vertex lighting correctly.

Another issue is that when using lightmaps, DirectionalLights with the Static bake mode still affect lightmapped surfaces:

Current appearance	Expected appearance (light is hidden)

This isn't an issue with omni or spot lights. You can hide the directional light after baking as a workaround, but then it won't affect dynamic objects.

[ywmaa]

@Calinou
these two commits should fix your issues (except flicker in Vulkan Mobile, or maybe it is fixed too ?)
I tested locally, but I need a confirm.

[AnalogFeelings]

Recording.2023-10-20.162439.mp4

It seems some polygons become unlit if they become partially out of view?

EDIT: I am on Forward+ by the way.

[ywmaa]

@AnalogFeelings Yes this happens to me too, I think this is light clustering in Forward+ and not a bug.

Maybe calinou knows.

[AnalogFeelings]

I also noticed setting the viewport to lighting only causes HEAVY artifacts.

SPOILER_20231020-1433-32.5621820.mp4

[ywmaa]

I also noticed setting the viewport to lighting only causes HEAVY artifacts.

Interesting, I'll check it out.

[ywmaa]

@AnalogFeelings

I am unable to reproduce your issue :

But I noticed that in lighting mode it shows vertex lighting like pixel lighting, so that will probably need a fix.

[Calinou]

But I noticed that in lighting mode it shows vertex lighting like pixel lighting, so that will probably need a fix.

Lighting mode replaces all materials in the scene with a basic ShaderMaterial that uses per-pixel lighting (it might just be the fallback material). It can't know whether the original material used vertex shading or not. The Lighting debug draw mode's appearance will probably have to be toggled with a project setting.

[ywmaa]

Just fixed some bugs that I noticed from my work.

[ywmaa]

So rebased to reslove the conflicts, @clayjohn the issue was with this section https://github.com/godotengine/godot/pull/83360/files#diff-94158bc3d510819bb8d7ebd62a8890eeebad49608d9e934d148b73b6b65aeff9L1422

which you removed anyway previously so I just accepted the incoming result "the vertex shading commit changes"

also for some reason your name does not show again in the commit even though I didn't change anything in the commit message ?! can this be solved ?

[clayjohn]

So rebased to reslove the conflicts, @clayjohn the issue was with this section https://github.com/godotengine/godot/pull/83360/files#diff-94158bc3d510819bb8d7ebd62a8890eeebad49608d9e934d148b73b6b65aeff9L1422

which you removed anyway previously so I just accepted the incoming result "the vertex shading commit changes"

That makes sense. It looks fine now

also for some reason your name does not show again in the commit even though I didn't change anything in the commit message ?! can this be solved ?

I'm not sure why that is. I wouldn't worry about it, the attribution is in the commit message anyway

[clayjohn]

Ready to merge :)

[akien-mga]

also for some reason your name does not show again in the commit even though I didn't change anything in the commit message ?! can this be solved ?

I'm not sure why that is. I wouldn't worry about it, the attribution is in the commit message anyway

That was because the commit description was missing a separation line between the first paragraph and the "Co-authored-by:" line. GitHub is very picky about this, that line needs to be properly separated from the rest of the commit message.

I amended the description to fix it.

[akien-mga]

Thanks @ywmaa and @clayjohn for the outstanding work! and @Calinou of course for amazing testing and review throughout the process! This should make a number of users very happy 🎉

[PurpBatBoi]

Would be nice if more flexibility were given to this over time,, for example Toon Shading with vertex-shading.
Yeah that's possible, the game the most people like to copy toon shading from (Wind Waker) actually uses vertex shading under the hood. Mostly because that the Gamecube/Wii used vertex-shading as their main method of lighting objects.
But some games faked phong/pixel-shading with some tricks, so some GC/Wii games had a per-pixel shading-like look.

Funfact Wii is essentially a repackaged Gamecube, the big differences are.. more RAM and better GPU for the Wii but anyway what i meant is the vertex shading mode should be able to handle more than just a basic diffuse shader

[PurpBatBoi]

(like in Godot 3.x), except in the Compatibility rendering method where lights with shadows are rendered with a multipass approach.

why isn't this the same on on forward+?

[clayjohn]

(like in Godot 3.x), except in the Compatibility rendering method where lights with shadows are rendered with a multipass approach.

why isn't this the same on on forward+?

The compatibility renderer is intended for older devices and the Forward+ renderer is intended for newer devices. Therefore the compatibility renderer uses a technique more suitable for older devices.