Add support for per-vertex lighting functions in shaders

[bunabyt3]

Describe the project you are working on

I'm working on a game in the style of something from the era of Half-Life 1 or Unreal Tournament. These games didn't use shaders, but they did achieve some interesting effects using the fixed-function pipeline; since we have to use shaders in Godot, as fixed-function is considered obsolete, I figured I would try recreating those effects, as well as adding new ones specific to my game in the same style.

Describe the problem or limitation you are having in your project

Godot 4.4 provides the ability to enable vertex lighting, which I think is great. However, custom lighting functions are not supported in this mode, forcing us to use Lambertian diffuse instead of any custom lighting model. The shader used by the StandardMaterial is able to use a more advanced lighting model which includes specular lighting, but doesn't support things like Blinn-Phong specular that would work better in vertex-lit mode.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

I think that when vertex lighting is turned on in a project, the light() function inside user-created shaders should be used for shading vertices rather than pixels. Alternatively, a separate vertex_light() function could be defined in a shader to use a custom vertex lighting model. This would allow for fully custom lighting models in vertex-lit mode, expanding the possibilities for vertex lighting effects instead of forcing everything to use Lambertian diffuse.
Another nice thing to have would be a feature where the results of light() calculations can be accessed inside of the fragment shader; this would allow, for example, creating a toon shader without having to use unnecessarily expensive (or aesthetically inappropriate) per-pixel lighting.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

This seems like something fairly simple to implement, but since I don't quite know my way around the rendering code, I'm not sure if it is. However, if my assumptions are correct, then one could simply:

• Check if vertex lighting is enabled
• If so, run the light() function per-vertex instead of per-pixel

I wasn't able to implement this on my own, so feel free to tell me how realistic of a request it really is.

If this enhancement will not be used often, can it be worked around with a few lines of script?

As far as I'm aware, scripts can't be used to inject or modify render passes, but please correct me if I'm wrong.

Is there a reason why this should be core and not an add-on in the asset library?

I believe the limits of what's possible without modifying the engine code would prevent such an add-on from being created. Once again, please correct me if I'm wrong.

[Calinou]

cc @ywmaa, as they implemented vertex shading in godotengine/godot#83360.

I figured I would try recreating those effects, as well as adding new ones specific to my game in the same style.

Which effects are you thinking about creating? Note that it's possible to emulate the per-vertex look using fragment shaders. It's less efficient but more flexible. If the shader is to be used on a few surfaces only, it might be a good enough solution.

[bunabyt3]

@Calinou At the moment, I want to use Blinn-Phong specular lighting, which I've already implemented in the light() function of my shader. The important thing here is that the light() function should be per-vertex when vertex lighting is enabled, and per-pixel when it isn't, regardless of use case.

[PurpBatBoi]

cc @ywmaa, as they implemented vertex shading in godotengine/godot#83360.

I figured I would try recreating those effects, as well as adding new ones specific to my game in the same style.

Which effects are you thinking about creating? Note that it's possible to emulate the per-vertex look using fragment shaders. It's less efficient but more flexible. If the shader is to be used on a few surfaces only, it might be a good enough solution.

TELL ME HOW, TELLL MEE HOWOWWWWW
i didnt know you could recreate vertex-lighting like shaders with fragment

tbh would be cool to be able to mix vertex shading but blinn phong specular, or vice-versa

[insomniacUNDERSCORElemon]

Which effects are you thinking about creating?

I am building a general-use material for textureless vertex-color models, adding per-instance parameters for CSG/textmesh, can use transparent vertex colors for effect color (custom/team color, glass effect, glowing eyes, blink-type animations etc). Cel-shade effect is available this way too.

Aside from performance/scalability (fragment may-or-may-not be causing crashes when editing for me), it seems difficult to get the desired aesthetic with fragment as it just looks pixel-y. Even for better gradient quality, per-pixel seems way more than needed especially for UHD users... one or two subdivisions or other filtering would go a long way... I could even imagine interpolated data usable sort of like an LoD system with various factors.

[PurpBatBoi]

cc @ywmaa, as they implemented vertex shading in godotengine/godot#83360.

I figured I would try recreating those effects, as well as adding new ones specific to my game in the same style.

Which effects are you thinking about creating? Note that it's possible to emulate the per-vertex look using fragment shaders. It's less efficient but more flexible. If the shader is to be used on a few surfaces only, it might be a good enough solution.

Godot_v4.4.1-stable_win64_51BsbOt0VW.mp4

after some R&D i managed to get a fake light point, but i need to figure out why it doesn't update in the editor realtime..

[PurpBatBoi]

@Calinou At the moment, I want to use Blinn-Phong specular lighting, which I've already implemented in the light() function of my shader. The important thing here is that the light() function should be per-vertex when vertex lighting is enabled, and per-pixel when it isn't, regardless of use case.

tbh it should be flexible, because i kind of want to create an effect where it can have both per-vertex shading and a per-pixel specular highlight, it shouldn't be either A or B, it should be flexible to do anything

[PurpBatBoi]

Has any real work been done on this?

So for some of the people searching and looking for solutions related to vertex-based lighting and effects.

So for now, on godot, you're best off creating a shader that can process light on the vertex function then use scripts/code assigned to a 3D node and send the relevant information to the shader in order to have controllable vertex lights.

for this video below i used godot cpp in order to have properly flexible vertex-based light&effects

8mb.video-1Vg-AmdpLBm4.mp4

[youporkchop5]

Has any real work been done on this?

So for some of the people searching and looking for solutions related to vertex-based lighting and effects.

So for now, on godot, you're best off creating a shader that can process light on the vertex function then use scripts/code assigned to a 3D node and send the relevant information to the shader in order to have controllable vertex lights.

for this video below i used godot cpp in order to have properly flexible vertex-based light&effects
8mb.video-1Vg-AmdpLBm4.mp4

How did you do this? looking for a similar solution for my game

[insomniacUNDERSCORElemon]

Inspired by a music video* I'm thinking about an unshaded+ambient material. Fully "illuminated" faces are unshaded, otherwise faces are multiplied by ambient light (from environment) for a night-time/sundown (and streetlights at night would look cool).

Render load aside (esp. as a global effect), I assume this could be done. Pseudo-code I'm imagining is something like: vertex_color * max(light_intensity, ambient_color). Perhaps it could be done just with render modes, except I don't think there's any way to have ambient color without standard shading too (nothing like diffuse_disabled or diffuse_light_disabled, and no ambient_light_enabled which likely wouldn't pair with unshaded anyway).

* Engelwood - Crystal Dolphin (official music video). The second half (non-daylight), not the shadows.

EDIT: I've tried using light() and it seems like the ambient color is just on (not referenced separately) which makes it harder (for me at least) to work with. I was able to get half-way there, as in light area works is okay and dark area is just ambient color (not mixed with albedo). Color accuracy is also a tiny-bit off with ambient mixing into even the lightest spot, whereas pure ambient color (even with light() being empty) is darker than what I have set for some reason.