Customising Shaders

[vorg]

Currently we can set material.vert and material.frag but that breaks as we also include them in the depth prepass shaders vert: material.vert || DEPTH_PASS_VERT. The correct (needs testing) way is to define all 3 shaders and use preprocessor to support all 3 cases

#ifdef DEPTH_PRE_PASS_ONLY
//copy pre pass shader
#elseif DEPTH_PASS_ONLY
//copy depth shader
#else
//pbr shader
#endif

That's painful.

Related

• ThreeJS discussion about string replacement before compile Material: Added onBeforeCompile() mrdoob/three.js#11475
• Use Shader Modifiers to Extend SceneKit Shading

[vorg]

After looking at that SceneKit shader modifiers our api could look e.g. like this:

material.shaderModifiers = {
   // model space, before skinning, world transform and projection
   geometry: `
      uniform float uDisplaceAmplitude;
      vertexData.position += vertexData.normal * uDisplaceAmplitude;
   `,
   // after reading all uniforms and texture values
   // before lighting, values in view space
   surface: `
      vec3 textureTriplanar(vec3 worldPos, vec3 normal) {        
         ...
      }
      surfaceData.baseColor = textureTriplanar(worldPos, worldNormal);
   `
   // after lighting (but before or after exposure and tonemapping?)
   fragment: `
      outputData.color.rgb = vec3(1.0) - outputData.color.rgb;
      // but also
      outputData.color.rgb = surfaceData.normal * 0.5 + 0.5;
   `
}

Notes

• geometry is run in vertex shader, surface and fragment in fragment shader
• surface stage assumes value in view space but we already compute values in world space e.g. for envmaps, those could be available too
• SceneKit includes another lighting stage that can execute in vertex or fragment shader
• SceneKit allows modification of texture coordinates, which means colors from textures are read after this stage
• fragment state in SceneKit is the final pixel color

Open questions:
How do you do you define uniforms?

1. we could detect and extract uniforms and functions as assumed in above example
2. we split modifier into declaration and code parts

material.shaderModifiers = {
   // model space, before skinning, world transform and projection
   geometry: {
      defs `uniform float uDisplaceAmplitude;`,
      code: `vertexData.position += vertexData.normal * uDisplaceAmplitude;`
   }
   `,

[simonharrisco]

• why geometry and surface ? is this additional jargon that might just add confusion?
• are you going to be be able to change these on the fly?
• detect and extract uniforms and functions this stinks of heartache and pain from edge cases and debugging etc

[vorg]

is this additional jargon that might just add confusion?

Hmm these are basic concepts in computer graphics. Geometry defines the shape of thing being render, and surface parameters how that shape responds to light. And they are not as much detached from renderer names and it might first seem.

geometry - it makes sense if you notice that those are raw values coming from geometry component attributes, you could also call it vertex

surface - this is excellent name to clarify where the modification happens, in our code we have one PBRData blob that contains raw values from uniforms and textures (surface data), color values computed from light based on BRDF (reflectance function output data) and final fragment shader before and after post-processing (if done inline, fragment data)

[vorg]

are you going to be be able to change these on the fly?

Same as currently recompilation happens when any of the material parameters modifying shader is changed.

detect and extract uniforms and functions this stinks of heartache and pain from edge cases and debugging etc

I think so too

[dmnsgn]

Open questions:
How do you do you define uniforms?

Could we make use of material._uniforms somehow?

With the shaderModifiers syntax above, it feels a big vague as to where custom code is added (is it at the top, uniform declaration, beginning/end of main()...).

[vorg]

@dmnsgn which part of it is vague?

If code then then way Scene Kit defines it is very clear: "after the above values are defined/read but before they are used. I.g. vertex before skinning, surface after getBaseColor() (read uniform or texture) but before applying lighting and frag at the very end.

The problem is e.g. if you want to declare function as well. Hence proposal for defs part so i can add eg. vec3 triplannarTexturing() { function just before main.

_uniforms is just cache
uniforms - is list of values if you have declared new uniforms in you modified shader

[vorg]

Code injection points
geometry: material.vert.js#L134 and depth-pass.vert.js#L124
surface: material.frag.js#L203
fragment: material.frag.js#L253

[dmnsgn]

@dmnsgn which part of it is vague?

How do I handle modifying vertex position before or after instancing if I only have one injection point? Same if I need to modify data.normalWorld before or after.
Not 100% sure these are valid use cases but worth thinking of how the injection point could be limiting.

[vorg]

Yes, it's limiting but also make things clear how they work and provides immunity from underlying shader code changes. Otherwise things get complicated fast just to cover remaining 5-10% of use cases and we end up where we are now "copy whole shader, modify what you want and hope it works in the X.1 release" https://twitter.com/bonzajplc/status/1177715088079368194?s=21

[vorg]

Another good thing i see coming out of it we could clean up code a bit as now values are defined and computed all over the place at different stages. e.g. reflectionWorld added to global PBRData struct only if we have reflection probes material.frag.js#L207

[vorg]

A good idea is to write a list of use cases:

Easy

• tri planar texture mapping -> surface
• generative colors (base color, metallic) -> surface
• toon shading -> fragment
• showing debug information (normals, ao, texture coords) -> fragment
• picking color id -> fragment

No clear how to prototype

• displacement mapping -> vertex + surface (frag based normal recalc?)
• subsurface scattering -> ?
• irradiance volumes -> more of a lighting modifier
• projective texturing (projection mapping, light cookies) -> post lighting blend?
• particle depth write -> fragment?

[vorg]

@dmnsgn to answer your earlier question about uniforms and varying. Similar to what thi.ng/umbrella is doing we could move to more modular shader definitions like here : https://github.com/thi-ng/umbrella/blob/master/examples/webgl-ssao/src/shaders.ts#L20

This way custom shader would be pushing it's uniforms to material.uniforms defs array and adding custom functions to eg. material.lib list of snippets and then custom part to material.modifiers.surface nice and clean... even possibly webgl2 compilable

[vorg]

For now this is what's happening

Renderer following shader chunks

this.shaders = {
    chunks: SHADERS_CHUNKS,
    pipeline: {
      material: {
        vert: PBR_VERT,
        frag: PBR_FRAG
      }
    }
  }

We customize material shader

material.frag = renderer.shaders.pipeline.material.frag.replace(...)`

but that then get's used for depth pre-pass

 return {
      flags: flags,
      vert: material.vert || DEPTH_PASS_VERT,
      frag: material.frag || DEPTH_PRE_PASS_FRAG
    }

What i think should happen is

depth prepass - renders geometry and outputs depth and normals then used for SSAO
depth pass - simplified shader for shadow maps depth rendering
material - normal pbr with full lighting

material.frag = `#ifdef DEPTH_PRE_PASS_ONLY
${renderer.shaders.pipeline.depthPrePass.frag.replace(...)}
#elseif DEPTH_PASS_ONLY
${renderer.shaders.pipeline.depthPass.frag.replace(...)}
#else
${renderer.shaders.pipeline.material.frag.replace(...)}
#endif