Depth Pre-Pass transparency breaks batching

[bikemurt]

Tested versions

-Reproducible in 4.4-stable, 4.4-rc3, 4.4-rc2, 4.4-rc1, 4.4-dev1
-Not reproducible in 4.3-stable

System information

Windows 11 - Vulkan Forward+ - NVidia GeForce RTX 3050 Ti Laptop GPU

Issue description

Batching breaks in the Forward+ renderer when a material has the Transparency set to Depth Pre-Pass. I tested both the RenderingServer API as well as the node approach (MeshInstance3D) and the result is the same.

In Godot 4.3-stable, we can see the expected result, 3 draw calls (shadows disabled for simplicity):

In Godot 4.4-dev1, we see the regression (showing 2433 draw calls):

Steps to reproduce

The core of the code is to instantiate a bunch of meshes with a few materials - we expect the batcher to be able to sort these properly and reduce draw calls in Forward+.

@onready var node_3d: Node3D = $Node3D
const BOX_MESH = preload("res://box_mesh.tres")
const MATERIALS := {
	0: preload("res://red.tres"),
	1: preload("res://blue.tres"),
	2: preload("res://green.tres")
}
func node_test() -> void:
	var mat := 0
	
	for x in range(100):
		for z in range(100):
			var m := MeshInstance3D.new()
			m.mesh = BOX_MESH
			
			var t := Transform3D()
			t.origin = Vector3(-100,0,-100) + Vector3(2 * x, 0, 2 * z)
			m.transform = t
			m.set_surface_override_material(0, MATERIALS[mat])
			
			node_3d.add_child(m)
			
			mat += 1
			if mat >= 3: mat = 0

Minimal reproduction project (MRP)

instancing-test.zip

Run the MRP, check the Total Draw Calls from the Monitors tab.

[clayjohn]

Looks like the problem was introduced in 4.4 dev1

Edit: Traced it back to #95640. In 4.3 we didn't include draw calls from transparent objects in the monitor. That bug was fixed by #95640

Looking at performance, on my M2 MBP, Performance is 5%-10% faster in 4.4 than it was in 4.3 with this test scene.

[bikemurt]

Interesting- so what I thought was a performance loss was just a bug with the counters, which has been fixed. I confirmed the actual rendering behaviour is the same in RenderDoc between 4.3-stable and 4.4-stable (roughly the same number of calls to vkCmdDrawIndexed).

Is an optimization even possible here? Maybe there is something about depth pre-pass that I don't understand. To my knowledge there's a pre-pass where alpha values below 0.99 are discarded, this is rendered as opaque for the depth buffer (and shadows?). Then the transparency is rendered back-to-front after as a part of the main render. Is there some fundamental conflict between sorting for depth pre-pass and sorting for the main render pass? My assumption has been that the front-to-back sorting for opaque materials was always handled separately from the back-to-front sorting for transparent draw calls.

Edit: depth layer is in the sort key, and presumably during depth pre-pass (alpha) is where transparent meshes get stacked back-to-front:

godot/servers/rendering/renderer_rd/forward_clustered/render_forward_clustered.h

Lines 487 to 499 in a77a28c

	struct {
	uint64_t lod_index : 8;
	uint64_t uses_softshadow : 1;
	uint64_t uses_projector : 1;
	uint64_t uses_forward_gi : 1;
	uint64_t uses_lightmap : 1;
	uint64_t depth_layer : 4;
	uint64_t surface_index : 8;
	uint64_t priority : 8;
	uint64_t geometry_id : 32;
	uint64_t material_id : 32;
	uint64_t shader_id : 32;

When checking for batching, we check for identical keys between calls (after sorting):

godot/servers/rendering/renderer_rd/forward_clustered/render_forward_clustered.cpp

Line 834 in a77a28c

if (prev_surface != nullptr && !cant_repeat && prev_surface->sort.sort_key1 == surface->sort.sort_key1 && prev_surface->sort.sort_key2 == surface->sort.sort_key2 && inst->mirror == prev_surface->owner->mirror && repeats < RenderElementInfo::MAX_REPEATS) {

Makes me wonder though - is being on a different depth layer an actual render state change? It would be interesting to mask out the depth layer information when checking to see if two keys are equal. I may give this a try.