No stencil buffer write with direct3d11/12

[lampysprites]

added hxcpp results in the end

Discussed in #1415

^{Originally posted by lampysprites April 20, 2022}
Hello!

I'm trying to create a mask for drawing using stencil buffer. Works in html5 as intended but something seems to be missing in other cases. In addition to the example below, tried drawing to an explicitly created Depth24Stencil8 image before the buffer.

Appreciate the help.

os: windows10
graphics: geforce gtx 560
kha: main @ f4fd02a5fd9a4bbc8133d985d1646a16498c44de

How it looks

Code

package;

import kha.Assets;
import kha.Framebuffer;
import kha.graphics4.PipelineState;
import kha.graphics4.VertexData;
import kha.graphics4.VertexStructure;
import kha.math.FastMatrix3;
import kha.Shaders;
import kha.System;

using kha.graphics2.GraphicsExtension;


class Main {
	static var pipeMask:PipelineState;
	static var pipeOver:PipelineState;


	static function render(fb:Framebuffer) {
		var t = kha.System.time,
			g = fb.g2;

		g.begin();

		g.pipeline = pipeMask;
		g.color = 0xffffffff;
		g.fillCircle(400, 300, 200);

		g.flush();

		g.pipeline = pipeOver;
		g.color = 0x88ff00ff;
		g.pushTransformation(FastMatrix3.translation(400, 300).multmat(FastMatrix3.rotation(t)).multmat(FastMatrix3.translation(150, 0)).multmat(FastMatrix3.rotation(-2 * t)));
		g.fillRect(-150, -150, 150, 150);
		g.popTransformation();

		g.end();
	}


	public static function main() {
		System.start({title: "Kha", width: 800, height: 600}, function (_) {
			Assets.loadEverything(function () {
				var structure = new VertexStructure();
				structure.add("vertexPosition", VertexData.Float32_3X);
				structure.add("vertexColor", VertexData.UInt8_4X_Normalized);

				pipeMask = new PipelineState();
				pipeMask.inputLayout = [structure];
				pipeMask.vertexShader = Shaders.painter_colored_vert;
				pipeMask.fragmentShader = Shaders.painter_colored_frag;

				pipeMask.stencilBothPass = Replace;
				pipeMask.stencilMode = Always;
				// pipeMask.colorWriteMask = false;
				pipeMask.stencilReferenceValue = Static(1);

				pipeMask.compile();

				pipeOver = new PipelineState();
				pipeOver.inputLayout = [structure];
				pipeOver.vertexShader = Shaders.painter_colored_vert;
				pipeOver.fragmentShader = Shaders.painter_colored_frag;

				pipeOver.stencilMode = Equal;
				pipeOver.stencilReferenceValue = Static(1);
	
				pipeOver.compile();

				// Avoid passing update/render directly, so replacing them via code injection works
				System.notifyOnFrames(function (framebuffers) { render(framebuffers[0]); });
			});
		});
	}
}

Seems that it is indeed directx thing. After trying different options, here's the results

• direct3d11 doesn't show the box at all
  - hen clearing with 1's, and circle being 0s, looks same as krom above. kinda as if only reading works?
• opengl - all works fine
• direct3d12 - same as krom, no masking
• direct3d9 - empty window even with default pipeline

using cl.exe == 19.29.30133 x64 in case that matters