Terrain render

[Retera]

When I am using a Java port of HiveWE terrain render on Warsmash, I encountered a visual issue with the way tiles were merged together in mosaics when a square of terrain had more than two types of tiles touching it. It appeared that a possible solution might be to change this line:
https://github.com/stijnherfst/HiveWE/blob/master/src/Base/Terrain.cpp#L541

At that point in the code, instead of checking if the corner was equal to the current value, I checked if it was greater than current value:

Before:

		index[0] = bottom_right == texture;
		index[1] = bottom_left == texture;
		index[2] = top_right == texture;
		index[3] = top_left == texture;

After:

		index[0] = bottom_right >= texture;
		index[1] = bottom_left >= texture;
		index[2] = top_right >= texture;
		index[3] = top_left >= texture;

In the specific edge case on the specific map I was testing with, this change appeared to make the display of the map look more War3 alike. But I do not know if this creates other problems in other edge cases, and it would be worth more testing before merging it to HiveWE.

Also, as a sidenote, although it occurred to me to document this today, in the past two years while I was using my Java port of HiveWE terrain there were times I made other undocumented changes without making a pull request back to HiveWE. Sorry about this. One big one that I remember was that all cliff models were wrong, because they were being rendered 90 degrees rotated from what I saw in Warcraft 3. But they are all so similar that this was not apparent -- and in many cases the original HiveWE rotation looked passable -- until a lot of time and scrutiny was spent looking at the terrain.

[Retera]

Upon further review, I am beginning to think that this change is not correct. It fixes a specific weird case in a specific map that I was making that used entirely custom terrain textures, but on some original Warcraft 3 terrain the == and not the >= appears correct.

[Retera]

I found that my results looked better, but also I think wouldn't break existing cases, if I left the above listed index assigning code the same but changed the blending behaviors in GLSL for multiple texture layers to use

	vec4 layerFragment = get_fragment(texture_indices.r & 31u, vec3(UV, texture_indices.r >> 5));
	color = layerFragment;
	layerFragment = get_fragment(texture_indices.g & 31u, vec3(UV, texture_indices.g >> 5));
	color = mix(color, vec4(layerFragment.rgb, 1), layerFragment.a);
	layerFragment = get_fragment(texture_indices.b & 31u, vec3(UV, texture_indices.b >> 5));
	color = mix(color, vec4(layerFragment.rgb, 1), layerFragment.a);
	layerFragment = get_fragment(texture_indices.a & 31u, vec3(UV, texture_indices.a >> 5));
	color = mix(color, vec4(layerFragment.rgb, 1), layerFragment.a);

Conceptually, adding the bottom layer first and then adding more up from below similar to the blend function of multiple triangles on top of each other, appeared to look better. I haven't gone deep into the original math that HiveWE originally used to understand how it is different from this -- they are quite similar -- but this one is looking better for me now.

[stijnherfst]

Can you elaborate on this

One big one that I remember was that all cliff models were wrong, because they were being rendered 90 degrees rotated from what I saw in Warcraft 3. But they are all so similar that this was not apparent -- and in many cases the original HiveWE rotation looked passable -- until a lot of time and scrutiny was spent looking at the terrain.

If I remove the rotation code then they will all be wrongly rotated. I've tried different variations of assembling the cliff model path but none seem to work without needing either an offset or rotation. Afaict you're using the same cliff model path code and do rotation in the shader in Warsmash.