Creating btBvhTriangleMeshShape assertion failure due to quantization round trip imprecision in double precision mode

[AnyOldName3]

With Bullet compiled in double precision mode, it is possible to trigger an assertion failure when creating a btBvhTriangleMeshShape with useQuantizedAabbCompression=true and buildBvh=true. The mesh data to reproduce this is (x y z 0):

7066.77392578125 -74515.5859375 -42946.16796875 0.0
15158.3681640625 -122238.71875 -30090.10546875 0.0
-23429.798828125 -110013.65625 -29742.275390625 0.0
-19381.376953125 -120035.671875 -23467.03125 0.0
51567.65625 -108494.265625 -29972.205078125 0.0
-44093.37109375 -70996.5546875 -29742.279296875 0.0
56455.1953125 -113386.4375 -23496.54296875 0.0
28079.998046875 -62962.3359375 -42325.5234375 0.0
17003.876953125 -127847.8046875 -23444.88671875 0.0
29838.9375 -108031.75 -24090.775390625 0.0
12363.068359375 -111046.9140625 -23998.677734375 0.0
41669.8046875 -101513.6015625 -24103.857421875 0.0
-886.5219116210938 -109011.7578125 -24014.009765625 0.0
57769.59375 -75651.796875 -24105.00390625 0.0
53263.07421875 -88029.796875 -24178.16796875 0.0
77056.375 -56279.6875 -23496.54296875 0.0
58847.25390625 -66555.203125 -24046.982421875 0.0
58150.140625 -57518.70703125 -24050.88671875 0.0
-24877.751953125 -90521.9296875 -24205.046875 0.0
-16489.24609375 -100338.421875 -24178.154296875 0.0
-48317.328125 -80945.9140625 -23467.037109375 0.0
-32248.9140625 -69049.515625 -24014.013671875 0.0
-29212.8515625 -82108.34375 -24269.576171875 0.0
-41137.4375 -32844.3984375 -23467.03125 0.0
-31106.619140625 -55709.1875 -24046.990234375 0.0
-26504.025390625 -43136.09375 -24014.017578125 0.0
-18003.25390625 -32486.8046875 -24105.0078125 0.0
-2047.7220458984375 -3908.493408203125 -23467.029296875 0.0
-32674.24609375 -33004.69921875 -30271.40234375 0.0
-18879.166015625 -16717.58203125 -28512.96484375 0.0
3543.465087890625 -50190.4375 -41361.40625 0.0
5244.2783203125 -9541.328125 -29866.73828125 0.0
11021.0048828125 -17545.0703125 -23459.15234375 0.0
-7379.37841796875 -24698.56640625 -24178.1484375 0.0
9835.255859375 -19881.59375 -24178.1640625 0.0
23217.447265625 -20991.12890625 -24269.5703125 0.0
35803.9765625 -25635.16796875 -24178.119140625 0.0
45946.46484375 -33626.5546875 -24205.037109375 0.0
54199.62109375 -44846.46484375 -24178.091796875 0.0
46053.7421875 -11088.31640625 -23467.0234375 0.0
41363.05078125 -13592.3857421875 -28598.724609375 0.0
67630.953125 -46670.1796875 -29972.189453125 0.0
25712.271484375 -25295.287109375 -35029.8125 0.0
68189.1953125 -79524.2734375 -29866.7265625 0.0

This will give:

• m_bvhAabbMin {-48319.328125000000, -127849.80468750000, -42948.167968750000, 0.0000000000000000}
• m_bvhAabbMax {77062.201368489914, -3906.4934082031104, -23442.291468074767, 0.0000000000000000}
• m_bvhQuantization {0.52266869182994469, 0.52873365511694559, 3.3596542046050302, 0.0000000000000000}

During the buildTree call, getAabbMax is called, which unQuantizes a point with a quantized y coordinate of 65533. This ends up as -3906.4934082030959, which is (slightly) above the maximum y of the AABB, so when mergeInternalNodeAabb calls quantize on this value, as assertion that it's within the AABB fails. All the maths appears to be sound, it's just that floating point, even double precision, isn't perfect.

Either the value returned from unQuantize should be clamped to the AABB max so it won't overshoot, or the assertions in quantize should include an epsilon to ensure that points that are effectively equivalent to the AABB max are accepted. It may also be possible to tweak things so that the same point isn't unquantized and then requantized without modifying it at all in between, as that's wasteful.