Description
Currently, the way we implement CPU picking for feature id textures and feature textures involves giving Unreal's line-trace query system (part of the physics system) an extra copy of all the texture coordinates for a given primitive. Later, a user might raycast a tile's geometry and they will be able to retrieve a UV coordinate (through Unreal) that corresponds to the metadata texture they are interested in. They can then use this UV coordinate with our BP metadata API to retrieve the actual feature data for the hit location.
This is nice except for the fact that we already have a CPU copy of the texture coordinates that we keep around. One option might be to clear our CPU copy of vertex attributes such as UVs once we've created all necessary CPU (physics) and GPU (rendering) resources in Unreal. This is non-trivial, since lots of the vertex attributes are dumped into the same contiguous buffers as other data (e.g., per-feature data). We would have to do a reference-hunting within the gltf for buffer ranges to figure out what we can remove (pretty much a mini garbage collector). Even if we could surgically remove large chunks from the buffer, we’d need to go back through and remap any buffer views into other sections of the buffer.
Luckily, at least in the case of metadata textures, there is a simpler option that lets us avoid giving Unreal’s physics system another copy of the UVs.
- We could instead give Unreal’s physics system a plain triangle mesh.
- Unreal raycast hit results will now still give us a triangle id, but no uv coordinates.
- We can provide a BP api to take a triangle id + hit location in Unreal world space and calculate the interpolated UV coordinates for the hit that corresponds to the desired feature id texture / feature texture.
This BP API would have to do the following:
- Find the corresponding triangle (by id) in our vertex buffer.
- Find the barycentric coordinates of the hit position on the triangle.
- Find the UV coordinates for the triangle vertices from the UV set that corresponds to the feature texture / feature id texture in question.
- Interpolate the UV coordinates for the hit position using the barycentric coordinates.
- Then plug into the existing functionality that takes UV coordinates for a texel and retrieves the actual per-feature data.
This would fix / improve a few other things as well:
- This would fix UE5 picking of feature id texture / feature texture not working. Currently we don't send Chaos any UV coordinates.
- This would also fix the issue where metadata texture picking only works if the corresponding feature texture / feature id texture is added to the
EncodedMetadataComponent. Unfortunately, doing so has the side effect of sending the extra texture coordinates to the renderer as well which may be unneeded if no styling is being done.