add gaussianSplat mat file for shader #7192
Conversation
|
Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes. |
| requires : [ | ||
| color, // f_dc and opacity | ||
| tangents, // rot | ||
| custom0, // scale |
There was a problem hiding this comment.
custom1-custom6 are optional, since SH order may be 0, 1 or 2:
0: only CUSTOM0
1: + CUSTOM1-3 (9 floats)
2: + CUSTOM4-6 (24 floats)
There was a problem hiding this comment.
Actually, our current encodings are as follows:
- POSITION: Vertex positions (FLOAT3)
- COLOR: DC component and opacity (FLOAT4)
- TANGENTS: Rotation quaternion (FLOAT4)
- CUSTOM0: Scale (FLOAT4)
- CUSTOM1 to CUSTOM6: SH coefficients (FLOAT4)
There was a problem hiding this comment.
Got it. So if input splat does not have f_rest, CUSTOM1-6 will not be needed. So we should not have them as requires.
There was a problem hiding this comment.
Update our current encodings as follows:
POSITION: Vertex positions (FLOAT3)
COLOR: Scale (FLOAT4)
TANGENTS: Rotation quaternion (FLOAT4)
CUSTOM0: DC component and opacity (FLOAT4)
CUSTOM1 to CUSTOM6: SH coefficients (FLOAT4)
The difference is that we use CUSTOM0 for f_dc and opacity, COLOR for scale.
The reason for this change is as following:
In the latest commit, we calculate color from f_dc and f_rest in vertex shader. But function getColor() is fragment shader only and we can not use it in vertex shader. So in order to get f_dc in vertex shader, we save f_dc in CUSTOM0 buffer rather than COLOR buffer.
After we calculate color results from f_dc and f_rest, we pass it into fragment shader by variables.
You can go to here for more details about variables. And for now, we calculate color results by calling function defined in glsl function.
If input splat does not have f_rest, the corresponding buffers will be filled with default vaule. So i think we have them as requires is ok.
| // material.baseColor.rgb = getWorldPosition(); | ||
|
|
||
| // Wrong Case3: Calculate RGB value using user world position. | ||
| // material.baseColor.rgb = getUserWorldPosition(); |
There was a problem hiding this comment.
Can you add here directly setting the color based on:
- scale
- rot1-3
- f_rest - any 3 values from SH order 1
- f_rest - any 3 values from SH order 2
Also, paste the resulting images in the PR - this will validate that the shader is able to receive the GaussianSplat data and display it.
There was a problem hiding this comment.
Sure, we are working on this.
There was a problem hiding this comment.
We have verified we can the shader can get data scale, f_rest of order 1, f_rest of order 2.
Please check the latest commit for gaussianSplat.mat.
Overall, if we can not get data, the resulting image is all black as following:
If we can get data, the resulting image has colors like following:
There was a problem hiding this comment.
And we are working on how to get rot.
There was a problem hiding this comment.
For now, we save rot in buffer Custom7.
Co-authored-by: hanyin-intel [email protected]
There was a problem hiding this comment.
[nit] separate into per SH order functions (see comment above).
Add docs (details of input, ie.. coeffs, and also link to original / reference cuda code.)
| // material.baseColor.rgb = getWorldPosition(); | ||
|
|
||
| // Wrong Case3: Calculate RGB value using user world position. | ||
| // material.baseColor.rgb = getUserWorldPosition(); |
Test case results are as following:
|
Type
Motivation and Context
Checklist:
python util/check_style.py --applyto apply Open3D code styleto my code.
updated accordingly.
results (e.g. screenshots or numbers) here.
Description