Visualizing gaussian splats #7194

billamiable · 2025-03-08T06:34:10Z

Type

Bug fix (non-breaking change which fixes an issue): Fixes #
New feature (non-breaking change which adds functionality). Resolves #
Breaking change (fix or feature that would cause existing functionality to not work as expected) Resolves #

Motivation and Context

This PR, along with two other PRs (#7192, #7193), will fix #7154

Checklist:

I have run python util/check_style.py --apply to apply Open3D code style
to my code.
This PR changes Open3D behavior or adds new functionality.
- Both C++ (Doxygen) and Python (Sphinx / Google style) documentation is
  updated accordingly.
- I have added or updated C++ and / or Python unit tests OR included test
  results (e.g. screenshots or numbers) here.
I will follow up and update the code if CI fails.
For fork PRs, I have selected Allow edits from maintainers.

Description

will create a new PR for this

update-docs · 2025-03-08T06:34:14Z

Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes.

benjaminum

Thanks for the PR! Please have a look at the comments

benjaminum · 2025-03-10T11:12:22Z

cpp/open3d/visualization/rendering/filament/GaussianSplatBuffers.cpp

+    for (const auto& check_item : check_list) {
+        if (geometry_.HasPointAttr(check_item) &&
+            geometry_.GetPointAttr(check_item).GetDtype() != core::Float32) {
+            auto check_item_instance = geometry_.GetPointAttr(check_item);


I think we need to check the attribute shapes here or in Pointcloud::IsGaussianSplat(). In ConstructBuffers() assumes that the attributes have specific sizes. f_rest can have different shapes (num_verts, num_coeffs, 3) with num_coeffs = 3, 8, or 15

Agree. PR #7153 adds this checking support for IsGassianSplat(). We can skip the checks here.

We are adopting similar logic inside PointCloudBuffers.cpp, see implementation here.

Agree. PR #7153 adds this checking support for IsGassianSplat(). We can skip the checks here.

@ssheorey
We should use the same checks as for IO here but I am not sure if we can skip checks here since the user can create a GS point cloud for visualization in memory. Am I missing something?

IsGaussianSplat() is already called before this ctor is run (see line 255 in FilamentGeometryBuffersBuilder.cpp). So we don't need to repeat the check.
We should add a commment about preconditions to this function:

///IsGaussianSplat() has verified that this point cloud contains the necessary data for a Gaussian Splat.

I think we can update this function after PR #7153 is merged.

cpp/open3d/visualization/rendering/filament/GaussianSplatBuffers.cpp

cpp/open3d/t/geometry/PointCloud.cpp

ssheorey · 2025-03-10T22:44:05Z

cpp/open3d/visualization/rendering/filament/FilamentResourceManager.cpp

@@ -1017,6 +1019,25 @@ void FilamentResourceManager::LoadDefaults() {
    lit_mat->setDefaultParameter("anisotropyMap", texture, default_sampler);
    materials_[kDefaultLit] = BoxResource(lit_mat, engine_);

+    const auto gaussian_path = resource_root + "/gaussianSplat.filamat";
+    auto gaussian_mat = LoadMaterialFromFile(gaussian_path, engine_);


TODO: Verify that color space is correct, i.e. input SH coeffs encode color in sRGB color space.

Also, are the rest of the values required by filament? They will not be used in our custom shaders.

Not sure what this means. Is it supposed to be the same as #7192 (comment)?

sRGB is a compressed color space commonly used for encoding color in images. Very approximately it's linearRGB = sRGB ^ 2.2 (color values in range 0-1). See the defaultUnlit.mat file. For rendering, we need to map back from sRGB to linear RGB. Otherwise, the images may look overly bright. If the rendering looks too bright compared to the reference viewer, this is the cause.

2nd question was about the PBR properties set below (baseCoor, baseRoughness, .....). We should remove them unless filament gives a warning or error, since we do not need them.

After some tests, i think color space is linear RGB based on the following:

the tpye of f_dc and f_rest is float32 in .npz file.

some of data in f_dc is negative.

I try to use linear_to_sRGB and sRGB_to_linear in .mat file
raw picture:

linear_to_sRGB:

sRGB_to_linear

About the negative f_dc values: this is due to "ringing" artifacts (wikipedia) that you see when trying to represent a high frequency signal with low frequency approximation. Spherical harmonics are 2D analog of Fourier series and exhibit the same issue. To mitigate this issue, we should clamp the final output color values to the [0, 1] range. Apply any color processing (e.g. linear_to_sRGB) after the clamping.

ssheorey · 2025-03-10T22:50:32Z

cpp/open3d/visualization/rendering/filament/GaussianSplatBuffers.cpp

+    for (const auto& check_item : check_list) {
+        if (geometry_.HasPointAttr(check_item) &&
+            geometry_.GetPointAttr(check_item).GetDtype() != core::Float32) {
+            auto check_item_instance = geometry_.GetPointAttr(check_item);


Agree. PR #7153 adds this checking support for IsGassianSplat(). We can skip the checks here.

cpp/open3d/visualization/rendering/filament/GaussianSplatBuffers.cpp

Co-authored-by: hanyin-intel [email protected]

cpp/open3d/t/geometry/PointCloud.cpp

cpp/open3d/visualization/rendering/filament/GaussianSplatBuffers.cpp

cpp/open3d/t/geometry/PointCloud.cpp

ssheorey · 2025-03-18T06:09:30Z

cpp/open3d/t/geometry/PointCloud.cpp

@@ -1413,9 +1413,10 @@ int PointCloud::GaussianSplatGetSHOrder() const {
                    "SH degree higher than 2 is not supported. The SH degree "
                    "of in input data is {}, we set it as 2.",
                    deg);
+            deg = 2;


Hi @billamiable we don't support sh degree > 2 for rendering, but we do support it for other operations (reading and writing 3DGS files, and in the future editing 3DGS data). I would skip this warning here, and instead only add it if the user tries to display a 3DGS point cloud with 3 or higher degree.

zhangyang-intel and others added 8 commits March 8, 2025 13:44

enable gaussian rendering based on filament

927463e

rename gaussian to gaussianSplat

7a5cab6

update buffer allocation

141a30b

update construct function

9b76ad4

overwrite TGaussianSplatBuffersBuilder class

b8ee922

add check logic for gaussian

0e8f9b0

remove .mat related files

2b25cb0

will create a new PR for this

add C++ API doc

8461701

benjaminum mentioned this pull request Mar 10, 2025

[Do not merge] enable gaussian rendering based on filament #7186

Draft

9 tasks

benjaminum requested changes Mar 10, 2025

View reviewed changes

ssheorey added the hackathon label Mar 10, 2025

ssheorey requested changes Mar 10, 2025

View reviewed changes

billamiable changed the title ~~Doc for visualizing gaussian splats~~ Visualizing gaussian splats Mar 12, 2025

zhangyang-intel added 6 commits March 12, 2025 15:29

update gaussian buffer

46b7657

Co-authored-by: hanyin-intel [email protected]

remove unnecessary, if-else

34d49a8

use template

dc68c3c

add PointCloud::GaussianSplatGetSHOrder() function

e80c591

allocate buffer dynamically

530f39b

add comment

c67ecc0

ssheorey reviewed Mar 13, 2025

View reviewed changes

cpp/open3d/t/geometry/PointCloud.cpp Outdated Show resolved Hide resolved

cpp/open3d/visualization/rendering/filament/GaussianSplatBuffers.cpp Outdated Show resolved Hide resolved

zhangyang-intel added 6 commits March 14, 2025 10:39

simplify GaussianSplatGetSHOrder format

9eb09b6

clang format

665b72b

update logic to calculate f rest buffer count

4250a9c

remove useless memset

1d580c3

bug fix

ef2c615

check sh degree in GaussianSplatGetSHOrder()

86d4727

ssheorey reviewed Mar 17, 2025

View reviewed changes

cpp/open3d/t/geometry/PointCloud.cpp Outdated Show resolved Hide resolved

zhangyang-intel added 2 commits March 18, 2025 10:13

bug fix

9d3d639

bug fix

ce708dc

zhangyang-intel added 2 commits March 18, 2025 14:02

fix bug

897601d

upate sh_degree in MapRecord

a24fda5

ssheorey reviewed Mar 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Visualizing gaussian splats #7194

Visualizing gaussian splats #7194

billamiable commented Mar 8, 2025 •

edited

Loading

update-docs bot commented Mar 8, 2025

benjaminum left a comment

benjaminum Mar 10, 2025

ssheorey Mar 10, 2025

billamiable Mar 11, 2025

benjaminum Mar 11, 2025 •

edited

Loading

ssheorey Mar 11, 2025

zhangyang-intel Mar 13, 2025 •

edited

Loading

ssheorey Mar 10, 2025

billamiable Mar 11, 2025

ssheorey Mar 11, 2025

zhangyang-intel Mar 14, 2025

ssheorey Mar 14, 2025 •

edited

Loading

ssheorey Mar 10, 2025

ssheorey Mar 18, 2025

Visualizing gaussian splats #7194

Are you sure you want to change the base?

Visualizing gaussian splats #7194

Conversation

billamiable commented Mar 8, 2025 • edited Loading

Type

Motivation and Context

Checklist:

Description

update-docs bot commented Mar 8, 2025

benjaminum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjaminum Mar 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhangyang-intel Mar 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssheorey Mar 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

billamiable commented Mar 8, 2025 •

edited

Loading

benjaminum Mar 11, 2025 •

edited

Loading

zhangyang-intel Mar 13, 2025 •

edited

Loading

ssheorey Mar 14, 2025 •

edited

Loading