Reproduce Reconstruction Results on the GSO dataset

[lzqsd]

Thanks for open source this great work to community!

I was running the inference demo on my local machine using images rendered from the GSO dataset. However, I cannot reproduce the results shown in the paper. Followings are 2 example results I obtained from the interactive demo. Would you please tell me if there is anything i should pay attention to? I can also send the testing images to you directly. Really appreciate your help and guidance in advance!

[ncan33]

I am also having a really hard time reproducing the results on the paper. The paper has Figure 1 like this:

The images for this figure are provided by the authors in the "examples" folder in the HuggingFace repository:

Using these images, I called run_views_to_3d:

import torch
import glob
import os
from freesplatter.webui.runner import FreeSplatterRunner

if torch.cuda.is_available():
    runner = FreeSplatterRunner(device="cuda")
elif not torch.cuda.is_available():
    raise ExceptionType("Do not even try running this without a GPU.")

image_paths = sorted(glob.glob(os.path.join("gt_test", "images/*.png")))

images_vis, gs_vis_path, video_path, mesh_path, pose_fig = runner.run_views_to_3d(
    image_files=image_paths,
    do_rembg=False,
    gs_type='2DGS',
    cache_dir="gt_test"
)

But with these images, this is the reconstruction I get:

gs.3.mp4

There's a big difference between this result and what we are seeing in the paper. Not sure how authors produced Figure 1. I am wondering if they took the pose estimates (which are created from plugging predicted Gaussians into the PnP-RANSAC solver) and plugged it into a conventional 3DGS reconstruction? The reported reconstructions are super quality but the predicted Gaussians I get from run_views_to_3d() and run_freesplatter_object() do not match the results I am seeing in the paper.

Authors, am I doing something wrong here? I cannot reproduce the results in the paper.

[bluestyle97]

@ncan33 The results you get look really weird. Did you initialize the environment following the guidance of this repo?

[ncan33]

@bluestyle97 I did set up the environment following the guidance of the repo. All of the dependencies installed and work well. I ran some experiments this morning. For the diffusion-based reconstruction, I am getting really strong results using run_img_to_3d(). For example, for this input image:

I get this diffusion synthesis:

And the reconstruction looks great:

gs.12.mp4

However, when I reconstruct using the real images of the object taken from multiple angles using run_views_to_3d(), the reconstruction looks bad:

gs.10.mp4

The real images of the object taken from multiple angles are:

I would expect that the reconstruction with the real images would outperform the reconstruction with the diffusion images (except for the bottom of the object maybe). However, it seems like the single-image reconstruction is outperforming the multi-image reconstruction.

I might be doing something wrong with the data preparation for run_views_to_3d(). To isolate the issue, I will take the diffusion images generated from run_img_to_3d() and plug input them into run_views_to_3d(), and see if I get the same result as the first video above (gs.12.mp4), or if it will look like the second video (gs.10.mp4).

I think FreeSplatter is an excellent, novel contribution. I think I am probably doing something wrong with the data prep. I will investigate more and submit and update with my findings.

[ncan33]

I think there were some inconsistencies in the way that run_views_to_3d() does data prep vs. the way run_img_to_3d() does data prep. I copied the data prep logic from run_img_to_3d() and I got this reconstruction with the real images of the object taken from multiple angles:

gs.18.mp4

This seems more reasonable. It's still a little more fuzzy/cloudy than the diffusion one, but that might be attributed to differences in viewing angles maybe. Not sure. For a more scientific comparison, I should capture photos of this mint box from the same exact viewing angles as how diffusion synthesized it.

After this fix, I re-ran the "Transformers" model and this is what I got:

gs.17.mp4

This is substantially better than before. Still a little cloudy, especially on the back. I think I still need to fix some stuff and make it better match the data prep of run_img_to_3d()

Thanks for this excellent paper! I will give another update if I can push the reconstruction quality.

[bluestyle97]

@ncan33 FreeSplatter-O was trained on Objaverse-rendered images. Each model was normalized into [-1, 1]^3 unit cube, and cameras are randomly sampled around the object, all facing the bounding box center [0, 0, 0]. This data rendering strategy follows the typical practice of previous LRMs, but it indeed leads to a camera distribution mis-matching problem when using real-captured images for inference, whose cameras often do not share a common "center". That's why you got better results using diffusion-generated images than real photos, since the multi-view diffusion model (Hunyuan3D or Zero123++) was also trained with centered cameras, making their generated views conform to the camera distribution towards the center. To improve the results, you can refer to a follow-up work named UAV-Splatter, which alleviate this issue by adding lora layers to FreeSplatter and fine-tuning with more diverse image.
Besides, the results you got with the example "Transformer" images look still significantly worse than what we got. I have no idea on the exact reason for now. Maybe I can look into this issue this weekend. Here is our reconstruction results for reference: Gaussians & video.
Thanks for your interest on our work and the valuable explorations!

[ncan33]

Thank you for this clear explanation and sharing UAV-Splatter. Thank you also for sharing the reconstructed Gaussians and video. Excited to follow your future works!

[ncan33]

Follow up to this: I was able to exactly reproduce the results of this paper on a variety of images. For example, here are my results on the "Transformers" model. There is no cloudiness whatsoever, and it is identical the the results of the authors. My results:

gs.mp4

[kk627627]

@ncan33

Follow up to this: I was able to exactly reproduce the results of this paper on a variety of images. For example, here are my results on the "Transformers" model. There is no cloudiness whatsoever, and it is identical the the results of the authors. My results:

gs.mp4

What specific strategies and improvements did you implement to achieve good results when inputting multi-view images?