AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Views

Project Page | Paper

Lihan Jiang*, Yucheng Mao*, Linning Xu, Tao Lu, Kerui Ren, Yichen Jin, Xudong Xu, Mulin Yu, Jiangmiao Pang, Feng Zhao, Dahua Lin, Bo Dai^†

Overview

Starting from a set of uncalibrated images, a transformer-based geometry encoder is followed by three decoder heads: F_G, F_D, and F_C, which respectively predict the Gaussian parameters (μ, σ, r, s, c), the depth map D, and the camera poses p. These outputs are used to construct a set of pixel-wise 3D Gaussians, which is then voxelized into pre-voxel 3D Gaussians with the proposed Differentiable Voxelization module. From the voxelized 3D Gaussians, multi-view images and depth maps are subsequently rendered. The rendered images are supervised using an RGB loss against the ground truth image, while the rendered depth maps, along with the decoded depth D and camera poses p, are used to compute geometry losses. The geometries are supervised by pseudo-geometry priors obtained by the pretrained VGGT.

The code and checkpoints are coming soon~ 🙂‍↕️