Adding docs for feature-splatting as an external method (#3557)

Add support for feature splatting as an external method

Author: RogerQi

docs/index.md

@@ -153,6 +153,7 @@ This documentation is organized into 3 parts:
 - [SIGNeRF](nerfology/methods/signerf.md): Controlled Generative Editing of NeRF Scenes
 - [K-Planes](nerfology/methods/kplanes.md): Unified 3D and 4D Radiance Fields
 - [LERF](nerfology/methods/lerf.md): Language Embedded Radiance Fields
+- [Feature Splatting](nerfology/methods/feature_splatting.md): Gaussian Feature Splatting based on GSplats
 - [Nerfbusters](nerfology/methods/nerfbusters.md): Removing Ghostly Artifacts from Casually Captured NeRFs
 - [NeRFPlayer](nerfology/methods/nerfplayer.md): 4D Radiance Fields by Streaming Feature Channels
 - [Tetra-NeRF](nerfology/methods/tetranerf.md): Representing Neural Radiance Fields Using Tetrahedra

docs/nerfology/methods/feature_splatting.md

@@ -0,0 +1,87 @@
+# Feature Splatting
+
+<h4>Feature Splatting</h4>
+
+```{button-link} https://feature-splatting.github.io/
+:color: primary
+:outline:
+Paper Website
+```
+
+```{button-link} https://github.com/vuer-ai/feature-splatting/
+:color: primary
+:outline:
+Code
+```
+
+<video id="teaser" muted autoplay playsinline loop controls width="100%">
+    <source id="mp4" src="https://feature-splatting.github.io/resources/basic_ns_demo_feature_only.mp4" type="video/mp4">
+</video>
+
+**Feature Splatting distills SAM-enhanced CLIP features into 3DGS for segmentation and editing**
+
+## Installation
+
+First install nerfstudio dependencies. Then run:
+
+```bash
+pip install git+https://github.com/vuer-ai/feature-splatting
+```
+
+## Running Feature Splatting
+
+Details for running Feature Splatting (built with Nerfstudio!) can be found [here](https://github.com/vuer-ai/feature-splatting).
+Once installed, run:
+
+```bash
+ns-train feature-splatting --help
+```
+
+Currently, we provide the following variants:
+
+| Method              | Description                                                     | Memory | Quality |
+| -----------         | -----------------------------------------------                 | ------ | ------- |
+| `feature-splatting` | Feature Splatting with MaskCLIP ViT-L/14@336px and MobileSAMv2  | ~8 GB  | Good    |
+
+Note that the reference features used in this version are different from the version used in the paper in two ways
+
+- The SAM-enhanced CLIP features are computed using MobileSAMv2, which is much faster than original SAM but slightly less accurate.
+- The CLIP features are computed only on the image-level.
+
+## Method
+
+Feature splatting distills CLIP features into 3DGS by view-independent rasterization, which allows open-vocabulary 2D segmentation and open-vocabulary 3D segmentation of Gaussians directly in the 3D space. This implementation supports simple editing applications by directly manipulating Gaussians.
+
+### Reference feature computation and joint supervision
+
+Feature splatting computes high-quality SAM-enhanced CLIP features as reference features. Compared to coarse CLIP features (such as those used in LERF), Feature splatting performs an object-level masked average pooling of the features to refine the boundary of objects. While the original ECCV'24 paper uses SAM for part-level masks, this implementation uses MobileSAMv2 for much faster reference features computation, which we hope would encourage downstream applications that require real-time performance.
+
+In addition to SAM-enhanced features, we also found that using DINOv2 features as a joint supervision helps regularize the internal structure of objects, which is similar to findings in existing work.
+
+### Scene Editing
+
+Thanks to the explicit representation of 3DGS, grouped Gaussians can be easily manipulated. While the original ECCV'24 paper proposes a series of editing primitives, to avoid introducing excessive dependencies or hacks, we support a subset of editing primitives in this implementation:
+
+Rigid operations
+- Floor estimation (for intuitive rotation and gravity estimation)
+- Translation
+- Transparent (highlights segmented object and turns background Gaussians transparent)
+- Rotation (yaw only w.r.t. estimated ground)
+
+Non-rigid operations
+- Sand-like melting (based on Taichi MPM method)
+
+<video id="teaser" muted autoplay playsinline loop controls width="100%">
+    <source id="mp4" src="https://feature-splatting.github.io/resources/ns_editing_compressed.mp4" type="video/mp4">
+</video>
+
+If you find our work helpful for your research, please consider citing
+
+```none
+@inproceedings{qiu-2024-featuresplatting,
+    title={Language-Driven Physics-Based Scene Synthesis and Editing via Feature Splatting},
+    author={Ri-Zhao Qiu and Ge Yang and Weijia Zeng and Xiaolong Wang},
+    booktitle={European Conference on Computer Vision (ECCV)},
+    year={2024}
+}
+```

docs/nerfology/methods/index.md

@@ -34,6 +34,7 @@ The following methods are supported in nerfstudio:
     SIGNeRF<signerf.md>
     K-Planes<kplanes.md>
     LERF<lerf.md>
+    Feature-Splatting<feature_splatting.md>
     Mip-NeRF<mipnerf.md>
     NeRF<nerf.md>
     Nerfacto<nerfacto.md>

nerfstudio/configs/external_methods.py

@@ -93,6 +93,21 @@ class ExternalMethod:
     )
 )
 
+# Feature Splatting
+external_methods.append(
+    ExternalMethod(
+        """[bold yellow]Feature-Splatting[/bold yellow]
+For more information visit: https://docs.nerf.studio/nerfology/methods/feature_splatting.html
+
+To enable Feature Splatting, you must install it first by running:
+  [grey]pip install git+https://github.com/vuer-ai/feature-splatting[/grey]""",
+        configurations=[
+            ("feature-splatting", "Feature Splatting with MaskCLIP ViT-L/14@336px, DINOv2 ViT-S/14, and MobileSAMv2"),
+        ],
+        pip_package="git+https://github.com/vuer-ai/feature-splatting",
+    )
+)
+
 # Tetra-NeRF
 external_methods.append(
     ExternalMethod(
@@ -213,7 +228,7 @@ class ExternalMethod:
 For more information visit https://docs.nerf.studio/nerfology/methods/zipnerf.html
 
 To enable Zip-NeRF, you must install it first by running:
-  [grey]pip install git+https://github.com/SuLvXiangXin/zipnerf-pytorch#subdirectory=extensions/cuda 
+  [grey]pip install git+https://github.com/SuLvXiangXin/zipnerf-pytorch#subdirectory=extensions/cuda
   and pip install git+https://github.com/SuLvXiangXin/zipnerf-pytorch[/grey]""",
         configurations=[
             ("zipnerf", "A pytorch implementation of 'Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields'")