Add support for local datasets.

.gitmodules

@@ -1,3 +1,7 @@
 [submodule "diff-gaussian-rasterization-w-depth.git"]
 	path = diff-gaussian-rasterization-w-depth.git
 	url = git@github.com:JonathonLuiten/diff-gaussian-rasterization-w-depth.git
+[submodule "spectacularAI-sdk"]
+	path = spectacularAI-sdk
+	url = https://github.com/SpectacularAI/sdk-examples
+	branch = nerfcapture-export

README.md

@@ -41,7 +41,7 @@
       <a href="#installation">Installation</a>
     </li>
     <li>
-      <a href="#demo">Online Demo</a>
+      <a href="#demo">Offline Demo</a>
     </li>
     <li>
       <a href="#usage">Usage</a>
@@ -66,7 +66,6 @@
 
 ## Installation
 
-##### (Recommended)
 SplaTAM has been benchmarked with Python 3.10, Torch 1.12.1 & CUDA=11.6. However, Torch 1.12 is not a hard requirement and the code has also been tested with other versions of Torch and CUDA such as Torch 2.3.0 & CUDA 12.1.
 
 The simplest way to install all dependences is to use [anaconda](https://www.anaconda.com/) and [pip](https://pypi.org/project/pip/) in the following steps: 
@@ -86,6 +85,7 @@ conda activate splatam
 ``` -->
 
 #### Docker Setup
+##### (Ekumen Recommended)
 
 We also provide a docker image. We recommend using a venv to run the code inside a docker image:
 
@@ -101,60 +101,38 @@ pip install -r venv_requirements.txt
 
 ## Demo
 
-### Online
-
-You can SplaTAM your own environment with an iPhone or LiDAR-equipped Apple device by downloading and using the <a href="https://apps.apple.com/au/app/nerfcapture/id6446518379">NeRFCapture</a> app.
-
-Make sure that your iPhone and PC are connected to the same WiFi network, and then run the following command:
-
- ```bash
-bash bash_scripts/online_demo.bash configs/iphone/online_demo.py
-```
-
-On the app, keep clicking send for successive frames. Once the capturing of frames is done, the app will disconnect from the PC and check out SplaTAM's interactive rendering of the reconstruction on your PC! Here are some cool example results:
-
-<p align="center">
-  <a href="">
-    <img src="./assets/collage.gif" alt="Logo" width="75%">
-  </a>
-</p>
-
 ### Offline
 
-You can also first capture the dataset and then run SplaTAM offline on the dataset with the following command:
+You can first capture the dataset with Spectacular AI tool and then run SplaTAM offline on the recorded dataset with the following command:
 
 ```bash
-bash bash_scripts/nerfcapture.bash configs/iphone/nerfcapture.py
+bash bash_scripts/nerfcapture.bash configs/iphone/nerfcapture.py <spectacularAI_dataset_location>
 ```
 
 ### Dataset Collection
 
-If you would like to only capture your own iPhone dataset using the NeRFCapture app, please use the following command:
-
-```bash
-bash bash_scripts/nerfcapture2dataset.bash configs/iphone/dataset.py
-```
+If you would like to only capture your own iPhone dataset, please use the Spectacular AI app.
 
 ## Usage
 
-We will use the iPhone dataset as an example to show how to use SplaTAM. The following steps are similar for other datasets.
-
-To run SplaTAM, please use the following command:
+If you already converted the dataset to the SplaTAM expected input, to run SplaTAM, please use the following command:
 
 ```bash
-python scripts/splatam.py configs/iphone/splatam.py
+python scripts/splatam.py configs/iphone/splatam.py --base_dir <dataset_parent_dir> --scene <dataset_dir> --frames <dataset_frames_count>
 ```
 
 To visualize the final interactive SplaTAM reconstruction, please use the following command:
 
 ```bash
-python viz_scripts/final_recon.py configs/iphone/splatam.py
+python viz_scripts/final_recon.py configs/iphone/splatam.py --base_dir <dataset_parent_dir> --scene <dataset_dir> --frames <dataset_frames_count>
 ```
 
+### Not supported by Ekumen
+
 To visualize the SplaTAM reconstruction in an online fashion, please use the following command:
 
 ```bash
-python viz_scripts/online_recon.py configs/iphone/splatam.py
+python viz_scripts/online_recon.py configs/iphone/splatam.py --base_dir <dataset_parent_dir> --scene <dataset_dir> --frames <dataset_frames_count>
 ```
 
 To export the splats to a .ply file, please use the following command:

RESULTS.md

@@ -0,0 +1,71 @@
+# 3D Mapping Workflow (Frodo)
+
+# Recording devices
+
+List of the recording devices used during the Frodo
+
+* iPhone Pro  
+  * Has lidar, which allows depth dataset recording.  
+* iPhone  
+  * No lidar, no depth capture.  
+* Android device  
+  * ARCore should suffice for the missing lidar, but there was no depth dataset saved after recording.  
+  * So no depth capture.
+
+# Recording tools
+
+## NeRF Capture
+
+* Only available for IOS.  
+* Provides 2 different recording modes:  
+  * Online mode & offline mode.
+
+### Online mode
+
+Requires the host to be connected into the recording device, so as to process the input while it is being captured.
+
+### Offline mode
+
+Saves a directory with the RBG captured frames and the depth frames \+ a transform.json file to feed into the processing tool.  
+Offline mode is currently bugged. There is an issue with the recording processing, which is not converting the depth buffer correctly into 16 bit numbers. See [this author comment](https://github.com/jc211/NeRFCapture/issues/10#issuecomment-1888164311) for more details.
+
+## Spectacular AI
+
+* Available for both Android and IOS.  
+* IOS’ version provides significantly less configuration options than Android’s.  
+  * But Android devices don’t generate depth dataset.
+
+Recording should be taken by moving through a room while pointing the device towards the walls, and occasionally performing a panoramic view of the room to capture objects that are centered or away from the walls. The open3d capture from the ping pong table shows a good example of a trajectory the recorder should follow.
+
+# Processing tools
+
+## SplaTAM
+
+### Spectacular AI dataset
+
+In order to run SplaTAM with Spectacular AI recorded data, the first step was to process it to match the output format of NeRF Capture.
+
+### NeRF Capture dataset
+
+To get the results, SplaTAM provides the [nerfcapture2dataset](https://github.com/ekumenlabs/SplaTAM/blob/main/scripts/nerfcapture2dataset.py) script to process the NeRF Capture output into the expected directory structure.
+This script was modified to work with the Offline mode, by allowing to load a previously recorded dataset 
+
+After running [splatam](https://github.com/ekumenlabs/SplaTAM/blob/main/scripts/splatam.py) algorithm, the output is saved into an params.npz file format that can then be visualized with open3d viewer to get ahold of the desired point cloud, alongside a metrics plot showing the PSNR and the captured depth of each frame, and additionally, each output frame rasterized.
+See [NpzFile](https://numpy.org/devdocs/reference/generated/numpy.lib.npyio.NpzFile.html#numpy.lib.npyio.NpzFile) for info on `npz` format and how to load it.
+
+### Results
+
+First Spectacular AI results show metrics plots that look good on the first third of the processed frames, but derailed on the rest. This behavior was repeated throughout 3 captured datasets with an iPhone 14 pro.    
+![image](assets/metrics.png)
+This leads to the rasterized images and the point cloud visualization to look awful, since the tracking of the capturing device’s position is not correct.
+
+![image](assets/hamilton-broken.png)
+
+Changing the amount of frames to be processed to the point where the PSNR drops shows a much better result. It can be seen how the device tracker is within the captured environment.  
+![image](assets/hamilton-02-open3d.png)
+![image](assets/pool-table-open3d.png)
+
+## Instant NGP
+
+\[TODO\]
+

bash_scripts/nerfcapture.bash

@@ -1,8 +1,8 @@
 #!/bin/bash
 
 # check rmem_max and wmem_max, and increase size if necessary
-if [ "$#" -ne 1 ]; then
-    echo "Usage: bash_scripts/nerfcapture.bash <config_file>"
+if [ "$#" -ne 2 ]; then
+    echo "Usage: bash_scripts/nerfcapture.bash <config_file> <input_dataset>"
     exit
 fi
 
@@ -15,21 +15,43 @@ if sysctl -a | grep -q "net.core.rmem_max = 2147483647"; then
     echo "rmem_max already set to 2147483647"
 else
     echo "Setting rmem_max to 2147483647"
-    sudo sysctl -w net.core.rmem_max=2147483647
+    sysctl -w net.core.rmem_max=2147483647
 fi
 
 if sysctl -a | grep -q "net.core.wmem_max = 2147483647"; then
     echo "wmem_max already set to 2147483647"
 else
     echo "Setting wmem_max to 2147483647"
-    sudo sysctl -w net.core.wmem_max=2147483647
+    sysctl -w net.core.wmem_max=2147483647
 fi
 
-# Capture Dataset
-python3 scripts/nerfcapture2dataset.py --config $1
+base_dir=$(dirname $2)
+scene="$(basename $2)_nerfcapture"
+echo $base_dir $scene
+
+# Convert Spectacular AI dataset to NeRF Capture.
+echo "Exporting dataset to NeRF Capture format..."
+python3 spectacularAI-sdk/python/mapping/replay_to_nerf.py $2 \
+    --format=nerfcapture --fast \
+    --image_format=png \
+    --device_preset=ios-tof \
+    --key_frame_distance=0.0001 \
+    $base_dir/$scene
+
+frames=$(ls -l ${base_dir}/${scene}/rgb | wc -l)
+if [ -z $frames ] || [ $frames -eq 0 ]; then
+    echo "ERROR: Not enough frames at $base_dir/$scene/rgb!"
+    exit
+fi
+
+echo "Building SplaTAM expected directories..."
+python3 scripts/nerfcapture2dataset.py --config $1 --base_dir $base_dir --scene $scene --frames $frames
 
 # Run SplaTAM
-python3 scripts/splatam.py $1
+echo "Running SplaTAM..."
+python3 scripts/splatam.py $1 --base_dir $base_dir --scene $scene --frames $frames
 
 # Visualize SplaTAM Output
-python3 viz_scripts/final_recon.py $1
+echo "Running visualizer..."
+python3 viz_scripts/final_recon.py $1 --base_dir $base_dir --scene $scene --frames $frames
+

configs/iphone/nerfcapture.py

@@ -10,8 +10,8 @@
 depth_scale = 10.0 # Depth Scale used when saving depth
 overwrite = False # Rewrite over dataset if it exists
 
-full_res_width = 1920
-full_res_height = 1440
+full_res_width = 640
+full_res_height = 480
 downscale_factor = 2.0
 densify_downscale_factor = 4.0
 
@@ -145,4 +145,4 @@
         viz_fps=5, # FPS for Online Recon Viz
         enter_interactive_post_online=False, # Enter Interactive Mode after Online Recon Viz
     ),
-)
+)

datasets/README.md

@@ -0,0 +1,4 @@
+# Datasets
+## Requesting Ekumen HQ recorded datasets
+If you wish to test out one of the datasets recorded at HQ please contact research@ekumenlabs.com.
+

docker/run.sh

@@ -1,11 +1,12 @@
 #!/bin/bash
 
 timestamp=$(date +%s)
+CONTAINER_NAME=splatam-$timestamp
+IMAGE_NAME=splatam-env
 
 docker run -it \
     --volume ./:/ws/SplaTAM/ \
     --volume /tmp/.X11-unix:/tmp/.X11-unix \
-    --rm \
     --env NVIDIA_VISIBLE_DEVICES=all \
     --env NVIDIA_DRIVER_CAPABILITIES=all \
     --env DISPLAY=$DISPLAY \
@@ -15,9 +16,27 @@ docker run -it \
     --group-add video \
     --ulimit memlock=-1 \
     --ulimit stack=67108864 \
-    --name splatam-$timestamp \
+    --name $CONTAINER_NAME \
     --ipc=host \
     --gpus all \
-    splatam-env \
+    $IMAGE_NAME \
     /bin/bash
 
+# Trap workspace exits and give the user the choice to save changes.
+function onexit() {
+  while true; do
+    read -p "Do you want to overwrite the image called '$IMAGE_NAME' with the current changes? [y/n]: " answer
+    if [[ "${answer:0:1}" =~ y|Y ]]; then
+      echo "Overwriting docker image..."
+      docker commit $CONTAINER_NAME $IMAGE_NAME
+      break
+    elif [[ "${answer:0:1}" =~ n|N ]]; then
+      break
+    fi
+  done
+  docker stop $CONTAINER_NAME > /dev/null
+  docker rm $CONTAINER_NAME > /dev/null
+}
+
+trap onexit EXIT
+