Depth Completion with Multiple Balanced Bases and Confidence for Dense Monocular SLAM

Weijian Xie, Guanyi Chu, Quanhao Qian, Yihao Yu, Hai Li, Danpeng Chen, Shangjin Zhai, Nan Wang, Hujun Bao and Guofeng Zhang

We present a specifically optimized multi-basis depth completion network, called BBC-Net, tailored to the characteristics of traditional sparse SLAM systems. BBC-Net can predict multiple balanced bases and a confidence map from a monocular image with sparse points generated by off-the-shelf keypoint-based SLAM systems. The final depth is a linear combination of predicted depth bases that can be \xwjminor{easily} optimized by tuning the corresponding weights.

Installation

conda create -n bbcnet

conda activate bbcnet

pip install -r requirements.txt

Data prepare

Prepare your data as follows:

Prepare the data: Save the images in .jpg format, and save the corresponding depth maps in .png format with the same filenames.

The depth values should be in millimeters (mm), and the data type should be uint16.

Assign a value of 0 to areas without depth information.

Generate file lists: Write the absolute paths of the images into two files: train.txt and valid.txt, and place them under the (./dataset/filenames/scannet directory). You can refer to the format of (dataset/filenames/nyu/nyu_val.txt) as an example.

In addition to directly using image paths as input, to speed up training, the data can also be packed using h5py. You can use the following script to pack the data. The generated data should also include train.txt and valid.txt files in the same format as described above. You can refer to (dataset/filenames/scannet/train.txt) for the format.

python data_preprocess/preprocess_scannet.py $src_data_path $out_h5py_path

Download Model

You can download our pre-trained model from here

Training

CUDA_VISIBLE_DEVICES=0 python train224_single_gpu.py --dataset lmdb_dataset --max_depth 20.0 \
    --max_depth_eval 20.0 --save_result \
    --epochs 25 --batch_size 32 --workers 8 --base_num 4 --exp_name res_34-rs \
    --cond_loss_weight 0.0 --crit_loss_weight 1.1 --dep_grad_loss_weight 0.1 \
    --log_dir $log_dir \
    --use_fast_corner --points_num 125 --save_model

To include confidence, simply add the --with_confidence flag.

Evaluation

Make sure the valid.txt file is prepared and placed in the correct directory.

Run the following command:

CUDA_VISIBLE_DEVICES=0 python eval.py --dataset nyudepthv2 --max_depth 20.0 \
    --max_depth_eval 20.0 --save_result \
    --epochs 25 --batch_size 32 --workers 8 --base_num 4 --exp_name res_34-rs \
    --cond_loss_weight 0.0 --crit_loss_weight 1.1 --dep_grad_loss_weight 0.1 \
    --log_dir $log_dir \
    --use_fast_corner --points_num 125 --resume --ckpt $ckpt_path  \
    --with_confidence

To save the evaluation results, add the --save_result flag. The --ckpt option should point to the model checkpoint file used for inference. The results will be saved in the $log_dir directory.

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@article{xie2024depth, title={Depth completion with multiple balanced bases and confidence for dense monocular slam}, author={Xie, Weijian and Chu, Guanyi and Qian, Quanhao and Yu, Yihao and Li, Hai and Chen, Danpeng and Zhai, Shangjin and Wang, Nan and Bao, Hujun and Zhang, Guofeng}, journal={IEEE Transactions on Visualization and Computer Graphics}, year={2024}, publisher={IEEE} }