Add sources

Author: atakagi-fixstars

CMakeLists.txt

@@ -0,0 +1,10 @@
+cmake_minimum_required(VERSION 3.18)
+
+project(CudaMultiViewStereo)
+
+if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
+	set(CMAKE_CUDA_ARCHITECTURES "52;61;72;75;86")
+endif()
+
+add_subdirectory(modules)
+add_subdirectory(samples)

README.md

@@ -0,0 +1,136 @@
+# cuda-multi-view-stereo
+
+C++/CUDA library for Multi-View Stereo
+
+---
+
+## Description
+cuda-multi-view-stereo (CUMVS) is a C++/CUDA library for Multi-View Stereo. This project currently supports depth map estimation using PatchMatch algorithms.
+
+## Features
+
+### Implementation of recent PatchMatch algorithms
+
+This project aims to incorporate the essence of recent PatchMatch algorithms such as [ACMM](https://github.com/GhiXu/ACMM) while simplifying the implementation as much as possible.
+
+### Performance optimization using GPU and CPU
+
+The performance of CUDA kernels for propagation is optimized using shared memory, texture fetching, projective transformation approximations, etc. In addition, the performance of some processes performed on the CPU is optimized using multi-core parallelization and reduced disk access.
+
+<div align="center">
+<img src="https://github.com/fixstars/cuda-multi-view-stereo/wiki/images/runtime.png" width=1024>
+</div>
+
+### Support for debugging and visualization
+
+The debug callback function allows you to check the status of the propagation during execution. In addition, you can instantly check the 3D reconstruction results using OpenCV's Viz module. Please check the [sample code](./samples/app_patch_match_mvs.cpp) for usage.
+
+<div align="center">
+<img src="https://github.com/fixstars/cuda-multi-view-stereo/wiki/images/demo.gif" width=1024><br/>
+Visualization of propagation using debug callback (from left: reference image, depth map, normal map, cost map)
+</div>
+
+---
+
+## Performance
+
+### Settings
+|Key|Value|
+|---|---|
+|CPU|Intel Core i7-13700K|
+|GPU|NVIDIA GeForce RTX 3080|
+|Maximum image size|3200 pixels for each dimension|
+
+### Results on ETH3D High-res multi-view training dataset [2cm]
+
+| method | Mean | courtyard | delivery_area | electro | facade | kicker | meadow | office | pipes | playground | relief | relief_2 | terrace | terrains |
+|--------|------|-----------|---------------|---------|--------|--------|--------|--------|-------|------------|--------|----------|---------|----------|
+| ACMM   | 78.9 | 87.5      | 83.7          | 86.8    | 70.8   | 77.5   | 66.4   | 64.9   | 70.1  | 72.1       | 85.4   | 84.9     | 89.9    | 85.5     |
+| CUMVS  | 81.8 | 86.4      | 86.1          | 86.3    | 69.2   | 86.9   | 76.1   | 75.2   | 73    | 74.6       | 85.8   | 84.2     | 89.8    | 90.2     |
+
+### Runtime (in second) on ETH3D High-res multi-view training dataset
+
+| SCENE | Total   | courtyard | delivery_area | electro | facade | kicker | meadow | office | pipes | playground | relief | relief_2 | terrace | terrains |
+|-------|---------|-----------|---------------|---------|--------|--------|--------|--------|-------|------------|--------|----------|---------|----------|
+| ACMM  | 12277.7 | 1051.8    | 1014          | 1173.1  | 2471.1 | 908.6  | 337.2  | 595.7  | 233.5 | 1093.3     | 855.6  | 813.4    | 520.2   | 1210.2   |
+| CUMVS | 2140.4  | 174.1     | 189.2         | 190.2   | 382.4  | 154.5  | 49.8   | 95.8   | 42.4  | 152.3      | 208.5  | 185.4    | 103.1   | 212.7    |
+
+---
+
+## Requirements
+|Package Name|Minimum Requirements|Note|
+|---|---|---|
+|CMake|version >= 3.18||
+|CUDA Toolkit|compute capability >= 6.0|
+|OpenCV|version >= 4.6.0||
+|OpenCV CUDA module|version >= 4.6.0|included in [opencv/opencv_contrib](https://github.com/opencv/opencv_contrib)|
+|OpenCV Viz module|version >= 4.6.0|included in [opencv/opencv_contrib](https://github.com/opencv/opencv_contrib)|
+|VTK|version >= 9.0|needed for OpenCV Viz module|
+
+---
+
+## How to build
+```bash
+$ git clone https://github.com/fixstars/cuda-multi-view-stereo.git
+$ cd cuda-multi-view-stereo
+$ mkdir build
+$ cd build
+$ cmake .. # specify CUDA architecture if necessary (e.g. -DCMAKE_CUDA_ARCHITECTURES=86)
+$ make
+```
+
+### CMake options
+|Option|Description|Default|
+|---|---|---|
+|CMAKE_CUDA_ARCHITECTURES|List of architectures to generate device code for|`52;61;72;75;86`|
+
+## How to run
+### samples
+
+|Command|Description|
+|---|---|
+|`./samples/app_initialize_ETH3D scene-directory [options]`|Input data preparation for ETD3D dataset|
+|`./samples/app_patch_match_mvs input-directory [options]`|Depth map estimation and 3D reconstruction|
+
+Use the `--help` or `-h` option for detailed information.
+
+### Example of running CUMVS on ETH3D High-res multi-view data
+
+Here we will use the "pipes" scene as an example.
+Download `pipes_dslr_undistorted.7z` from [ETH3D datasets](https://www.eth3d.net/datasets) and unzip it.
+
+Create input data using `app_initialize_ETH3D`.
+
+```bash
+./samples/app_initialize_ETH3D /path_to_eth3d/pipes_dslr_undistorted/pipes --output-directory=inputs/pipes
+```
+
+Test `app_patch_match_mvs`.
+The default image size is large and time consuming, so we recommend trying a smaller size first.
+The option `-d` specifies debug display.
+
+```bash
+./samples/app_patch_match_mvs inputs/pipes --max-image-size=800 -d
+```
+
+If there are no problems, run the application with your preferred options.
+After execution, a point cloud file `output/point_cloud_dense.ply` will be generated.
+
+---
+
+## Future work
+- Improving depth estimation accuracy using plane prior and adaptive patch deformation
+- Speeding up [ETH3D Multi-View Evaluation Program](https://github.com/ETH3D/multi-view-evaluation) and incorporate it into this project
+- Implementing processes after mesh reconstruction
+
+---
+
+## Author
+The "adaskit Team"
+
+The adaskit is an open-source project created by [Fixstars Corporation](https://www.fixstars.com/) and its subsidiary companies including [Fixstars Autonomous Technologies](https://at.fixstars.com/), aimed at contributing to the ADAS industry by developing high-performance implementations for algorithms with high computational cost.
+
+---
+
+## License
+Apache License 2.0

modules/CMakeLists.txt

@@ -0,0 +1 @@
+add_subdirectory(cuda_multi_view_stereo)

modules/cuda_multi_view_stereo/CMakeLists.txt

@@ -0,0 +1,30 @@
+cmake_minimum_required(VERSION 3.18)
+
+set(PROJECT_NAME cuda_multi_view_stereo)
+set(INTERFACE_INCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/include)
+set(INTERNAL_INCLUDE_DIR ${INTERFACE_INCLUDE_DIR})
+
+# create project
+project(${PROJECT_NAME} LANGUAGES CXX CUDA)
+
+# dependent packages
+find_package(OpenCV REQUIRED core cudev)
+find_package(OpenMP REQUIRED)
+find_package(CUDAToolkit REQUIRED)
+
+# target configuration
+file(GLOB SRCS src/*.cpp src/*.cu src/*.h* ${INTERNAL_INCLUDE_DIR}/*.h*)
+add_library(${PROJECT_NAME})
+target_sources(${PROJECT_NAME} PRIVATE ${SRCS})
+target_include_directories(${PROJECT_NAME} PRIVATE ${INTERNAL_INCLUDE_DIR} ${OpenCV_INCLUDE_DIRS})
+target_compile_features(${PROJECT_NAME} PRIVATE cxx_std_17)
+target_link_libraries(${PROJECT_NAME} PUBLIC ${OpenCV_LIBS} OpenMP::OpenMP_CXX CUDA::cudart)
+set_target_properties(${PROJECT_NAME} PROPERTIES INTERFACE_INCLUDE_DIRECTORIES ${INTERFACE_INCLUDE_DIR})
+
+target_compile_options(${PROJECT_NAME} PRIVATE
+$<$<COMPILE_LANG_AND_ID:CXX,GNU>:-Wall -O3>
+$<$<COMPILE_LANG_AND_ID:CXX,Clang>:-Wall -O3>
+$<$<COMPILE_LANG_AND_ID:CXX,MSVC>:/wd4819>
+$<$<COMPILE_LANGUAGE:CUDA>:-lineinfo>
+$<$<AND:$<COMPILE_LANGUAGE:CUDA>,$<CXX_COMPILER_ID:MSVC>>:-lineinfo -Xcompiler="\"/wd4819 /wd4244\"">
+)

modules/cuda_multi_view_stereo/include/cuda_multi_view_stereo.h

@@ -0,0 +1,85 @@
+/*
+Copyright 2025 Fixstars Corporation
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http ://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+#ifndef __CUDA_MULTI_VIEW_STEREO_H__
+#define __CUDA_MULTI_VIEW_STEREO_H__
+
+#include <opencv2/core.hpp>
+
+namespace cv
+{
+namespace cuda
+{
+
+/** @brief The PatchMatch methods
+*/
+enum PatchMatchMethod
+{
+	GIPUMA_FAST, //!< Gipuma, fast settings
+	ACMH,        //!< Adaptive Checkerboard sampling and Multi-Hypothesis joint view selection
+};
+
+/** @brief The class implements PatchMatch-based multi-view stereo.
+*/
+class PatchMatchMVS
+{
+public:
+
+	/** @brief The callback function for debugging and visualization
+	*/
+	using DebugCallBack = void(*)(InputArray image, InputArray depths, InputArray normals, InputArray costs, const String& title);
+
+	/** @brief The PatchMatchMVS constructor
+	@param method PatchMatch method, see #PatchMatchMethod.
+	@param pmIters Number of innier PatchMatch iterations.
+	@param gcIters Number of outer PatchMatch iterations performed with geometric consistency.
+	@param multiScale If true, the algorithm uses multi-scale method
+	*/
+	static Ptr<PatchMatchMVS> create(PatchMatchMethod method = ACMH, int pmIters = 3, int gcIters = 0, bool multiScale = false);
+
+	/** @brief The destructor
+	*/
+	virtual ~PatchMatchMVS();
+
+	/** @brief Computes depth maps for the given images and reprojects to 3D space.
+	@param images Input array of color images.
+	@param cameras Input array of camera matrices.
+	@param rotations Input array of 3x3 rotation matrices.
+	@param translations Input array of 3D translation vectors.
+	@param viewIdSets Input array of "viewIdSet"s,
+	a viewIdSet consists of [ reference view index, neighbor view index #1, neighbor view index #2, ... ]
+	@param initialObjectPoints Iutput array of sparse 3D points in object coordinates.
+	@param objectPoints Output array of densified 3D points in object coordinates.
+	@param objectColors Output array of colors of the same size with objectPoints.
+	@param objectNormals Output array of normals of the same size with objectPoints.
+	@param depths Output array of depth maps.
+	*/
+	virtual void compute(InputArrayOfArrays images, InputArrayOfArrays cameras, InputArrayOfArrays rotations,
+		InputArrayOfArrays translations, InputArrayOfArrays viewIdSets, InputArray initialObjectPoints,
+		OutputArray objectPoints, OutputArray objectColors, OutputArray objectNormals,
+		OutputArrayOfArrays depths = noArray()) = 0;
+
+	virtual void setPatchMatchMethod(PatchMatchMethod method) = 0;
+	virtual void setPatchMatchIters(int pmIters) = 0;
+	virtual void setGeometricConsistencyIters(int gcIters) = 0;
+	virtual void setMultiScale(bool multiScale) = 0;
+	virtual void setDebugCallBack(DebugCallBack callBack) = 0;
+};
+
+} // namespace cuda
+} // namespace cv
+
+#endif // !__CUDA_MULTI_VIEW_STEREO_H__

modules/cuda_multi_view_stereo/src/cuda_macro.h

@@ -0,0 +1,30 @@
+/*
+Copyright 2025 Fixstars Corporation
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http ://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+#ifndef __CUDA_MACRO_H__
+#define __CUDA_MACRO_H__
+
+#include <cstdio>
+#include <cuda.h>
+
+#define CUDA_CHECK(err) \
+do {\
+	if (err != cudaSuccess) { \
+		printf("[CUDA Error] %s (code: %d) at %s:%d\n", cudaGetErrorString(err), err, __FILE__, __LINE__); \
+	} \
+} while (0)
+
+#endif // !__CUDA_MACRO_H__