Segmentation Fault when loading images from banana dataset #194
Description
OS: Linux *** 5.15.0-25-generic #25-Ubuntu SMP Wed Mar 30 15:54:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
GPU: RX 6700 XT (gfx1030)
Driver: ROCm 5.6
Built Opensplat using this command:
cmake -GNinja
-DCMAKE_BUILD_TYPE=Debug
-DCMAKE_PREFIX_PATH="$HOME/Documents/opensplat/libtorch"
-DTorch_DIR="$HOME/Documents/opensplat/libtorch/share/cmake/Torch"
-DGPU_RUNTIME=HIP
-DOPENSPLAT_BUILD_SIMPLE_TRAINER=ON
-DOPENSPLAT_USE_HIP=ON
..
CMake log:
-- The C compiler identification is Clang 16.0.0
-- The CXX compiler identification is Clang 16.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rocm/llvm/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rocm/llvm/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
OpenSplat Version: 1.1.5 (git commit 80378df)
-- Using the multi-header code from /home/user/Documents/opensplat/OpenSplat/build/_deps/nlohmann_json-src/include/
-- nanoflann version: 1.5.5
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- GLM: Version 1.0.1
-- GLM: Disable -Wc++98-compat warnings
-- GLM: Build with C++ features auto detection
-- Found HIP: /opt/rocm (found version "5.6.31061-8c743ae5d")
-- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS
-- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS - Success
-- The HIP compiler identification is Clang 16.0.0
-- Detecting HIP compiler ABI info
-- Detecting HIP compiler ABI info - done
-- Check for working HIP compiler: /opt/rocm/llvm/bin/clang++ - skipped
-- Detecting HIP compile features
-- Detecting HIP compile features - done
Building PyTorch for GPU arch: gfx1030
-- Found HIP: /opt/rocm (found suitable version "5.6.31061-8c743ae5d", minimum required is "1.0")
HIP VERSION: 5.6.31061-8c743ae5d
-- Caffe2: Header version is: 5.6.0
***** ROCm version from rocm_version.h ****
ROCM_VERSION_DEV: 5.6.0
ROCM_VERSION_DEV_MAJOR: 5
ROCM_VERSION_DEV_MINOR: 6
ROCM_VERSION_DEV_PATCH: 0
ROCM_VERSION_DEV_INT: 50600
HIP_VERSION_MAJOR: 5
HIP_VERSION_MINOR: 6
TORCH_HIP_VERSION: 506
***** Library versions from dpkg *****
rocm-dev VERSION: 5.6.0.50600-67~22.04
rocm-developer-tools VERSION: 5.6.0.50600-67~22.04
rocm-device-libs VERSION: 1.0.0.50600-67~22.04
rocm-device-libs-17 VERSION: 6.0+git20231212.5a852ed-2
hsakmt-roct-dev VERSION: 20230421.1.8.50600-67~22.04
hsa-rocr-dev VERSION: 1.9.0.50600-67~22.04
***** Library versions from cmake find_package *****
CMake Deprecation Warning at /opt/rocm/lib/cmake/hip/hip-config.cmake:20 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
Call Stack (most recent call first):
/home/user/Documents/opensplat/libtorch/share/cmake/Caffe2/public/LoadHIP.cmake:36 (find_package)
/home/user/Documents/opensplat/libtorch/share/cmake/Caffe2/public/LoadHIP.cmake:150 (find_package_and_print_version)
/home/user/Documents/opensplat/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:74 (include)
/home/user/Documents/opensplat/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
CMakeLists.txt:180 (find_package)
-- hip::amdhip64 is SHARED_LIBRARY
hip VERSION: 5.6.23242
hsa-runtime64 VERSION: 1.9.50600
amd_comgr VERSION: 2.5.0
CMake Deprecation Warning at /opt/rocm/lib/cmake/hip/hip-config.cmake:20 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
Call Stack (most recent call first):
/usr/share/cmake-3.28/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
/opt/rocm/lib/cmake/rocrand/rocrand-config.cmake:90 (find_dependency)
/home/user/Documents/opensplat/libtorch/share/cmake/Caffe2/public/LoadHIP.cmake:36 (find_package)
/home/user/Documents/opensplat/libtorch/share/cmake/Caffe2/public/LoadHIP.cmake:153 (find_package_and_print_version)
/home/user/Documents/opensplat/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:74 (include)
/home/user/Documents/opensplat/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
CMakeLists.txt:180 (find_package)
-- hip::amdhip64 is SHARED_LIBRARY
rocrand VERSION: 2.10.17
hiprand VERSION: 2.10.16
rocblas VERSION: 3.0.0
hipblas VERSION: 1.0.0
miopen VERSION: 2.20.0
hipfft VERSION: 1.0.12
hipsparse VERSION: 2.3.6
CMake Deprecation Warning at /opt/rocm/lib/cmake/hip/hip-config.cmake:20 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
Call Stack (most recent call first):
/usr/share/cmake-3.28/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
/opt/rocm/lib/cmake/rccl/rccl-config.cmake:90 (find_dependency)
/home/user/Documents/opensplat/libtorch/share/cmake/Caffe2/public/LoadHIP.cmake:36 (find_package)
/home/user/Documents/opensplat/libtorch/share/cmake/Caffe2/public/LoadHIP.cmake:164 (find_package_and_print_version)
/home/user/Documents/opensplat/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:74 (include)
/home/user/Documents/opensplat/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
CMakeLists.txt:180 (find_package)
-- hip::amdhip64 is SHARED_LIBRARY
rccl VERSION: 2.16.5
rocprim VERSION: 2.13.0
hipcub VERSION: 2.13.1
rocthrust VERSION: 2.18.0
hipsolver VERSION: 1.8.0
-- Found Torch: /home/user/Documents/opensplat/libtorch/lib/libtorch.so
-- Found OpenCV: /home/user/miniforge3 (found version "4.11.0")
-- Configuring done (5.3s)
CMake Warning at CMakeLists.txt:253 (add_executable):
Cannot generate a safe runtime search path for target opensplat because
files in some directories may conflict with libraries in implicit
directories:
runtime library [libhipfft.so] in /opt/rocm-5.6.0/lib may be hidden by files in:
/home/user/Documents/opensplat/libtorch/lib
Some of these libraries may not be found correctly.
CMake Warning at CMakeLists.txt:282 (add_executable):
Cannot generate a safe runtime search path for target simple_trainer
because files in some directories may conflict with libraries in implicit
directories:
runtime library [libhipfft.so] in /opt/rocm-5.6.0/lib may be hidden by files in:
/home/user/Documents/opensplat/libtorch/lib
Some of these libraries may not be found correctly.
-- Generating done (0.0s)
CMake Warning:
Manually-specified variables were not used by the project:
OPENSPLAT_USE_HIP
-- Build files have been written to: /home/user/Documents/opensplat/OpenSplat/build
Build runs without errors and then when trying to process banana dataset:
user@user-A520M-DS3H-V2:~/Documents/opensplat/OpenSplat/build$ ./opensplat ../../banana/ -n 2000
Using CUDA
Reading 14241 points
Loading ../../banana/images/frame_00001.JPG
Loading ../../banana/images/frame_00002.JPG
Loading ../../banana/images/frame_00003.JPG
Loading ../../banana/images/frame_00004.JPG
Loading ../../banana/images/frame_00005.JPG
Loading ../../banana/images/frame_00006.JPG
Loading ../../banana/images/frame_00008.JPG
Loading ../../banana/images/frame_00009.JPG
Loading ../../banana/images/frame_00010.JPG
Loading ../../banana/images/frame_00011.JPG
Loading ../../banana/images/frame_00013.JPG
Loading ../../banana/images/frame_00015.JPG
Loading ../../banana/images/frame_00014.JPG
Loading ../../banana/images/frame_00016.JPG
Segmentation fault (core dumped)
The GPU is accessible and rocminfo and clinfo return correct information. Compiled binary with cpu flag works as normal.