Torch-MLIR build takes up over 63 gigabytes #4450
Description
I followed the build instructions on the development.md page, using the in-tree setup, as well as enabling local end-to-end tests, the exact command history follows below.
While running cmake --build build, it showed around 6000 items to compile/link, occassionally the command would crash due to a lack of memory, despite having over 12GB ram + 12 GB swap, but that's the lesser issue.
After a while, the output in the build folder would grow to the point where there is no more free disk space left on the device, at which point I have to either give up or resize the VM, I'm not sure how much more than 63GiB would it actually require.
git clone https://github.com/llvm/torch-mlir && cd torch-mlir
git submodule update --init --progress
sudo apt install python3-dev
python3 -m venv mlir_venv
source mlir_venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt -r torchvision-requirements.txt
pip install pre-commit
pre-commit install
sudo apt install clang ccache lld
cmake -GNinja -Bbuild `# Enables "--debug" and "--debug-only" flags for the "torch-mlir-opt" tool` -DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_ENABLE_ASSERTIONS=ON -DPython3_FIND_VIRTUALENV=ONLY -DPython_FIND_VIRTUALENV=ONLY -DMLIR_ENABLE_BINDINGS_PYTHON=ON -DLLVM_TARGETS_TO_BUILD=host `# For building LLVM "in-tree"` externals/llvm-project/llvm -DLLVM_ENABLE_PROJECTS=mlir -DLLVM_EXTERNAL_PROJECTS="torch-mlir" -DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="$PWD" -DTORCH_MLIR_ENABLE_PYTORCH_EXTENSIONS=ON -DTORCH_MLIR_ENABLE_JIT_IR_IMPORTER=ON
cmake --build build
This the output of the cmake configuration command:
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- bolt project is disabled
-- clang project is disabled
-- clang-tools-extra project is disabled
-- compiler-rt project is disabled
-- cross-project-tests project is disabled
-- libclc project is disabled
-- lld project is disabled
-- lldb project is disabled
-- mlir project is enabled
-- openmp project is disabled
-- polly project is disabled
-- flang project is disabled
-- libc project is disabled
-- torch-mlir project is enabled
-- Found Python3: /home/student/torch-mlir/mlir_venv/bin/python3.12 (found suitable version "3.12.3", minimum required is "3.8") found components: Interpreter
-- Looking for __GLIBC__
-- Looking for __GLIBC__ - found
-- Looking for valgrind/valgrind.h
-- Looking for valgrind/valgrind.h - not found
-- Looking for FE_ALL_EXCEPT
-- Looking for FE_ALL_EXCEPT - found
-- Looking for FE_INEXACT
-- Looking for FE_INEXACT - found
-- Performing Test HAVE_BUILTIN_THREAD_POINTER
-- Performing Test HAVE_BUILTIN_THREAD_POINTER - Success
-- Looking for CrashReporterClient.h
-- Looking for CrashReporterClient.h - not found
-- Looking for linux/magic.h
-- Looking for linux/magic.h - found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Looking for pthread_rwlock_init in pthread
-- Looking for pthread_rwlock_init in pthread - found
-- Looking for pthread_mutex_lock in pthread
-- Looking for pthread_mutex_lock in pthread - found
-- Looking for dlopen in dl
-- Looking for dlopen in dl - found
-- Looking for shm_open in rt
-- Looking for shm_open in rt - found
-- Looking for pfm_initialize in pfm
-- Looking for pfm_initialize in pfm - not found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.3")
-- Looking for compress2
-- Looking for compress2 - found
-- Found LibXml2: /usr/lib/x86_64-linux-gnu/libxml2.so (found version "2.9.14")
-- Looking for xmlReadMemory
-- Looking for xmlReadMemory - found
-- Could NOT find LibEdit (missing: LibEdit_INCLUDE_DIRS LibEdit_LIBRARIES)
-- Looking for arc4random
-- Looking for arc4random - found
-- Looking for backtrace
-- Looking for backtrace - found
-- backtrace facility detected in default set of libraries
-- Found Backtrace: /usr/include
-- Performing Test C_SUPPORTS_WERROR_UNGUARDED_AVAILABILITY_NEW
-- Performing Test C_SUPPORTS_WERROR_UNGUARDED_AVAILABILITY_NEW - Failed
-- Looking for __register_frame
-- Looking for __register_frame - found
-- Looking for __deregister_frame
-- Looking for __deregister_frame - found
-- Looking for __unw_add_dynamic_fde
-- Looking for __unw_add_dynamic_fde - not found
-- Looking for _Unwind_Backtrace
-- Looking for _Unwind_Backtrace - found
-- Looking for getpagesize
-- Looking for getpagesize - found
-- Looking for sysconf
-- Looking for sysconf - found
-- Looking for getrusage
-- Looking for getrusage - found
-- Looking for isatty
-- Looking for isatty - found
-- Looking for futimens
-- Looking for futimens - found
-- Looking for futimes
-- Looking for futimes - found
-- Looking for getauxval
-- Looking for getauxval - found
-- Looking for sigaltstack
-- Looking for sigaltstack - found
-- Looking for mallctl
-- Looking for mallctl - not found
-- Looking for mallinfo
-- Looking for mallinfo - found
-- Looking for mallinfo2
-- Looking for mallinfo2 - found
-- Looking for malloc_zone_statistics
-- Looking for malloc_zone_statistics - not found
-- Looking for posix_spawn
-- Looking for posix_spawn - found
-- Looking for pread
-- Looking for pread - found
-- Looking for sbrk
-- Looking for sbrk - found
-- Looking for strerror_r
-- Looking for strerror_r - found
-- Looking for strerror_s
-- Looking for strerror_s - not found
-- Looking for setenv
-- Looking for setenv - found
-- Performing Test HAVE_STRUCT_STAT_ST_MTIMESPEC_TV_NSEC
-- Performing Test HAVE_STRUCT_STAT_ST_MTIMESPEC_TV_NSEC - Failed
-- Performing Test HAVE_STRUCT_STAT_ST_MTIM_TV_NSEC
-- Performing Test HAVE_STRUCT_STAT_ST_MTIM_TV_NSEC - Success
-- Looking for pthread_getname_np
-- Looking for pthread_getname_np - found
-- Looking for pthread_setname_np
-- Looking for pthread_setname_np - found
-- Looking for pthread_get_name_np
-- Looking for pthread_get_name_np - not found
-- Looking for pthread_set_name_np
-- Looking for pthread_set_name_np - not found
-- Looking for dlopen
-- Looking for dlopen - found
-- Looking for proc_pid_rusage
-- Looking for proc_pid_rusage - not found
-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB - Success
-- Performing Test HAVE_CXX_ATOMICS64_WITHOUT_LIB
-- Performing Test HAVE_CXX_ATOMICS64_WITHOUT_LIB - Success
-- Performing Test LLVM_HAS_ATOMICS
-- Performing Test LLVM_HAS_ATOMICS - Success
-- LLVM host triple: x86_64-unknown-linux-gnu
-- Native target architecture is X86
-- Threads enabled.
-- Doxygen disabled.
-- Ninja version: 1.13.0.git.kitware.jobserver-pipe-1
-- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
-- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
-- OCaml bindings disabled.
-- Could NOT find Python module pygments
-- Could NOT find Python module pygments.lexers.c_cpp
-- Found Python module yaml
-- LLVM default target triple: x86_64-unknown-linux-gnu
-- Performing Test LLVM_USES_LIBSTDCXX
-- Performing Test LLVM_USES_LIBSTDCXX - Success
-- Performing Test LLVM_DEFAULT_TO_GLIBCXX_USE_CXX11_ABI
-- Performing Test LLVM_DEFAULT_TO_GLIBCXX_USE_CXX11_ABI - Success
-- Performing Test C_SUPPORTS_FPIC
-- Performing Test C_SUPPORTS_FPIC - Success
-- Performing Test CXX_SUPPORTS_FPIC
-- Performing Test CXX_SUPPORTS_FPIC - Success
-- Building with -fPIC
-- Performing Test C_SUPPORTS_FNO_SEMANTIC_INTERPOSITION
-- Performing Test C_SUPPORTS_FNO_SEMANTIC_INTERPOSITION - Success
-- Performing Test CXX_SUPPORTS_FNO_SEMANTIC_INTERPOSITION
-- Performing Test CXX_SUPPORTS_FNO_SEMANTIC_INTERPOSITION - Success
-- Performing Test SUPPORTS_FVISIBILITY_INLINES_HIDDEN_FLAG
-- Performing Test SUPPORTS_FVISIBILITY_INLINES_HIDDEN_FLAG - Success
-- Performing Test C_SUPPORTS_CXX98_COMPAT_EXTRA_SEMI_FLAG
-- Performing Test C_SUPPORTS_CXX98_COMPAT_EXTRA_SEMI_FLAG - Failed
-- Performing Test CXX_SUPPORTS_CXX98_COMPAT_EXTRA_SEMI_FLAG
-- Performing Test CXX_SUPPORTS_CXX98_COMPAT_EXTRA_SEMI_FLAG - Failed
-- Performing Test CXX_SUPPORTS_SUGGEST_OVERRIDE_FLAG
-- Performing Test CXX_SUPPORTS_SUGGEST_OVERRIDE_FLAG - Success
-- Performing Test CXX_WSUGGEST_OVERRIDE_ALLOWS_ONLY_FINAL
-- Performing Test CXX_WSUGGEST_OVERRIDE_ALLOWS_ONLY_FINAL - Success
-- Performing Test C_WCOMMENT_ALLOWS_LINE_WRAP
-- Performing Test C_WCOMMENT_ALLOWS_LINE_WRAP - Failed
-- Performing Test C_SUPPORTS_CTAD_MAYBE_UNSPPORTED_FLAG
-- Performing Test C_SUPPORTS_CTAD_MAYBE_UNSPPORTED_FLAG - Failed
-- Performing Test CXX_SUPPORTS_CTAD_MAYBE_UNSPPORTED_FLAG
-- Performing Test CXX_SUPPORTS_CTAD_MAYBE_UNSPPORTED_FLAG - Success
-- Performing Test LINKER_SUPPORTS_COLOR_DIAGNOSTICS
-- Performing Test LINKER_SUPPORTS_COLOR_DIAGNOSTICS - Failed
-- Looking for os_signpost_interval_begin
-- Looking for os_signpost_interval_begin - not found
-- Looking for flock
-- Looking for flock - found
-- Linker detection: GNU ld
-- Performing Test LLVM_LINKER_SUPPORTS_B_SYMBOLIC_FUNCTIONS
-- Performing Test LLVM_LINKER_SUPPORTS_B_SYMBOLIC_FUNCTIONS - Success
-- Performing Test HAS_WERROR_GLOBAL_CTORS
-- Performing Test HAS_WERROR_GLOBAL_CTORS - Failed
-- Looking for __x86_64__
-- Looking for __x86_64__ - found
-- Found Git: /usr/bin/git (found version "2.43.0")
-- Looking for logf128
-- Looking for logf128 - found
-- Targeting X86
-- Performing Test C_SUPPORTS_WERROR_IMPLICIT_FUNCTION_DECLARATION
-- Performing Test C_SUPPORTS_WERROR_IMPLICIT_FUNCTION_DECLARATION - Success
-- Performing Test C_SUPPORTS_WUNDEF
-- Performing Test C_SUPPORTS_WUNDEF - Success
-- Performing Test C_SUPPORTS_WERROR_MISMATCHED_TAGS
-- Performing Test C_SUPPORTS_WERROR_MISMATCHED_TAGS - Failed
-- Performing Test CXX_SUPPORTS_WNO_UNUSED_BUT_SET_PARAMETER
-- Performing Test CXX_SUPPORTS_WNO_UNUSED_BUT_SET_PARAMETER - Success
-- Found Python3: /home/student/torch-mlir/mlir_venv/bin/python3.12 (found suitable version "3.12.3", minimum required is "3.10") found components: Interpreter Development Development.Module Development.Embed
-- Found Python3: /home/student/torch-mlir/mlir_venv/bin/python3.12 (found suitable version "3.12.3", minimum required is "3.10") found components: Interpreter Development.Module
-- Found Python: /home/student/torch-mlir/mlir_venv/bin/python3.12 (found suitable version "3.12.3", minimum required is "3.10") found components: Interpreter Development.Module
-- Found python include dirs: /usr/include/python3.12
-- Found python libraries: /usr/lib/x86_64-linux-gnu/libpython3.12.so
-- Found numpy v:
-- Python extension suffix for modules: 'cpython-312-x86_64-linux-gnu'
-- Checking for nanobind in python path...
-- found (/home/student/torch-mlir/mlir_venv/lib/python3.12/site-packages/nanobind/cmake)
-- Found nanobind v2.10.2: /home/student/torch-mlir/mlir_venv/lib/python3.12/site-packages/nanobind/include
-- Python prefix = '', suffix = '', extension = '
-- Performing Test C_SUPPORTS_WERROR_GLOBAL_CONSTRUCTOR
-- Performing Test C_SUPPORTS_WERROR_GLOBAL_CONSTRUCTOR - Failed
-- Performing Test CXX_SUPPORTS_WERROR_GLOBAL_CONSTRUCTOR
-- Performing Test CXX_SUPPORTS_WERROR_GLOBAL_CONSTRUCTOR - Failed
-- Performing Test COMPILER_SUPPORTS_WARNING_WEAK_VTABLES
-- Performing Test COMPILER_SUPPORTS_WARNING_WEAK_VTABLES - Failed
CMake Deprecation Warning at /home/student/torch-mlir/CMakeLists.txt:21 (cmake_policy):
The OLD behavior for policy CMP0116 will be removed from a future version
of CMake.
The cmake-policies(7) manual explains that the OLD behaviors of all
policies are deprecated and that a policy should be set to OLD only under
specific short-term circumstances. Projects should be ported to the NEW
behavior and not rely on setting a policy to OLD.
-- Torch-MLIR in-tree build.
-- Building torch-mlir project at /home/student/torch-mlir (into /home/student/torch-mlir/build/tools/torch-mlir)
-- Found Python3: /home/student/torch-mlir/mlir_venv/bin/python3.12 (found suitable version "3.12.3", minimum required is "3.10") found components: Interpreter Development Development.Module Development.Embed
-- Found Python3: /home/student/torch-mlir/mlir_venv/bin/python3.12 (found suitable version "3.12.3", minimum required is "3.10") found components: Interpreter Development.Module
-- Found python include dirs: /usr/include/python3.12
-- Found python libraries: /usr/lib/x86_64-linux-gnu/libpython3.12.so
-- Found numpy v:
-- Python extension suffix for modules: 'cpython-312-x86_64-linux-gnu'
-- Checking for nanobind in python path...
-- found (/home/student/torch-mlir/mlir_venv/lib/python3.12/site-packages/nanobind/cmake)
-- Found nanobind v2.10.2: /home/student/torch-mlir/mlir_venv/lib/python3.12/site-packages/nanobind/include
-- Python prefix = '', suffix = '', extension = '
-- Using LLVM_ENABLE_DEBUGLOC_COVERAGE_TRACKING in: DISABLED
CMake Deprecation Warning at /home/student/torch-mlir/externals/stablehlo/CMakeLists.txt:34 (cmake_policy):
The OLD behavior for policy CMP0116 will be removed from a future version
of CMake.
The cmake-policies(7) manual explains that the OLD behaviors of all
policies are deprecated and that a policy should be set to OLD only under
specific short-term circumstances. Projects should be ported to the NEW
behavior and not rely on setting a policy to OLD.
-- Building StableHLO embedded in another project
-- Building with -fPIC
-- Found Python3: /home/student/torch-mlir/mlir_venv/bin/python3.12 (found suitable version "3.12.3", minimum required is "3.10") found components: Interpreter Development Development.Module Development.Embed
-- Found Python3: /home/student/torch-mlir/mlir_venv/bin/python3.12 (found suitable version "3.12.3", minimum required is "3.10") found components: Interpreter Development.Module
-- Found python include dirs: /usr/include/python3.12
-- Found python libraries: /usr/lib/x86_64-linux-gnu/libpython3.12.so
-- Found numpy v:
-- Python extension suffix for modules: 'cpython-312-x86_64-linux-gnu'
-- Using explicit nanobind cmake directory: /home/student/torch-mlir/mlir_venv/lib/python3.12/site-packages/nanobind/cmake (-Dnanobind_DIR to change)
-- Found nanobind v2.10.2: /home/student/torch-mlir/mlir_venv/lib/python3.12/site-packages/nanobind/include
-- Python prefix = '', suffix = '', extension = '
-- Enabling PyTorch C++ dep (features depend on it)
-- Checking for PyTorch using /home/student/torch-mlir/mlir_venv/bin/python3.12 ...
-- Found PyTorch installation at /home/student/torch-mlir/mlir_venv/lib/python3.12/site-packages/torch/share/cmake
-- Checking PyTorch ABI settings...
-- PyTorch C++ Dual ABI setting: "1"
-- Could not find `torch._C._PYBIND_BUILD_ABI`. This was removed in torch 2.9.0 (as of nightly release: dev20250826), and the TORCH_CXX_FLAGS manipulation is no longer required.
CMake Warning at /home/student/torch-mlir/mlir_venv/lib/python3.12/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
/home/student/torch-mlir/mlir_venv/lib/python3.12/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:125 (append_torchlib_if_found)
/home/student/torch-mlir/projects/CMakeLists.txt:48 (find_package)
-- Found Torch: /home/student/torch-mlir/mlir_venv/lib/python3.12/site-packages/torch/lib/libtorch.so (Required is at least version "1.11")
-- TORCH_CXXFLAGS is = -D_GLIBCXX_USE_CXX11_ABI=1
-- TORCH_LIBRARIES = torch;torch_library;/home/student/torch-mlir/mlir_venv/lib/python3.12/site-packages/torch/lib/libc10.so
-- Linking TorchMLIRJITImporter with torch;torch_library;/home/student/torch-mlir/mlir_venv/lib/python3.12/site-packages/torch/lib/libc10.so
-- Building PyTorch1 compatibility project
-- LTC Backend build is disabled
-- Registering ExampleIRTransforms as a pass plugin (static build: OFF)
-- Registering Bye as a pass plugin (static build: OFF)
-- Failed to find LLVM FileCheck
-- Google Benchmark version: v0.0.0, normalized to 0.0.0
-- Looking for shm_open in rt
-- Looking for shm_open in rt - found
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_WOLD_STYLE_CAST
-- Performing Test HAVE_CXX_FLAG_WOLD_STYLE_CAST - Success
-- Performing Test HAVE_CXX_FLAG_WSUGGEST_OVERRIDE
-- Performing Test HAVE_CXX_FLAG_WSUGGEST_OVERRIDE - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS - Success
-- Performing Test HAVE_CXX_FLAG_FNO_EXCEPTIONS
-- Performing Test HAVE_CXX_FLAG_FNO_EXCEPTIONS - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Compiling and running to test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Compiling and running to test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- success
-- Compiling and running to test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Compiling and running to test HAVE_PTHREAD_AFFINITY
-- Performing Test HAVE_PTHREAD_AFFINITY -- success
-- Configuring done (16.9s)
-- Generating done (6.5s)
-- Build files have been written to: /home/student/torch-mlir/build