Make nupiccore, sequential make py bindings triggers rebuild of whole c++ lib

[breznak]

1. I build the whole nupic.cpp library

cmake ... 
make -j8 install

(any sequential code modifications and re-compilation trigger only rebuild for the changed files).

2. but when I decide to build also the py bindings (ie to run py tests), the whole c++ core library is recompiled:
   python ../../setup.py test

Output:

[==========] 338 tests from 41 test cases ran. (17708 ms total)
[  PASSED  ] 338 tests.
real    0m17.713s
user    0m19.562s
sys     0m0.184s
(pyenv) mmm@mmm-U2442:/mnt/store/devel/HTM/htm-community/nupic.cpp/build/scripts$ python ../../setup.py test
setup.py: Calling /mnt/store/devel/HTM/htm-community/nupic.cpp/bindings/py/packaging/setup.py
-- REPOSITORY_DIR       = /mnt/store/devel/HTM/htm-community/nupic.cpp
-- CMAKE_BINARY_DIR     = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/scripts
-- CMAKE_BUILD_TYPE     = Release
-- CMAKE_INSTALL_PREFIX = /home/mmm/devel/HTM/htm-community/nupic.cpp/build/Debug
-- FORCE_CPP11          = OFF
-- FORCE_BOOST          = OFF
-- BINDING_BUILD        = Python2
-- Environment var CXX  = 
GNU gold (GNU Binutils for Ubuntu 2.30) 1.15
Copyright (C) 2018 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
Using ld.gold as LINKER.
-- CMAKE_CXX_COMPILER_ID= GNU
-- CMAKE_CXX_COMPILER_VERSION=7.3.0
-- CMAKE_CXX_COMPILER   = /usr/bin/g++-7
-- CMAKE_CXX_STANDARD   = 11
-- PLATFORM             = linux
-- BITNESS              = 64
-- NEEDS_BOOST          = ON
-- INTERNAL_CXX_FLAGS   = -fvisibility-inlines-hidden;-static-libstdc++;-static-libgcc;-fdiagnostics-show-option;-m64;-fPIC;-Werror;-Wextra;-Wreturn-type;-Wunused;-Wno-unused-variable;-Wno-unused-parameter;-Wno-missing-field-initializers;-Wall;-pipe;-O2;-mtune=generic;-fuse-ld=gold;-fuse-linker-plugin;-flto-report;-flto
-- INTERNAL_LINKER_FLAGS= -m64;-Wl,--no-undefined;-O2;-flto
-- COMMON_COMPILER_DEFINITIONS_STR= -DHAVE_UNISTD_H -DNTA_COMPILER_GNU
-- COMMON_OS_LIBS       = pthread;dl
-- external import
--    CMAKE_INSTALL_PREFIX  = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty
--    CMAKE_BUILD_TYPE      = Release
--    EXTERNAL_CXX_FLAGS    = -fPIC
--    NEEDS_BOOST           = ON
--    BINDING_BUILD         = Python2
-- Obtaining yaml-cpp
-- obtaining Eigen
-- obtaining Boost
-- Obtaining gtest
-- obtaining PyBind11
-- obtaining MNIST data
-- Downloading/updating mnist
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/mnist_data/mnist-download
Scanning dependencies of target mnist-download
[100%] Built target mnist-download
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty
[ 82%] Built target yaml-cpp
[ 88%] Built target gtest
[ 94%] Built target gtest_main
[100%] Built target mnist_example
-- Results from external build:
--   yaml-cpp_INCLUDE_DIRS = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/yaml-cpp/yaml-cpp-src/include
--   yaml-cpp_LIBRARIES = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/yaml-cpp/yaml-cpp-build/libyaml-cpp.a
--   eigen_INCLUDE_DIRS = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/eigen/eigen-src
--   BOOST_ROOT = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/boost/Boost_download-src
--   Boost_INCLUDE_DIRS = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/boost/Boost_download-src
--   Boost_LIBRARIES = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/boost/Boost_download-src/stage/lib/libboost_filesystem.a;/mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/boost/Boost_download-src/stage/lib/libboost_system.a
--   gtest_INCLUDE_DIRS = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/gtest/googletest-src/googletest/include
--   gtest_LIBRARIES = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/gtest/googletest-build/googletest/libgtest$<$<CONFIG:Debug>:d>.a
--   pybind11_SOURCE_DIR = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/pybind11/pybind11-src
--   pybind11_BINARY_DIR = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/pybind11/pybind11-build
--   mnist_INCLUDE_DIRS = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/mnist_data/mnist-src/include
--   mnist_SOURCE_DIR = /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/mnist_data/mnist-src
-- Configuring nupic_core src
-- MERGE_STATIC_LIBRARIES LIB_TARGET=nupic_core, STATIC_LIBS = nupic_core_solo;/mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/yaml-cpp/yaml-cpp-build/libyaml-cpp.a;/mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/boost/Boost_download-src/stage/lib/libboost_filesystem.a;/mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/boost/Boost_download-src/stage/lib/libboost_system.a
-- MERGE_STATIC_LIBRARIES:     nupic_core_solo
-- MERGE_STATIC_LIBRARIES:     /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/yaml-cpp/yaml-cpp-build/libyaml-cpp.a - non-target lib.
-- MERGE_STATIC_LIBRARIES:     /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/boost/Boost_download-src/stage/lib/libboost_filesystem.a - non-target lib.
-- MERGE_STATIC_LIBRARIES:     /mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/boost/Boost_download-src/stage/lib/libboost_system.a - non-target lib.
-- ExpandStaticLib: LIB_TARGET=nupic_core_shared, src-lib=/mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/yaml-cpp/yaml-cpp-build/libyaml-cpp.a
-- ExpandStaticLib: LIB_TARGET=nupic_core_shared, src-lib=/mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/boost/Boost_download-src/stage/lib/libboost_filesystem.a
-- ExpandStaticLib: LIB_TARGET=nupic_core_shared, src-lib=/mnt/store/devel/HTM/htm-community/nupic.cpp/build/ThirdParty/boost/Boost_download-src/stage/lib/libboost_system.a
-- Configuring nupic_core tests
-- Configuring Python interface
-- pybind11 v2.2.4
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/store/devel/HTM/htm-community/nupic.cpp/build/scripts
Scanning dependencies of target LibrarySource
[  0%] Building CXX object src/CMakeFiles/LibrarySource.dir/nupic/algorithms/Anomaly.cpp.o
[  1%] Building CXX object src/CMakeFiles/LibrarySource.dir/nupic/algorithms/AnomalyLikelihood.cpp.o
[  2%] Building CXX object src/CMakeFiles/LibrarySource.dir/nupic/algorithms/BacktrackingTMCpp.cpp.o
[  3%] Building CXX object src/CMakeFiles/LibrarySource.dir/nupic/algorithms/Cell.cpp.o
[  3%] Building CXX object src/CMakeFiles/LibrarySource.dir/nupic/algorithms/Cells4.cpp.o
[  4%] Building CXX object src/CMakeFiles/LibrarySource.dir/nupic/algorithms/Connections.cpp.o
[  5%] Building CXX object src/CMakeFiles/LibrarySo

Hint: is it because a PY_BUILD=python2.7 cmake flag is added?

Current:

• c++ library is rebuild for py bindings even if compiler before

Expected behavior:

• bindings just link to the existing library/binary.

[breznak]

CC @dkeeney ?

[ctrl-z-9000-times]

Why does it compile in release mode but install to folder named "Debug"? The python installer has "Release" hardcoded into it in file"bindings/py/packaging/setup.py". Maybe this issue is related to yours?

…

On Mon, Mar 4, 2019, 10:59 AM breznak ***@***.***> wrote: CC @dkeeney <https://github.com/dkeeney> ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#302 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ASq_q-xpE3GDROGxzXe8p3mWO2jDeAozks5vTULlgaJpZM4bcqD7> .

[dkeeney]

The PY_BUILD=python2.7 may be causing a problem but the Debug vs Release thing would definitely cause a full recompile.

pybind11 will be different depending on the python version. So that is why we want to use the setup.py being ran by the target Python version to build the extension libraries. If you want a Debug version of the extension library then we either have to modify setup.py or add an option flag of some sort. I assumed it would be just our team that would ever want to do Debug on this part.

[breznak]

Why does it compile in release mode but install to folder named "Debug"? The python installer has "Release" hardcoded into it in

I'm not sure, but this might be a separate bug.
Our cmake would always install to "REPO/build/Release/", even in Debug mode.
Now it seems to have changed and both folders (Release, Debug) can be created and live alongside each other.
That is OK, but bindings then have to honor CMAKE_BUILD_TYPE from c++ lib (now it'd crash).

The PY_BUILD=python2.7 may be causing a problem but the Debug vs Release thing would definitely cause a full recompile.

both "c++ lib" were compiled in Release (see above, otherwise bindings crash), so probably the specification of Python2.7 causes the whole recompilation.

[dkeeney]

When you run from setup.py you are setting CMAKE_BUILD_TYPE to Release. It is hard coded. Like I was trying to say...If you do want a Debug version of the interface library then you have to edit setup.py and change its CMAKE_BUILD_TYPE in setup.py to Debug, then rebuild.

Now that we have MSVC the Release and Debug folders must be different.

[dkeeney]

Hmmm, I do see something odd in the log you gave us:

 CMAKE_INSTALL_PREFIX = /home/mmm/devel/HTM/htm-community/nupic.cpp/build/Debug

It should not be Debug because you are building Release.