Prepare the MeVisLab thirdparty libraries for the 3.7.2 release

Author: usiems

README.md

@@ -2,7 +2,7 @@
 
 Recipes for MeVisLab Thirdparty packages
 
-The `recipes` directory contains all recipes.
+The `recipes` directory contains all recipes. The overall build order is currently defined in the `build_configuratuion.yml`.
 
 ## Local package development
 
@@ -35,9 +35,66 @@ Now conan should be installed:
 conan --version
 ```
 
+### Install MeVisLab Build Tool
+
+`mbt`, the MeVisLab Build Tool, is a small wrapper for Conan and the Conan Package Tools.
+It mainly ensures that Conan is properly configured and initialized before invoking conan itself.
+
+```
+python -m pip install -U git+http://gitlab.mevis.lokal/mevislab/mbt.git
+```
+
 ## Build a package
 
+### Using mbt, conan-package-tools or conan
+
+All wrappers (`mbt`, `conan-package-tools`) are just convenience.
+They make certain things easier, while making advanced things very difficult.
+That seems to be the nature of wrappers. Therefore it is always possible to
+work without these tools.
+
+So a recipe can be built directly with `mbt`:
+
+```
+cd recipes/<recipe_name>
+mbt build
+```
+
+_Note_: If you are building under Windows, have several Visual Studio versions installed and want to use _not_ the
+newest one, you have to set an environment variable with the desired version to communicate this to `mbt`:
+```
+SET CONAN_VISUAL_VERSIONS=16
+```
+
+Or using `conan-package-tools`:
+```
+cd recipes/<recipe_name>
+./build.py
+```
+
+Or using `conan`:
 ```
 cd recipes/<recipe_name>
 conan create .
 ```
+
+## Jenkins Integration
+
+### Pipeline Description
+There are mainly two Jenkinsfiles that control the build of all thirdparty library and tools: Jenkinsfile and JenkinsfileLibrary. The first parses the `build_configuration.yml` and triggers the build of one tier after the other, starting with tier0. For further details see the chapter Build Configuration
+
+The conan package system is written in a way that it checks if the package itself or one of its dependencies has been changed since the last successful build. If nothing has changed then the build and archiving step is been skipped.
+Since even this check takes a certain amount of time, an additional optimization has been implemented. All changed files are collected since the last successful build of the complete thirdparty/branch build. And only those directly affected projects and the tiers below are triggered to be rebuild. This shortens the build time enormously. If one of the files `Jenkinsfile`, `JenkinsfileLibrary`, or `build_configuration.yml` is part of any changeset, then this optimisation is skipped and all tiers with all projects are build.
+
+### Build Configuration
+The build configuration is organised in the `build_configuration.yml` file. All libraries and tools that are needed to build MeVisLab are organised in this file. Since it very tedios and cumpersome to descripe smallest possible dependecy graph, the dependencies are broken down in different tiers. The content of the different tier is organised according to the dependencies of the individual libraries and tools. So if a e.g. a library A depends on an other library B, then B must be referenced in an one of the upper tiers.
+
+The build of each library and tool for all specified platform and build types must finish in a defined time. There is a default, that is been used for all packages. In case that a certain package needs much more time, e.g. qt5, then there is the possibility to define a timeout exception.
+
+These are the parameters that are defined in the file:
+
+| Parameter | Description |
+| --------- | ------------|
+| timeout_default: | Default time in minutes for all not other specified packages. |
+| timeout_exception: | A map of all packages with their individual timeouts. |
+| tiers: | It must contain an ordered list of all tiers, starting with tier0 |

recipes/boost/conandata.yml

@@ -2,18 +2,13 @@ type: cpp
 name: boost
 display_name: Boost
 mli_name: "None"
-version: "1.81.0"
+version: "1.83.0"
 homepage: http://www.boost.org
 description: Boost provides free peer-reviewed portable C++ source libraries
 license: BSL-1.0
 
 sources:
-  "1.81.0":
-    sha256: 71feeed900fbccca04a3b4f2f84a7c217186f28a940ed8b7ed4725986baf99fa
+  "1.83.0":
+    sha256: 6478edfe2f3305127cffe8caf73ea0176c53769f4bf1585be237eb30798c3b8e
     url: https://boostorg.jfrog.io/artifactory/main/release/${version}/source/boost_${version_major}_${version_minor}_${version_patch}.tar.bz2
     base_path: boost_${version_major}_${version_minor}_${version_patch}
-
-patches:
-  "1.81.0":
-    # commit 3f42566a7d5ab4d7d9ed81613fa61609e2084e07 on boostorg/multiprecision:
-    - patch_file: "patches/${version}-redundant_warning_pop.patch"

recipes/boost/conanfile.py

@@ -50,9 +50,6 @@ def source(self):
         self.default_source()
         os.rename(os.path.join("sources", "LICENSE_1_0.txt"), os.path.join("sources", "LICENSE"))
 
-        # change rpath on macOS
-        tools.replace_in_file('sources/tools/build/src/tools/clang-darwin.jam', '-install_name "@rpath/$(<[1]:D=)" "$(>)"', '-install_name "$(<[1]:D=)" "$(>)"')
-
 
     def build(self):
         self._bootstrap()
@@ -86,7 +83,6 @@ def package(self):
         self.copy(pattern="*.dll",      dst="bin", src=out_lib_dir, keep_path=False)
         self.copy(pattern="boost*.pdb", dst="bin", src=out_lib_dir, keep_path=False)
 
-        self.patch_binaries()
         self.default_package()
 
 
@@ -233,7 +229,7 @@ def _get_build_flags(self):
         flags.append("-sNO_BZIP2=1")
         flags.append("-sNO_LZMA=1")
         flags.append("-sNO_ZSTD=1")
-        
+
         if self.settings.compiler == "Visual Studio":
             # Avoid WINAPI mismatch in boost::log:
             flags.append("define=BOOST_USE_WINAPI_VERSION=0x600")

recipes/boost/patches/1.81.0-redundant_warning_pop.patch

@@ -1,16 +1,16 @@
 type: pure_c
 name: c-blosc
-version: "1.21.4"
+version: "1.21.5"
 homepage: https://www.blosc.org
 description: An extremely fast, multi-threaded, meta-compressor library
 license: BSD-3-Clause
 
 sources:
-  "1.21.4":
-    sha256: e72bd03827b8564bbb3dc3ea0d0e689b4863871ce3861d946f2efd7a186ecf3e
+  "1.21.5":
+    sha256: 32e61961bbf81ffea6ff30e9d70fca36c86178afd3e3cfa13376adec8c687509
     url: https://github.com/Blosc/c-blosc/archive/refs/tags/v${version}.tar.gz
     base_path: c-blosc-${version}
 
 patches:
-  "1.21.4":
+  "1.21.5":
     - patch_file: patches/${version}-fix_cmake_find_package.cmake

recipes/c-blosc/conandata.yml

@@ -2,6 +2,7 @@
 
 import io
 import os
+import re
 import glob
 import textwrap
 from typing import List, Dict, Optional
@@ -12,6 +13,7 @@
 
 MLI_DIRECTORY: str = 'MeVis/ThirdParty/Configuration/Installers/Libraries'
 
+PYTHON_UNIX_DIR_RE: re.Pattern = re.compile(r"python\d\.\d+")
 
 def _is_optional(entry):
     # make pdb/debug files optional, so we don't get an error if they can't be copied
@@ -143,7 +145,8 @@ def create_mli_content(self, variant):
         def _binExcludes(entry):
             # do not add .lib files for Windows,
             # and do also not add the whole site-packages path
-            return (tools.os_info.is_windows and (entry.endswith('.lib') or entry.endswith('.exp'))) or entry in ['python', 'site-packages']
+            return (tools.os_info.is_windows and (entry.endswith('.lib') or entry.endswith('.exp'))) or \
+                entry in ['python', 'site-packages'] or PYTHON_UNIX_DIR_RE.fullmatch(entry)
 
         # DLLs can have executable rights on Linux, better to check the suffix, too:
         def _hasDllSuffix(name):

…ches/1.21.4-fix_cmake_find_package.cmake …ches/1.21.5-fix_cmake_find_package.cmakerecipes/c-blosc/patches/1.21.4-fix_cmake_find_package.cmake renamed to recipes/c-blosc/patches/1.21.5-fix_cmake_find_package.cmake recipes/c-blosc/patches/1.21.4-fix_cmake_find_package.cmake renamed to recipes/c-blosc/patches/1.21.5-fix_cmake_find_package.cmake

@@ -2,15 +2,15 @@ type: cpp
 name: cryptopp
 cmake_name: CryptoPP
 display_name: Crypto++
-version: "8.7.0"
+version: "8.8.0"
 homepage: http://www.cryptopp.com
 description: A library of cryptographic schemes
 license: BSL-1.0
 
 sources:
     source:
-      sha256: "8d6a4064b8e9f34cd3e838f5a12c40067ee7b95ee37d9173ec273cb0913e7ca2"
-      url: "https://github.com/weidai11/cryptopp/archive/CRYPTOPP_8_7_0.tar.gz"
+      sha256: "f8b8f632533b279ee297694e651e9204824bba6022ce66e60ebebb08b551fe7a"
+      url: "https://github.com/weidai11/cryptopp/archive/CRYPTOPP_8_8_0.tar.gz"
     cmake:
-      sha256: "c113aba8069842edb201b4aaa2925e2c9d6ca081c4fd6b049ff74fedf6e8bae3"
-      url: "https://github.com/abdes/cryptopp-cmake/archive/CRYPTOPP_8_7_0.tar.gz"
+      sha256: "a10068ab47fb46785dfe43ebe45369008855b03f6711ac7d1af6f45aa9238290"
+      url: "https://github.com/abdes/cryptopp-cmake/archive/CRYPTOPP_8_8_0.tar.gz"

recipes/conan_generator/mli.py

@@ -3,6 +3,7 @@
 from conan import ConanFile
 from conan.tools.cmake import CMake, CMakeToolchain, cmake_layout
 from conan.tools.files import export_conandata_patches, get
+from conans import tools
 import shutil
 import os
 
@@ -20,6 +21,8 @@ def layout(self):
     def source(self):
         get(self, **self.conan_data["sources"]["source"], strip_root=True)
         get(self, **self.conan_data["sources"]["cmake"], destination=os.path.join(self.source_folder, "cryptopp-cmake"), strip_root=True)
+        with tools.chdir(self.source_folder):
+            tools.patch(base_path=".", patch_file=f"../patches/{self.version}-CMakeLists.patch", strip=1)
 
     def generate(self):
         tc = CMakeToolchain(self)

recipes/cryptopp/conandata.yml

@@ -0,0 +1,13 @@
+diff --git i/cryptopp-cmake/CMakeLists.txt w/cryptopp-cmake/CMakeLists.txt
+index b0ee7a2..d0ba2fd 100644
+--- i/cryptopp-cmake/CMakeLists.txt
++++ w/cryptopp-cmake/CMakeLists.txt
+@@ -192,7 +192,7 @@ include(cmake/FasterBuild.cmake)
+ # If CMake is invoked with an explicit option (cryptopp_SOURCE_DIR), setting
+ # the location for user-provided sources of crypto++, or if the automatic
+ # download fails, we use that option value to find the sources.
+-find_package(Git)
++# find_package(Git)   leads to crash because it assumed the temp conan directory is git controlled
+ 
+ if(NOT CRYPTOPP_SOURCES)
+   include(GetCryptoppSources)