[question] Consistency of settings in tools only packages

[SpaceIm]

As of 2020/10/09, here are the settings contributing to package_id in pure utilities packages:

Utilities only packages	settings	settings contributing to package_id
7zip	"os", "arch", "compiler"	"os", "arch"
autoconf	"os", "arch"	"os", "arch"
automake	"os"	"os"
b2 (portable)	"os", "arch"	"os", "arch"
b2 (standard)	"os", "arch"	"os", "arch"
cc65	"os", "arch", "compiler", "build_type"	"os", "arch" (not with Visual Studio), "compiler", "build_type"
cccl	"compiler"	None
cmake	"os", "arch", "compiler", "build_type"	"os", "arch", "compiler", "build_type"
depot_tools	"os_build"	"os_build"
doxygen	"os", "arch", "compiler", "build_type"	"os", "arch", "compiler", "build_type"
flatc	"os_build", "arch_build"	"os_build", "arch_build"
ftjam	"os", "arch", "compiler", "build_type"	"os", "arch"
gettext	"os_build", "arch_build", "compiler"	"os_build", "arch_build"
gnulib	"os_build", "arch_build"	"os_build", "arch_build"
gperf	"os_build", "arch_build", "compiler"	"os_build", "arch_build"
guetzli	"os", "arch", "compiler"	"os", "arch"
jom	"os"	"os" (Windows only anyway)
kcov	"os", "arch", "compiler", "build_type"	"os", "arch", "compiler", "build_type"
lemon	"os", "arch", "compiler", "build_type"	"os", "arch", "compiler", "build_type"
lzip	"os", "arch", "compiler", "build_type"	"os", "arch", "compiler", "build_type"
m4	"os", "arch", "compiler", "build_type"	"os", "arch", "compiler", "build_type"
make	"os_build", "arch_build", "compiler"	"os_build", "arch_build"
makefile-project-workspace-creator	"os"	"os"
meson	None	None
mozilla-build	"os_build", "arch_build"	"os_build", "arch_build"
msys2	"os_build", "arch_build"	"os_build", "arch_build"
nasm	"os_build", "arch_build", "compiler"	"os_build", "arch_build"
ninja 1.9.x	"os_build", "arch_build", "compiler"	"os_build", "arch_build"
ninja 1.10.x	"os", "arch", "compiler", "build_type"	"os", "arch", "compiler", "build_type"
nodejs	"os_build": ["Windows", "Linux", "Macos"], "arch_build": ["x86_64"]	"os_build", "arch_build"
premake	"os", "arch", "compiler", "build_type"	"os", "arch", "compiler", "build_type"
ragel	"os", "arch", "compiler"	"os", "arch"
re2c	"os", "arch", "build_type", "compiler"	"os", "arch", "build_type", "compiler"
scons	"os"	"os"
strawberryperl	"os_build", "arch_build"	"os_build" (only Windows anyway), "arch_build"
swig	"os", "arch", "compiler", "build_type"	"os", "arch", "compiler", "build_type"
verilator	"os", "arch", "compiler", "build_type"	"os", "arch", "compiler" (not the version), "build_type"
waf	"os_build", "arch_build"	"os_build", "arch_build"
winflexbison	"os", "arch", "compiler", "build_type"	"os" (Windows only anyway), "arch", "compiler", "build_type"
yasm	"os_build", "arch_build", "compiler"	"os_build", "arch_build"

Packages are not really consistent with each other. What should be the proper settings?

(I would like also to centralize discussions about settings in utilities packages in this thread)

[uilianries]

@jgsogo commented some time ago to use full settings (os, arch, build_type, compiler), but change the package_id as we need. Seems be reasonable, we will need to add one more line, customizing the package id, but, we will have a pattern for settings.

[Croydon]

Related: #3153

[madebr]

My 2c:
The package_Id of tools-only packages should include os, arch, compiler, build_type (where applicable) because they also might need to be debugged.
These recipes should be as tools and c3i should know to only build Release+MT for them.

[boussaffawalid]

tools are tools and should be be debugged as part of a build.

• It is an unneeded complication to build N configurations for ninja for example while just Release is sufficient/Preferred.
• It does not sounds correct to use Debug ninja to build a debug configuration of a library.

[ohanar]

My 2c:
In an ideal world, package_id of tools should include all 4: arch, build_type, compiler, os. Using split build and host profiles, as will likely be the default for conan 2, this shouldn't really be an issue, as build profiles will almost always have build_type=Release.

That said, this doesn't work great for the status quo, so I would be inclined to remove build_type and compiler in package_info.

[madebr]

Keeping only os and arch is the easy solution, but is not possible once a build requirement also provides libraries.
e.g. python.

For example:
Python can be used as a build requirement for executing python script (e.g. to fix a certain version)
But can also be used as a requirement for creating python modules or embedding the interpreter.

So keeping only arch and os is a short-term solution. The problem will not go away.

[boussaffawalid]

Why not keeping only arch and os, and If a requirement provides libraries, and extra package can be used. Something like

• python: tools only
• python-dev: also include dev libraries

[jgsogo]

Hi! We've been talking internally about this issue again, and these are the conclusions:

Note.- We are always talking about installers, packages that provide an executable that is used as a tool. If, at some point in time, that package provides a library and it is also used as a regular requirement it will no longer be considered a installer.

A bit of context

Conan v2.0 will likely be using the new approach with the two profiles/contexts (build and host) by default. Some details are to be defined, but it looks like it is the only way to handle tools properly as build_requires. Old os_build and arch_build seems not to be flexible enough and make recipes harder to reuse in host contexts.

As said above by @ohanar , typically a user will use build_type=Release in the build context, so for sure we need to provide those binaries. But, as @madebr has pointed out several times, sometimes it is useful/needed to debug a tool running in the build context (attach the running process and debug step by step).

Proposal

• We need to list all the required settings in the recipe like any other regular package: this is needed in order to generate the binary matching the configuration provided in the CLI

  settings = "os", "arch", "compiler", "build_type"

  conan install cmake/3.18.2@ -s build_type=Debug  --build=cmake  # Generates Debug binaries
  conan install cmake/3.18.2@ -s build_type=Release --build=cmake  # Generates Release binaries
  

  Of course, if the package is not compiling from sources, some settings are not needed (like any other regular package).

• When debugging a tool, usually one needs to switch from Debug to Release, it is not enough to be able to compile the package in Debug, we need to switch from one to the other and it is not affordable to compile the binary each time (if they have the same package_id, the binaries are overriding). That's the reason we agree we should preserve the build_type in the package_id()

• We remove the compiler in the package_id(self) method, we think no one finds it useful to have 20 different versions of CMake just because the package_id is different for every different compiler version. One of the value propositions of Conan is that you can remove some combinatorial from the settings and reuse the same binary for different configurations.

  def package_id(self):
      del self.info.settings.compiler

Future

In ConanCenter we will generate and provide binaries only for the Release configuration of these tools, that movement will require two developments:

• we need to implement in the CI something like [service] Ability to purposefully NOT build all versions for a recipe #2813 to be able to select the packages to compile. Basically, we will remove every package-ID corresponding to a Debug build (assuming the corresponding Release one is not raising a ConanInvalidConfiguration).
• recipes for these installers will include a line to declare that the Release package is compatible with the build_type=Debug. This will be achieved via compatible_packages(docs). By that time it should be a stable feature.

Hope this makes sense. Thanks for all the effort you are dedicating to the ecosystem! 🙌