[feature] Handle library loops (--start-group, --end-group)

[shellshocked2003]

It seems that going from v1.21 to v1.22, the library ordering specified in self.cpp_info.libs is no longer maintained. Is this true or am I missing something?

The package I'm creating depends on Intel MKL, which requires a specific linking ordering as well as -Wl,--start-group and -Wl,--end-group surrounding the MKL libraries being linked to. I have the ordering specified in a Bintray package for MKL which was working fine for v1.21. When updating to v1.22, the ordering specified for the libraries in the conanfile.py for MKL is no longer maintained, so linking fails.

Is there a way to force conan to maintain the ordering or what is the best way to deal with this? Seems like this issue is related to #6510.

[jgsogo]

Hi! AFAIK, the order of the libraries in the cpp_info.libs list is not modified until they arrive to the generator, so this issue could be related to changes in the generators.

In v1.22 we modified the CMake generator (those using targets) to fix some issues related to the link order and the dependencies between CMake targets. This is the PR #6298. It is true that we are not considering the --start-group, --end-group in the list of libraries.

Which generator were you using when you came across this issue?

---

Issue #6510 is about the build-order between Conan packages computed from graphlock, it sholdn't be related to the link-order of libraries within the same package

[shellshocked2003]

Hi @jgsogo ! Thanks for the quick response! I'm using cmake as the generator and linking to the conan package as follows:

target_link_libraries(smelt_static CONAN_PKG::ipp-static CONAN_PKG::mkl-static)

This worked before, but now it seems to push the --start-group to just before the --end-group, so it no longer brackets the MKL libraries being linked to.

[jgsogo]

Hi!

This is totally related to the fix applied in #6298, it modified the way we are creating the targets in our CMake function conan_package_library_targets. The problem is that we expect in the cpp_info.libs just the list of libraries, without extra items or flags... we look for the item in the filesystem, if we find it then we add it as an imported target; if not, we consider it to be a system library. For sure, --start-group and --end-group are not found in the filesystem and they are added to the system libraries list which is appended to the end of the linker line.

There is nothing in Conan taking into account those command-line arguments, so we need to approach this issue like a feature request. I'm very sorry it has broken your build, but we can try to work on a workaround before the actual feature is ready.

The reason people use those keywords is that there is a circular dependency between the libraries, those keywords are only a convenient way to avoid repeating the libraries in the linker line, so we can try a few things:

1. Workaround: does it work if you take care inside the recipe of the link loop? I mean, you can repeat the libraries as needed:

   self.cpp_info.libs = ["mkl_intel_lp64", "mkl_sequential", "mkl_intel_lp64", "mkl_core"]  # I don't know where is the loop, probably it requires more repeated libraries
   

2. Feature: provide a way to express the link loop in the cpp_info component of the Conan recipe:
   • Implementation for CMake: we can have a look at the LINK_INTERFACE_MULTIPLICITY variable... and add it to Conan's cpp_info somehow
   • Other build systems, they probably need the --start-group/--end-group in the linker

[shellshocked2003]

Okay, thanks for the help! The workaround we're using is not pretty, but here it is:

self.cpp_info.exelinkflags = ["-Wl,--start-group {0}/lib/libmkl_intel_lp64.a {0}/lib/libmkl_sequential.a\ {0}/lib/libmkl_core.a -Wl,--end-group".format(self.cpp_info.rootpath)]

You can see the full recipe here. Do I need to submit a formal feature request, or is this something that is already in the works?

[jgsogo]

Yes, that's a workaround for sure.... I'll change the title of this issue and it is enough 😉

I'm thinking about something like this (totally speculative):

def package_info(self):
    self.cpp_info.libs = ['libA', LibGroup('libB', 'libC'), ..., 'libH']

and, under the hood, Conan will do something with that LibGruop object and translate it into the proper flags, properties,... or whatever is needed depending on the generator.

And we also have to consider this in the scope of the components feature...

[jgsogo]

I'm having a look at some CMake modules for MKL and I'm seeing some crazy stuff (i.e. pytorch).

I have a doubt, this inter-dependency can be handled just between libraries, or CMake is able to handle it between targets too?