Correctly pick up & use `Protobuf_PROTOC_EXECUTABLE` in cross-compilation

[h-vetinari]

It seems that when using find_package(Protobuf CONFIG), any eventual setting of Protobuf_PROTOC_EXECUTABLE is ignored.

This makes it hard for us in conda-forge, because find_package(Protobuf CONFIG) is necessary to avoid picking up CMake's own implementation of finding protobuf (and prefer an up-to-date protobuf-config.cmake; which is necessary for correctly linking abseil for example).

On the other hand, we simply must cross-compile on several architectures (e.g. no osx-arm64 runners at scale), and so if Protobuf_PROTOC_EXECUTABLE gets ignored (which carefully points to the build environment that matches the arch or the agent, and not the target environment for the final binary) we run into Bad CPU type in executable when it calls the wrong protoc.

The work-around we have come up with (and keep reinventing) in several places boils down to (courtesy @traversaro):

 find_package(Protobuf CONFIG)
 if(Protobuf_FOUND)
     message(STATUS "Found protobuf via cmake config")
+    if(Protobuf_PROTOC_EXECUTABLE)
+      set_target_properties(protobuf::protoc PROPERTIES
+        IMPORTED_LOCATION_RELEASE "${Protobuf_PROTOC_EXECUTABLE}"
+      )
+    endif()
 else()
     message(WARNING "Falling back to cmake FindProtobuf as Protobuf was not found via CONFIG")
     find_package(Protobuf REQUIRED)
 endif()

However, not everyone encountering this issue might be able to investigate and come up with this solution, so we keep running into it. I don't see why this shouldn't be supported out of the box, hence this issue.

What version of protobuf and what language are you using?
Version: 4.24.3
Language: C++

What operating system (Linux, Windows, ...) and version?

linux/osx

What runtime / compiler are you using (e.g., python version or gcc version)

What did you do?
Cross-compile a package that requires protoc, pass -DProtobuf_PROTOC_EXECUTABLE=... to CMake invocation

What did you expect to see

Correct protoc (corresponding to Protobuf_PROTOC_EXECUTABLE) gets used.

What did you see instead?

Wrong protoc gets picked up.

Anything else we should know about your project / environment

Some examples from conda-forge:

• conda-forge/opencv-feedstock@bbe4604
• conda-forge/protozfits-feedstock@e752d33

[mkruskal-google]

Thanks for reporting this! If you happen to know what needs to change to fix this I'd be happy to review a PR. I'm not very familiar with cross-compiling via cmake though, so this might take me some time to track down and fix

[h-vetinari]

@traversaro, is this something you think you could help with? 🙃

[traversaro]

@traversaro, is this something you think you could help with? 🙃

Yes, for sure if there is interest upstream I can prepare a PR. As a preliminary analysis, after thinking a bit I am afraid of the the fine details on how we can do this "overloading" of the location and the unintended side-effects, especially when using multiple config CMake generators. Probably a simpler solution is to just make sure that Protobuf_PROTOC_EXECUTABLE by default is defined as protobuf::protoc, and that ${Protobuf_PROTOC_EXECUTABLE} is used in place of protobuf::protoc in

protobuf/cmake/protobuf-generate.cmake

Line 146 in 92619cd

COMMAND protobuf::protoc

. The downside of this solution is that downstream projects that use protobuf::protoc instead of the protobuf_generate CMake function will not work out of the box, but anyhow the fix for those cases would be quite straightforward (use ${Protobuf_PROTOC_EXECUTABLE} in place of protobuf::protoc). To get an idea, we can do something similar to what we did in gazebosim/gz-msgs#392 .

[ekigwana]

Setting Protobuf_PROTOC_EXECUTABLE used to work. Now it looks like the new variable is protobuf_PROTOC_EXE but as mentioned earlier it gets ignored

[ekigwana]

It seems that when using find_package(Protobuf CONFIG), any eventual setting of Protobuf_PROTOC_EXECUTABLE is ignored.

This makes it hard for us in conda-forge, because find_package(Protobuf CONFIG) is necessary to avoid picking up CMake's own implementation of finding protobuf (and prefer an up-to-date protobuf-config.cmake; which is necessary for correctly linking abseil for example).

On the other hand, we simply must cross-compile on several architectures (e.g. no osx-arm64 runners at scale), and so if Protobuf_PROTOC_EXECUTABLE gets ignored (which carefully points to the build environment that matches the arch or the agent, and not the target environment for the final binary) we run into Bad CPU type in executable when it calls the wrong protoc.

The work-around we have come up with (and keep reinventing) in several places boils down to (courtesy @traversaro):

 find_package(Protobuf CONFIG)
 if(Protobuf_FOUND)
     message(STATUS "Found protobuf via cmake config")
+    if(Protobuf_PROTOC_EXECUTABLE)
+      set_target_properties(protobuf::protoc PROPERTIES
+        IMPORTED_LOCATION_RELEASE "${Protobuf_PROTOC_EXECUTABLE}"
+      )
+    endif()
 else()
     message(WARNING "Falling back to cmake FindProtobuf as Protobuf was not found via CONFIG")
     find_package(Protobuf REQUIRED)
 endif()

However, not everyone encountering this issue might be able to investigate and come up with this solution, so we keep running into it. I don't see why this shouldn't be supported out of the box, hence this issue.

What version of protobuf and what language are you using? Version: 4.24.3 Language: C++

What operating system (Linux, Windows, ...) and version?

linux/osx

What runtime / compiler are you using (e.g., python version or gcc version)

What did you do? Cross-compile a package that requires protoc, pass -DProtobuf_PROTOC_EXECUTABLE=... to CMake invocation

What did you expect to see

Correct protoc (corresponding to Protobuf_PROTOC_EXECUTABLE) gets used.

What did you see instead?

Wrong protoc gets picked up.

Anything else we should know about your project / environment

Some examples from conda-forge:

• conda-forge/opencv-feedstock@bbe4604
• conda-forge/protozfits-feedstock@e752d33

I had to set IMPORTED_LOCATION_RELWITHDEBINFO so even that seems like fun trap that just cost me 4 hours to discover

[traversaro]

I had to set IMPORTED_LOCATION_RELWITHDEBINFO so even that seems like fun trap that just cost me 4 hours to discover

Yes, I was referring exactly to that with "I am afraid of the the fine details on how we can do this "overloading" of the location and the unintended side-effects, especially when using multiple config CMake generators.", sorry about that. I hope to prepare a concrete proposal to address this in the next week.

[github-actions]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[traversaro]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

Ops, sorry about that. I forgot to follow up on my original " I hope to prepare a concrete proposal to address this in the next week.".