[Tool] fix connector library link order for thirdparty deps#76292
Conversation
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6b5f043a52
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The concrete connector modules (Connector/ConnectorLake/ConnectorFile/ ConnectorHive/ConnectorIceberg) reference thirdparty archives such as avro/avrocpp, arrow/parquet and roaring, but they reach the starrocks_be binary only as transitive PRIVATE deps of ModuleBootstrap/Exec. CMake appends those transitive deps after the thirdparty dependency group, so the connector objects (parquet_reader, avro_scanner, iceberg/paimon/delta delete builders, deletion_bitmap) end up behind their thirdparty archives on the static link line and their references are left unresolved -> the release BE build fails with a flood of undefined references to avro/roaring/parquet. The unit-test and benchmark binaries already link the connectors explicitly (DIRECT_CONNECTOR_LINK_LIBS) so they were unaffected, which is why PR gates stayed green while the release/daily build broke. List the connector modules explicitly in STARROCKS_LINK_LIBS ahead of the format libraries and the thirdparty group, mirroring the existing fix for the concrete format libraries.
6b5f043 to
bf3d0f3
Compare
|
Good catch, fixed in the latest push. Confirmed the facts:
|
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[BE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
Why I'm doing:
The daily release / benchmark BE build (
starrocks_benchmark_main_cloud) fails at the final link ofstarrocks_bewith a flood of undefined references to thirdparty symbols:Root cause: after the connector-module refactor (#76143 / #76232 / #76237 / #76238), the concrete connector modules (
Connector/ConnectorLake/ConnectorFile/ConnectorHive/ConnectorIceberg/ ...) reach thestarrocks_bebinary only as transitive PRIVATE deps ofModuleBootstrap/Exec. CMake appends those transitive deps after the thirdparty dependency group, so the connector objects (parquet_reader,avro_scanner, the iceberg/paimon/delta delete builders,deletion_bitmap) end up behind their thirdparty archives (avro/avrocpp, arrow/parquet, roaring) on the static link line, leaving their references unresolved.Why the PR gates stayed green while the release build broke: the PR
BUILDgate does fully linkstarrocks_be, but it runs on the Ubuntu toolchain image, which setsSTARROCKS_LINKER=lld(docker/dockerfiles/toolchains/toolchains-ubuntu.Dockerfile). lld resolves backward references into already-scanned archives by default (--warn-backrefsis off), so the misordered link line still succeeds silently. The release/benchmark build runs on the CentOS 7 image, which has no lld: glibc 2.17 < 2.29 makesbe/CMakeLists.txtforce the gold linker, and gold (like GNU ld) does strict single-pass left-to-right archive scanning — the connector objects behind the thirdparty group fail with undefined references. (The BE UT gate additionally builds withMAKE_TEST=ON, which never linksstarrocks_beat all.)So this class of link-order regression is invisible to the PR gates today. A follow-up worth considering: pass
-Wl,--warn-backrefsto lld in the Ubuntu build so archive-order breakage fails in PR CI instead of in the CentOS release build.This is the same class of link-order bug that #75841 fixed for the concrete format libraries.
What I'm doing:
List the concrete connector modules explicitly in
STARROCKS_LINK_LIBS, ahead of the format libraries and the thirdparty dependency group, mirroring the existing format-library fix. The list mirrors exactly the connector setModuleBootstrappulls: the six always-on connectors (includingConnectorCacheStats) plus the optional benchmark/Elasticsearch/JDBC/MySQL connectors under the sameWITH_CONNECTOR_*guards. TheDIRECT_CONNECTOR_LINK_LIBSdefinition is moved aboveSTARROCKS_LINK_LIBSso it can be reused there; the test/benchmark link lists are left untouched (the now-redundant explicit copy is harmless for static libs and keeps the already-green test/bench link unchanged).Fixes the release BE build break at
main.What type of PR is this:
Does this PR entail a change in behavior?
Checklist: