Skip to content

Improve consistency by prefering bazel_dep over Go #2065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 14, 2025

Conversation

zecke
Copy link
Contributor

@zecke zecke commented Apr 13, 2025

Letting the bazel_dep and Go dependency participate equally in the version resolution is creating various inconsistencies. This can lead to to a root module having to either use inject_repo or use_repo and override_repo.

Resolve this by consistently preferring the bazel_dep over the Go dependency. Keep the existing code that warns (or can fail) if the dependencies are at different versions.

Example MODULE.bazel:

module(
    name = "bazel_dep_should_win",
)

bazel_dep(name = "rules_go", version = "0.53.0")
bazel_dep(name = "gazelle", version = "0.42.0")
bazel_dep(name = "circl", version = "1.3.8")

go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps")
go_deps.from_file(go_mod = "//:go.mod")

And the go.mod referring to a newer version.

Fixes #2060

What type of PR is this?

Uncomment one line below and remove others.

Bug fix
Feature
Documentation
Other

What package or component does this PR mostly affect?

For example:

language/go
cmd/gazelle
go_repository
all

What does this PR do? Why is it needed?

Which issues(s) does this PR fix?

Fixes #

Other notes for review

@zecke
Copy link
Contributor Author

zecke commented Apr 13, 2025

Any idea on how to test this? I think we can easily pin to an older version of circl and have the go.mod file have a newer entry. If things build without use_repo(go_deps, "com_github_cloudflare_circl") we should be fine.

@fmeum
Copy link
Member

fmeum commented Apr 13, 2025

That sounds decent. If you can't get that to work, I'm also fine with this PR as it is - it only deletes cases after all.

@zecke zecke force-pushed the zecke-freyth-2060 branch from 9335ec8 to 73f7823 Compare April 14, 2025 13:29
@zecke
Copy link
Contributor Author

zecke commented Apr 14, 2025

Upgrading circl to v1.6.1 was enough. Without the patch I get the below build error showing that the go.mod dep was used instead of the bazel_dep (different reponame).

FAIL: //:gazelle_test (see /private/var/tmp/_bazel_ich/c0f651c55b5a6db13b7b6f55fed6bc48/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/gazelle_test/test.log)
INFO: From Testing //:gazelle_test:
==================== Test output for //:gazelle_test:
--- pkg/BUILD.bazel	1970-01-01 00:00:00.000000001 +0000
+++ pkg/BUILD.bazel	1970-01-01 00:00:00.000000001 +0000
@@ -52,9 +52,9 @@
     embed = [":pkg"],
     deps = [
         "//pkg/data",
-        "@circl//dh/x25519",
         "@com_github_bazelbuild_buildtools//labels:go_default_library",
         "@com_github_bmatcuk_doublestar_v4//:doublestar",
+        "@com_github_cloudflare_circl//dh/x25519",
         "@com_github_datadog_sketches_go//ddsketch",
         "@com_github_envoyproxy_protoc_gen_validate//validate",
         "@com_github_fmeum_dep_on_gazelle//:dep_on_gazelle",

Letting the bazel_dep and Go dependency participate equally in the
version resolution is creating various inconsistencies. This can lead to
to a root module having to either use `inject_repo` or `use_repo` and
`override_repo`.

Resolve this by consistently preferring the `bazel_dep` over the Go
dependency. Keep the existing code that warns (or can fail) if the
dependencies are at different versions.

Example MODULE.bazel:
```
module(
    name = "bazel_dep_should_win",
)

bazel_dep(name = "rules_go", version = "0.53.0")
bazel_dep(name = "gazelle", version = "0.42.0")
bazel_dep(name = "circl", version = "1.3.8")

go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps")
go_deps.from_file(go_mod = "//:go.mod")
```

And the go.mod referring to a newer version.

Fixes bazel-contrib#2060
@fmeum fmeum force-pushed the zecke-freyth-2060 branch from 73f7823 to 893a52c Compare April 14, 2025 16:35
@fmeum fmeum enabled auto-merge (squash) April 14, 2025 16:35
@fmeum fmeum merged commit d7a25a2 into bazel-contrib:master Apr 14, 2025
14 checks passed
@ash2k
Copy link
Contributor

ash2k commented Apr 28, 2025

Hi! I'm struggling to update from gazelle 0.42.0 to 0.43.0 because of this change. If I revert it locally my build works. I may be doing something wrong, but so far I couldn't figure out how to make things work. Any help would be appreciated. Here is what I'm doing:

Even with 0.42.0 I get a warning:

(03:42:18) DEBUG: /home/ci/.cache/bazel/_bazel_ci/9ca8268df963885560ea55034139bea5/external/gazelle+/internal/bzlmod/go_deps.bzl:607:40: 
Mismatch between versions requested for Go module cel.dev/expr:
  bazel_dep version (MODULE.bazel): 0.15.0 (as "cel-spec")
  Go module version (go.mod):       0.23.1
 
Either ensure that you have
  bazel_dep(module_name = "cel-spec", version = "0.23.1")
in your MODULE.bazel file or downgrade the Go module version via
  bazel run @rules_go//go -- get cel.dev/[email protected]

Attempt 1: just bump gazelle. I get the following error:

(03:42:48) ERROR: no such package '@@[unknown repo 'dev_cel_expr' requested from @@gazelle++go_deps+com_github_google_cel_go]//': The repository '@@[unknown repo 'dev_cel_expr' requested from @@gazelle++go_deps+com_github_google_cel_go]' could not be resolved: No repository visible as '@dev_cel_expr' from repository '@@gazelle++go_deps+com_github_google_cel_go'
(03:42:48) ERROR: /home/ci/.cache/bazel/_bazel_ci/9ca8268df963885560ea55034139bea5/external/gazelle++go_deps+com_github_google_cel_go/common/types/BUILD.bazel:8:11: no such package '@@[unknown repo 'dev_cel_expr' requested from @@gazelle++go_deps+com_github_google_cel_go]//': The repository '@@[unknown repo 'dev_cel_expr' requested from @@gazelle++go_deps+com_github_google_cel_go]' could not be resolved: No repository visible as '@dev_cel_expr' from repository '@@gazelle++go_deps+com_github_google_cel_go' and referenced by '@@gazelle++go_deps+com_github_google_cel_go//common/types:go_default_library'
(03:42:48) ERROR: Analysis of target '//internal/module/kubernetes_api/agent:agent_test' failed; build aborted: Analysis failed

Attempt 2: bump gazelle and cel-spec to try to fix the warning. Same error:

(03:42:33) ERROR: no such package '@@[unknown repo 'dev_cel_expr' requested from @@gazelle++go_deps+com_github_google_cel_go]//': The repository '@@[unknown repo 'dev_cel_expr' requested from @@gazelle++go_deps+com_github_google_cel_go]' could not be resolved: No repository visible as '@dev_cel_expr' from repository '@@gazelle++go_deps+com_github_google_cel_go'
(03:42:33) ERROR: /home/ci/.cache/bazel/_bazel_ci/9ca8268df963885560ea55034139bea5/external/gazelle++go_deps+com_github_google_cel_go/cel/BUILD.bazel:7:11: no such package '@@[unknown repo 'dev_cel_expr' requested from @@gazelle++go_deps+com_github_google_cel_go]//': The repository '@@[unknown repo 'dev_cel_expr' requested from @@gazelle++go_deps+com_github_google_cel_go]' could not be resolved: No repository visible as '@dev_cel_expr' from repository '@@gazelle++go_deps+com_github_google_cel_go' and referenced by '@@gazelle++go_deps+com_github_google_cel_go//cel:go_default_library'
(03:42:33) ERROR: Analysis of target '//internal/module/kubernetes_api/agent:agent_test' failed; build aborted: Analysis failed

The warning with this change looks like this:

(03:42:00) DEBUG: /home/ci/.cache/bazel/_bazel_ci/9ca8268df963885560ea55034139bea5/external/gazelle+/internal/bzlmod/go_deps.bzl:607:40: 
Mismatch between versions requested for Go module cel.dev/expr:
  bazel_dep version (MODULE.bazel): 0.23.0 (as "cel-spec")
  Go module version (go.mod):       0.23.1
 
Either ensure that you have
  bazel_dep(module_name = "cel-spec", version = "0.23.1")
in your MODULE.bazel file or downgrade the Go module version via
  bazel run @rules_go//go -- get cel.dev/[email protected]

I don't think the version difference itself causes the problem, but probably the fact of the difference? I cannot use the exact same version since there is no cel-spec 0.23.1 in BCR.

p.s. there is no module_name attribute, its just name. The warning needs to be fixed.

What am I doing wrong and how do I fix it? 🤔

@zecke
Copy link
Contributor Author

zecke commented Apr 28, 2025

Let me have a closer look tonight. Given it says "@@[unknown repo 'dev_cel_expr'.." have you tried to use the following in your attempt #2?

bazel_dep(name = "cel-spec", version = "0.23.0", repo_name = "dev_cel_expr")

@ash2k
Copy link
Contributor

ash2k commented Apr 28, 2025

Hm, I did not realize cel-spec and cel.dev/expr is the same thing! But it still doesn't work:

(04:51:39) ERROR: no such package '@@[unknown repo 'dev_cel_expr' requested from @@gazelle++go_deps+com_github_google_cel_go]//': The repository '@@[unknown repo 'dev_cel_expr' requested from @@gazelle++go_deps+com_github_google_cel_go]' could not be resolved: No repository visible as '@dev_cel_expr' from repository '@@gazelle++go_deps+com_github_google_cel_go'
(04:51:39) ERROR: /home/ci/.cache/bazel/_bazel_ci/9ca8268df963885560ea55034139bea5/external/gazelle++go_deps+com_github_google_cel_go/cel/BUILD.bazel:7:11: no such package '@@[unknown repo 'dev_cel_expr' requested from @@gazelle++go_deps+com_github_google_cel_go]//': The repository '@@[unknown repo 'dev_cel_expr' requested from @@gazelle++go_deps+com_github_google_cel_go]' could not be resolved: No repository visible as '@dev_cel_expr' from repository '@@gazelle++go_deps+com_github_google_cel_go' and referenced by '@@gazelle++go_deps+com_github_google_cel_go//cel:go_default_library'
(04:51:39) ERROR: Analysis of target '//internal/module/kubernetes_api/agent:agent_test' failed; build aborted: Analysis failed
(04:51:39) INFO: Elapsed time: 40.173s, Critical Path: 0.95s

@zecke
Copy link
Contributor Author

zecke commented Apr 28, 2025

Something like this is the offender: https://github.com/google/cel-go/blob/master/MODULE.bazel#L82

We stop generating dev_cel_expr as a "go.mod" dependency and instead rely on the bazel_dep (somewhere in the dependency tree). As a workaround inject_repo(go_deps, "dev_cel_expr") is your friend. The fix will be to use the bazel dep consistently (or get rid of it).

(Let me have a closer look later tonight)

@zecke
Copy link
Contributor Author

zecke commented Apr 28, 2025

Something like this is the offender: https://github.com/google/cel-go/blob/master/MODULE.bazel#L82

We stop generating dev_cel_expr as a "go.mod" dependency and instead rely on the bazel_dep (somewhere in the dependency tree). As a workaround inject_repo(go_deps, "dev_cel_expr") is your friend. The fix will be to use the bazel dep consistently (or get rid of it).

(Let me have a closer look later tonight)

I think the above is accurate. The following workaround does seem to do the right thing (my build on macos fails with gRPC/protobuf/clang incompatibility issues). We should try to get upstream use the bazel-dep instead of a go_repository as dependency for dev_cel_expr.

@@ -31,6 +31,7 @@ go_sdk.download(version = "1.23.8")
 
 go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps")
 go_deps.from_file(go_mod = "//:go.mod")
+inject_repo(go_deps, "dev_cel_expr")
 
 # Update version imported via rules_proto_grpc_go
 go_deps.module(

@ash2k
Copy link
Contributor

ash2k commented Apr 28, 2025

I wonder if the "right" fix should be to have the Go module map into the BCR module automatically somehow? Right now this is not the case simply because the BCR module has a weird name (?). Would it work if BCR module was dev_cel_expr instead?

This seems like a fundamental issue in the bazel module system (I may be missing something!) - we can go from the BCR module name to the Go module name (by looking at the go.mod) but not the other way around.

We should try to get upstream use the bazel-dep instead of a go_repository as dependency for dev_cel_expr.

This might be problematic for them - there may not be a corresponding BCR version. At the moment the project is self-contained but that would add an "external" dependency they cannot control to it.

I wonder why this project even needs to be a BCR module. Probably so that people can consume the proto files from it...

@zecke
Copy link
Contributor Author

zecke commented May 4, 2025

I wonder if the "right" fix should be to have the Go module map into the BCR module automatically somehow? Right now this is not the case simply because the BCR module has a weird name (?). Would it work if BCR module was dev_cel_expr instead?

Gazelle sees both the Go module and the BCR module for the given importpath and will pick one. Before this change it picked the larger version number and this creates difficulties (see #2060) that cannot be automatically resolved (sometimes override_repo, sometimes inject_repo or it is just broken).

The repo_name is necessary as google/cel-go has picked that name for the dependency and has BUILD files that use that name (and no repo alias exists).

This seems like a fundamental issue in the bazel module system (I may be missing something!) - we can go from the BCR module name to the Go module name (by looking at the go.mod) but not the other way around.

MODULE.bazel -> bazel_dep -> go_deps.from_file inside the MODULE.bazel

We should try to get upstream use the bazel-dep instead of a go_repository as dependency for dev_cel_expr.

This might be problematic for them - there may not be a corresponding BCR version. At the moment the project is self-contained but that would add an "external" dependency they cannot control to it.

I wonder why this project even needs to be a BCR module. Probably so that people can consume the proto files from it...

Given that the cel-spec and cel-go are owned by Google. I am sure they can sprinkle a bit of automation on the cel-spec release process to publish the release to the BCR as well. I ended up creating bazelbuild/bazel-central-registry#4500 to see where it is going.

I think the BCR version of cel-spec is useful for non Go languages. In the go only case, we could probably move https://github.com/google/cel-go/blob/845f2a8ec46a147297bb648edf41cfc2b68fb189/MODULE.bazel#L47-L59 to gazelle and be done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Challenges migrating to bzlmod with go_deps vs. bazel_dep in non-monorepository environment
3 participants