Default overrides not taken into account when using `bazel_dep()`

[mering]

Consider the following MODULE.bazel:

module(name = "test")

bazel_dep(name = "rules_go", version = "0.54.0")

go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk")
go_sdk.download(version = "1.24.2")

bazel_dep(name = "gazelle", version = "0.43.0")

go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps")
go_deps.from_file(go_mod = "//:go.mod")
use_repo(
    go_deps,
    "com_github_envoyproxy_go_control_plane_envoy",
)

bazel_dep(name = "cel-spec", version = "0.19.2")
bazel_dep(name = "xds", version = "0.0.0-20240423-555b57e")

and go.mod:

module test

go 1.24.2

require (
	github.com/cncf/xds/go v0.0.0-20240905190251-b4127c9b8d78
	github.com/envoyproxy/go-control-plane/envoy v1.32.3
	go.opentelemetry.io/proto/otlp v1.0.0
)

require (
	cel.dev/expr v0.19.0 // indirect
	github.com/envoyproxy/go-control-plane v0.13.4 // indirect
	github.com/envoyproxy/protoc-gen-validate v1.2.1 // indirect
	github.com/expr-lang/expr v1.16.9 // indirect
	github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
	github.com/prometheus/client_model v0.6.0 // indirect
	golang.org/x/net v0.34.0 // indirect
	golang.org/x/sys v0.29.0 // indirect
	golang.org/x/text v0.21.0 // indirect
	google.golang.org/genproto/googleapis/api v0.0.0-20241202173237-19429a94021a // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20250115164207-1a7da9e5054f // indirect
	google.golang.org/grpc v1.70.0 // indirect
	google.golang.org/protobuf v1.36.4 // indirect
)

When running

USE_BAZEL_VERSION=8.2.1 bazel build @com_github_envoyproxy_go_control_plane_envoy//...

it fails with

ERROR: /home/builder/.cache/bazel/_bazel_builder/c539c54c80f7c450e25d6c6854dab449/external/xds+/xds/annotations/v3/BUILD: no such target '@@xds+//xds/annotations/v3:annotations': target 'annotations' not declared in package 'xds/annotations/v3' defined by /home/builder/.cache/bazel/_bazel_builder/c539c54c80f7c450e25d6c6854dab449/external/xds+/xds/annotations/v3/BUILD

When appending the following to MODULE.bazel:

go_deps.gazelle_override(
    build_file_generation = "on",
    path = "github.com/cncf/xds/go",
)

it succeeds even though there is

bazel-gazelle/internal/bzlmod/default_gazelle_overrides.bzl

Line 18 in d7a25a2

"github.com/cncf/xds/go": "on",

[fmeum]

We can't apply overrides to bazel_deps, so we do fall back to using the go_repository instead if the user defines any manual overrides. The same doesn't happen for default overrides and if it did, that would be surprising and somewhat broken in its own ways.

We should submit a new BCR version that incorporates the default override. What do you think? Would you be willing to send one?

[mering]

Honestly, I don't understand what those overrides do.

Are you saying that the bazel_dep is not used when using a go.mod override?
I find this hard to believe as I would assume that this would break our build somewhere else which was the reason to use a bazel_dep instead of the overrides in the first place.

[mering]

For example the BUILD.bazel file in the root folder seems to be overriden by Gazelle. For example in https://github.com/grpc-ecosystem/grpc-gateway/blob/main/BUILD.bazel there are several go_proto_compiler targets defined which we use in other places. This was one of the breakages why we couldn't use go.mod version with Gazelle.

[fmeum]

I'm not entirely sure, but I think that the bazel_dep is not used if you specify any explicit go_deps.*_overrides.

If anything doesn't work with a bazel_dep with otherwise default settings, we should definitely try to fix the BCR module first.

If certain overrides unexpectedly make it so that the bazel_dep isn't used, we should fix that as well. Unifying BCR modules and language specific deps is still a pretty unexplored space and it's quite likely that go_deps doesn't do a great job at this yet. Happy to hear any suggestions in this area.

[mering]

This might be related to the changes in #2065.

To me, it feels like this issue might be caused by something not using the bazel_dep and therefore the impossibility to fix this in the BCR module.

I couldn't figure out a fix for the reproduction example in this issue description. Do you have a suggestion on what should be fixed where for this specific example?

[fmeum]

Without the explicit override envoy depends on @xds (the BCR module):

$ USE_BAZEL_VERSION=8.2.1 bazel query @xds//xds/annotations/v3:all
@xds//xds/annotations/v3:pkg
@xds//xds/annotations/v3:pkg_cc_proto
@xds//xds/annotations/v3:pkg_cc_proto_cc_proto
@xds//xds/annotations/v3:pkg_cc_proto_validate
@xds//xds/annotations/v3:pkg_go_proto
@xds//xds/annotations/v3:pkg_py_proto

With the explicit override envoy depends on @com_github_cncf_xds_go (the Go dep):

$ USE_BAZEL_VERSION=8.2.1 bazel query @@gazelle++go_deps+com_github_cncf_xds_go//xds/annotations/v3:all
@@gazelle++go_deps+com_github_cncf_xds_go//xds/annotations/v3:annotations
@@gazelle++go_deps+com_github_cncf_xds_go//xds/annotations/v3:go_default_library

The BUILD files of xds at https://github.com/cncf/xds don't match what Gazelle generates. The fix would be to send a PR to that repo that adds the targets Gazelle would generate. Feel free me to mention me on any such PR.

[fmeum]

FYI @keith as you initially submitted xds to the BCR

[mering]

xds has the generated go proto bindings checked in at https://github.com/cncf/xds/tree/main/go/xds/annotations/v3. So shouldn't go proto generation be disabled in Gazelle and consumers use those instead?

Also the go module is defined at the subfolder /go while the Bazel module uses the root of the repository. So IIUC they are actually different modules.

[fmeum]

Also the go module is defined at the subfolder /go while the Bazel module uses the root of the repository. So IIUC they are actually different modules.

Okay, that makes the situation much more delicate. We now have a BCR module a submodule of which subsumes a Go dependency. That's not a situation I have considered before.

Given that xds (the BCR module) doesn't contain any functional BUILD files for Go and the Go functionality doesn't depend on the rest of the module, how about we just disentangle the two by making https://github.com/cncf/xds/blob/ae57f3c0d45fc76d0b323b79e8299a83ccb37a49/MODULE.bazel#L28 a dev_dependency? That would ensure that gazelle doesn't pick up the BCR module as a replacement for the Go module, which you pretty clearly showed it isn't.

[mering]

IIUC they need it for their down development/ci:
https://github.com/cncf/xds/blob/ae57f3c0d45fc76d0b323b79e8299a83ccb37a49/xds/annotations/v3/BUILD#L5 macro resolves to https://github.com/cncf/xds/blob/ae57f3c0d45fc76d0b323b79e8299a83ccb37a49/bazel/api_build_system.bzl#L110-L124 which generates a go_proto_library().

[fmeum]

Looks like that pkg_go_proto target is public, which means that they do need a non-development dependency on gazelle go_deps. Which means that we would need a way for a Bazel module to use go_deps.from_file without opting into representing this Go module. We could add that feature to the next version of Gazelle.

A workaround could be to migrate xds to use go_deps.module instead or to just accept the breakage of making the pkg_go_proto target private and turning go_deps into a real dev dependency.

[mering]

What would be the recommended approach for a library project using Bazel but also providing compatibility for non-Bazel go.mod projects?

• Not creating go_proto_library() targets and instead adding BUILD files with go_library() targets into the folders with the generated .pb.go files?
• Moving the go.mod for protos into a sub-folder and somehow depending on this via Gazelle go.mod from the root folder?

[fmeum]

You should be able to use pregenerated proto files as long as you add the relevant Gazelle directive (such as proto disable) to the top-level BUILD file so that the BUILD file contents Gazelle generates also use the pregenerated files.

The big problem with xds really is the fact that it doesn't correspond to the Go module's directory layout.