Gazelle 2.0 proposal

[jayconrod]

Purpose

This proposal collects ideas for a v2 release of Gazelle. This release decouples the extension interfaces to improve forward compatibility and to make extensions more flexible and powerful. It also removes legacy functionality, changes default behaviors, and completely moves Go and protobuf functionality into extensions.

We usually strive to maintain compatibility as much as possible. Incompatible changes require migration effort from users and extension authors, which is rarely an enjoyable task. However, legacy functionality and early design mistakes cost time and effort too, and sometimes incompatible changes are worth the migration effort.

To keep the v2 release small and tractable and to a avoid a v3 release within the next several years, we're focusing only on incompatible changes that we anticipate are necessary. New features that don't require incompatible changes may be proposed and implemented separately, either in v1 or v2.

Proposed changes

Extension interfaces

The Gazelle language interface is decomposed into a cleaner set of interfaces. An extension may implement any subset of interfaces.

Most methods can now return an error. Multiple errors may be returned together with errors.Join. In most cases, an extension should return "best effort" results together with an error. Gazelle may report the error to the user as a printed warning, or it may exit quickly but gracefully, depending on whether the -strict flag is used. Extensions should not call os.Exit or log.Fatal and should avoid panicking when possible.

Most methods now also support cancellation with context.Context. This should be useful for extensions that communicate on the network or interact with external processes. When the user interrupts Gazelle by pressing ^C, Gazelle should shut down quickly and gracefully.

More methods now have arguments and return values wrapped in struct types. New fields may be added to these structs in the future, effectively adding parameters or return values without further breaking compatibility.

config.Configurer.

Configurer parses directive comments in build files and applies them to the configuration for the current directory.

• The RegisterFlags and CheckFlags methods are removed. Extensions are no longer able to register command-line flags. All configuration must be done with directives.
• Configure now returns error.

type Configurer interface {
  // KnownDirectives returns a list of directive names this Configurer can
  // interpret.
  KnownDirectives() []string

  // Configure modifies the configuration using directives and other information
  // extracted from a build file. Configure is called in each directory.
  Configure(args ConfigureArgs) error
}

type ConfigureArgs struct {
  // Context is used for cancellation.
  Context context.Context

  // Config is the configuration for the current directory.
  Config *config.Config

  // Rel is the slash-separated path to the directory, relative to the
  // repository root ("" for the root directory itself).
  Rel string

  // File is the build file for the directory. File is nil if there is
  // no existing build file.
  File *rule.File
}

resolve.Indexer

Indexer extracts the strings by which a library may be imported.

• Indexer is split from the v1 interface resolver.Resolver.
• The Imports and Embeds methods are combined. Most languages don't need anything like Embeds, so it's easier to ignore.
• Arguments and results are moved into struct types for better extensibility.
• Imports now accepts a context for cancellation.
• Imports can now return an error.

type Indexer interface {
  // Imports reads a rule and returns a set of strings that may be used to
  // import it, if it is importable at all.
  Imports(ImportsArgs) ImportsResult
}

type ImportsArgs struct {
  // Context is used for cancellation.
  Context context.Context

  // Config is the configuration for the current directory.
  Config *config.Config

  // Rule is the non-nil rule to index.
  Rule *rule.Rule

  // File is the build file that contains Rule.
  File *rule.File
}

type ImportResult struct {
  // Imports is a list of language names and strings by which the rule may be
  // imported.
  Imports []ImportSpec

  // Embeds is a list of library labels that this rule embeds. Both the embedder
  // and embeddee may be importable by the same string, but in case of ambiguity,
  // the dependency resolver prefers the embedder.
  Embeds []label.Label

  // NotImportable is set for a rule that can't be imported, such as a test.
  // Its not necessary to set this when both Imports and Embeds are empty.
  NotImportable bool

  // Error may be set to indicate an error reading imports.
  Error error
}

type ImportSpec struct {
  Lang string
  Imp string
}

resolver.Resolver

Resolver performs dependency resolution and modifies a rule, typically by setting its "deps" attribute.

• Resolver is split from the v1 interface of the same name.
• The Name method may be dropped, if possible. It's mostly used to ensure Imports and Resolve are called using the same extension that generated a rule, but Gazelle can track that internally.
• The *repo.RemoteCache parameter is dropped. It's only used by Go and shouldn't actually be used anymore at all.
• Arguments are moved into a struct type for better extensibility.
• Resolve now accepts a context for cancellation.
• Resolve can now return an error.

type Resolver interface {
  // Resolve performs dependency resolution on a rule, setting its
  // "deps" attribute.
  Resolve(ResolveArgs) error
}

type ResolveArgs struct {
  // Context is used for cancellation.
  Context context.Context

  // Config is the configuration for the current directory.
  Config *config.Config

  // Index is a table mapping import strings to bazel labels for known
  // libraries, constructed using Indexer.
  Index *resolve.RuleIndex

  // Rule is the rule to resolve dependencies for. Resolve should modify its
  // "deps" attribute (or language-appropriate equivalent).
  Rule *rule.Rule

  // From is Rule's Bazel label.
  From label.Label

  // Imports is an opaque value returned by language.Generator for each
  // generated rule, containing information about imported libraries.
  // Resolver must know how to interpret this.
  Imports any
}

language.Generator

Generator generates rules to be merged into a new or existing build file.

• Generator is split from the language.Language interface.
• The Loads method and ModuleAwareLanguage interface are dropped. These were used to declare which .bzl file a symbol could be loaded from. We can add this to rule.KindInfo instead.
• Generate now accepts a context for cancellation.
• Generate can now return an error.

type Generator struct {
  // Name returns the name of the extension.
  Name() string

  // Kinds returns a map from rule kinds to information on how to match and
  // merge attributes that may be found in rules of those kinds. All kinds of
  // rules generated for this language may be found here.
  Kinds() map[string]rule.KindInfo

  // Generate reads source files in a directory and produces a list of rules
  // that should appear in that directory's build file.
  Generate(args GenerateArgs) GenerateResult
}

type GenerateArgs struct {
  // Context is used for cancellation.
  Context context.Context

  // Config is the configuration for the current directory.
  Config *config.Config

  // Dir is the canonical absolute path to the directory.
  Dir string

  // Rel is the slash-separated path to the directory, relative to the
  // repository root ("" for the root directory itself). This may be used
  // as the package name in labels.
  Rel string

  // File is the build file for the directory. File is nil if there is
  // no existing build file.
  File *rule.File

  // Subdirs is a list of subdirectories in the directory, including
  // symbolic links to directories that Gazelle will follow.
  // RegularFiles is a list of regular files including other symbolic
  // links.
  // GenFiles is a list of generated files in the directory
  // (usually these are mentioned as "out" or "outs" attributes in rules).
  // These slices must not be modified.
  Subdirs, RegularFiles, GenFiles []string

  // OtherEmpty is a list of empty rules generated by other languages.
  // OtherGen is a list of generated rules generated by other languages.
  OtherEmpty, OtherGen []*rule.Rule
}

type GenerateResult struct {
	// Gen is a list of rules generated from files found in the directory
	// GenerateRules was asked to process. These will be merged with existing
	// rules or added to the build file.
	Gen []*rule.Rule

	// Empty is a list of rules that cannot be built with the files found in the
	// directory GenerateRules was asked to process. These will be merged with
	// existing rules. If the merged rules are empty, they will be deleted.
	Empty []*rule.Rule

	// Imports contains information about the imported libraries for each
	// rule in Gen. Gen and Imports must have the same length, since they
	// correspond. These values are passed to Resolve after merge. The type
	// is opaque since different languages may use different representations.
	Imports []any

	// RelsToIndex is a list of additional directories to index for dependency
	// resolution, expressed as slash-separated paths relative to the repository
	// root, or "" for the root directory itself. If indexing is enabled,
	// libraries in these directories are indexed before dependencies are
	// resolved. Subdirectories are not recursively indexed. This list may
	// contain non-existent directories.
	RelsToIndex []string

  // Error may be set to indicate an error generating rules.
  Error error
}

language.Fixer

Fixer can make arbitrary changes to a build file, before generator.Generate is called. Typically, it's used to migrate away from deprecated functionality in old rule sets.

• Fixer is split from language.Language.
• Arguments are moved into a struct type for better extensibility.
• Fix now accepts a context for cancellation.
• Fix can now return an error.

type Fixer interface {
  // Fix repairs deprecated usage of language-specific rules in f. This is
  // called before the file is indexed. Unless c.ShouldFix is true, fixes
  // that delete or rename rules should not be performed.
  Fix(FixerArgs) error
}

type FixerArgs struct {
  // Context is used for cancellation.
  Context context.Context

  // Config is the configuration for the current directory.
  Config *config.Config

  // Rel is the slash-separated path to the directory, relative to the
  // repository root ("" for the root directory itself). This may be used
  // as the package name in labels.
  Rel string

  // File is the build file for the directory. File is nil if there is
  // no existing build file.
  File *rule.File
}

Life cycle hooks

These interfaces let extensions expose methods to be called at certain times during Gazelle's execution. Each method is called once, globally.

• Replaces language.FinishableLanguage and language.LifecycleManager.
• Each method accepts a context for cancellation.
• These methods may be used to manage an external connection server process that Gazelle communicates with.

type OnStart interface {
  // Called when Gazelle starts, before walking the directory tree and before
  // Configure is called on any extension.
  OnStart(ctx context.Context) error
}

type OnFinish interface {
  // Called after Gazelle has resolved dependencies on all rules and written
  // updated build files.
  OnFinish(ctx context.Context) error
}

NOTE: The main use case for these methods is starting and stopping external processes. Gazelle would call the methods here at different times than it currently calls Before, DoneGeneratingRules, and AfterResolvingDeps. If you have a use case not satisfied by OnStart and OnFinish, please let us know. We may add more interfaces here for hooks called at differen times.

Compatibility

Gazelle v2 is backward compatible with extensions written against the v1 interfaces, so it's possible to use an existing extension without migration. Gazelle v1 is also forward compatible with extensions written against the v2 interfaces, so it's possible to use newer extensions without losing legacy functionality.

Each extension package must provide either or both of the following functions. Gazelle calls one of these to load the extension.

// Load an extension for v1.
func NewLanguage() language.Language

// Load an extension for v2.
func NewV2() any

The call is generated by the gazelle_binary rule, which can inspect the extension's source code to determine which function to call.

Gazelle v1 prefers to call NewLanguage() language.Language, and Gazelle v2 prefers to call NewV2() any, but both versions are aware of both functions, and an extension only needs to implement one. Both Gazelle v1 and v2 can call the "wrong" function and wrap the result with an adapter.

If a v3 release is necessary in the future, we expect to support a NewV3() any function with suitable adapters as well.

Command Line Interface

• Lazy indexing is enabled by default when Gazelle is invoked on specific directories. Concretely, -index=lazy is the new default, and when Gazelle has at least one positional argument, -r=false is set by default. This should let users update specific build files much more quickly, but it requires coordination from language extensions.
• All flags that can be expressed as directives are dropped. This mostly applies to Go and proto flags introduced before the directive mechanism was added.
  • -build_file_name
  • -build_tags
  • -exclude
  • -external
  • -go_grpc_compiler
  • -go_naming_convention
  • -go_naming_convention_external
  • -go_prefix
  • -go_proto_compiler
  • -known_import
  • -proto
  • -proto_group
• Flags for reading and writing build files in different locations are dropped. This functionality was replaced by -mode=diff and -patch.
  • -experimental_read_build_files_dir
  • -experimental_write_build_files_dir
• The update-repos subcommand is dropped. The purpose of this command was to manage go_repository declarations in WORKSPACE. It's no longer needed for MODULE.bazel files. It never supported other languages.

go_repository

• Drop protobuf support. This never actually worked correctly. go_repository shall only generate Go targets.
• Always generate build files and remove existing targets (build_file_generation = "clean"). If a dependency already has working build files, it should be depended on as a Bazel module. The build_file_generation attribute is removed.
• Drop GOPATH support. The canonical_id, commit, tag, vcs, remote, urls, strip_prefix, type, and sha256 attributes are removed.
• Attributes corresponding to removed command-line arguments are also removed, specifically build_external, build_file_name, build_naming_convention, build_tags, build_file_proto_mode, and build_extra_args. Use build_directives instead.

Other incompatible changes

• All Go- and proto-specific functionality moves into //language/go, or a package nested within that directory. These extensions are no longer special.
• The Go module path changes from github.com/bazelbuild/bazel-gazelle to github.com/bazel-contrib/bazel-gazelle/v2, reflecting both the new major version (required by Go) and Gazelle's new home within bazel-contrib.
• The gazelle Bazel module name remains the same. Users don't need to update MODULE.bazel files, aside from selecting a new version.

Wait, what about Gazelle 1.0?

The current version of Gazelle is 0.45.0. We'll tag 1.0.0 soon without including any incompatible changes.

Under semantic versioning, we could make incompatible changes without incrementing the major version, as long as we do so under v0. However, Gazelle has been maintaining compatibility for years now, so truthfully, we should have tagged 1.0.0 long ago. We'll tag 1.0.0 belatedly, then use 2.0.0 to mark the actual imcompatible changes.

[jayconrod]

Tagging active maintainers of open source Gazelle extensions:

• C++: @WojciechMazur @mateusz-olczyk
• Go, protobuf: @fmeum @linzhp
• Haskell: @avdv
• Java: @shs96c
• JavaScript, TypeScript, Kotlin: @jbedard @alexeagle
• Python: @rickeylev @aignas
• Swift: @cgrindel

Please tag anyone else you think might be interested!

[dougthor42]

Overall I these changes seem pretty reasonable to me. I like the Resolver/Indexer split.

---

Gazelle may report the error to the user as a printed warning, or it may exit quickly but gracefully, depending on whether the -strict flag is used. Extensions should not call os.Exit or log.Fatal and should avoid panicking when possible.

I welcome this change, though I'd like to know more about the -strict flag. From the wording, it sounds like there is no "in between" where we can exit early when -strict is not set. For example, rules_python uses log.Fatal to handle unparsable directive values (example). If a directive is not parsed and -strict is not set, then generated files might be modified in an unexpected way. What should we do instead? Just force -strict for all users?

---

The RegisterFlags and CheckFlags methods are removed. Extensions are no longer able to register command-line flags. All configuration must be done with directives.

Sounds like this should simplify things 👍. rules_python doesn't use either.

---

Drop protobuf support. This never actually worked correctly. go_repository shall only generate Go targets.
...
All Go- and proto-specific functionality moves into //language/go, or a package nested within that directory

To clarify: Will the the bazel-contrib/bazel-gazelle maintainers continue to provide protobuf target generation support via @bazel_gazelle//language/proto¹? Or will protocolbuffers/protobuf create a Gazelle extension for generating protobuf build targets?

I assume that all $LANG_proto_library generation will be owned by the respective languages (eg py_proto_library from rules_python, rust_(proto|prost)_library from rules_rust, etc).

Footnotes

1. Or language/go/proto, I guess? Based on the "or a package nested within that directory" comment. ↩

[aignas]

Thank you @dougthor42 for joining in the discussion!

[jbedard]

I welcome this change, though I'd like to know more about the -strict flag. From the wording, it sounds like there is no "in between"

I had similar thoughts. Some examples that I think are normally handled differently:

1. config errors such as unparseable directives
2. code parse errors which probably require a dev to update code
3. unresolved imports which probably require a dev to update code or add resolve

I think the -strict flag is mainly meant to change (3) to stop generation vs log + continue. I assume (1) would continue to be log.Fatal, I'm not sure about (2)?

[jayconrod]

Currently, -strict means that Gazelle should exit as soon as it encounters an error. But since extensions don't have a way to report errors at all right now, the behavior is inconsistent. Some extensions call log.Fatal when strict mode is on. Others don't.

I think we could use a change here. Since extensions will be able to return errors in v2, I think it would be better for Gazelle to run until the end, print all the errors, then either write out updated files or exit with a non-zero status depending on whether -strict mode is enabled. And I'd kind of like it to be enabled by default.

About handling different errors differently: I think (1) and (3) should be fatal errors in strict mode. Not sure about (2) because it's unrealistic for extensions to be fully capable of parsing and extracting imports from complicated languages like C++ or Scala, so the likelihood of errors is higher. We could introduce "severity" as a separate error type to distinguish errors and warnings, but I'd still lean away from doing so: in every code base I've worked on that allowed warnings, they tend to accumulate without being fixed, just adding noise.

[mzeren-vmw]

brainstorming: add -keep-going and remove -strict?

[jayconrod]

-keep-going implies to me that the default behavior would be to stop immediately after the first error. But Gazelle shouldn't have any side effects until it writes build files at the end, and ideally (naïvely) it shouldn't take that long. I think it's better to gather all the errors, print them either way, and either exit non-zero or write build files and exit 0.

[jayconrod]

@dougthor42 Responding to this in a separate thread:

To clarify: Will the the bazel-contrib/bazel-gazelle maintainers continue to provide protobuf target generation support via @bazel_gazelle//language/proto1? Or will protocolbuffers/protobuf create a Gazelle extension for generating protobuf build targets?

For now, there are no plans to change ownership or write a new extension here. Gazelle will continue to maintain the current extension unless someone's interested in taking ownership and moving it elsewhere.

I assume that all $LANG_proto_library generation will be owned by the respective languages (eg py_proto_library from rules_python, rust_(proto|prost)_library from rules_rust, etc).

Yes.

[rbosetti]

Thanks @jayconrod I'm excited to see this better factoring, especially re: protobuf support.

resolve.Indexer

Indexer extracts the strings by which a library may be imported.

• Indexer is split from the v1 interface resolver.Resolver.
• The Imports and Embeds methods are combined. Most languages don't need anything like Embeds, so it's easier to ignore.
• Arguments and results are moved into struct types for better extensibility.
• Imports now accepts a context for cancellation.
• Imports can now return an error.

type Indexer interface {
  // Imports reads a rule and returns a set of strings that may be used to
  // import it, if it is importable at all.
  Imports(ImportsArgs) ImportsResult
}

type ImportsArgs struct {
  // Context is used for cancellation.
  Context context.Context

  // Config is the configuration for the current directory.
  Config *config.Config

  // Rule is the non-nil rule to index.
  Rule *rule.Rule

  // File is the build file that contains Rule.
  File *rule.File
}

type ImportResult struct {
  // Imports is a list of language names and strings by which the rule may be
  // imported.
  Imports []ImportSpec

  // Embeds is a list of library labels that this rule embeds. Both the embedder
  // and embeddee may be importable by the same string, but in case of ambiguity,
  // the dependency resolver prefers the embedder.
  Embeds []label.Label

  // NotImportable is set for a rule that can't be imported, such as a test.
  // Its not necessary to set this when both Imports and Embeds are empty.
  NotImportable bool

  // Error may be set to indicate an error reading imports.
  Error error
}

I'd be interested in implementing this interface e.g. to better support bundled/unbundled TypeScript generation for stairwell.com internal tools. Can you say more about the difference between the two indexing results:

ImportResult{
  Imports: nil,
  NotImportable: false,
}
// and...
ImportResult{
  Imports: []ImportSpec{{Lang: "typescript", Imp: "my/module/index"}},
  NotImportable: true,
}

[jayconrod]

Neither result would be importable.

The purpose of the the NotImportable flag is to support libraries that by themselves are not importable but may be embedded (with Embeds) in another library that is importable but doesn't know the import strings. Something like this:

go_library(
    name = "a",
    importpath = "example.com/a",
)

go_library(
    name = "b",
    embeds = [":a"],
    srcs = ["a.go", "b.go"],
)

The current methods describe this behavior in the handling of nil:

	// Imports returns a list of ImportSpecs that can be used to import the rule
	// r. This is used to populate RuleIndex.
	//
	// If nil is returned, the rule will not be indexed. If any non-nil slice is
	// returned, including an empty slice, the rule will be indexed.
	Imports(c *config.Config, r *rule.Rule, f *rule.File) []ImportSpec

	// Embeds returns a list of labels of rules that the given rule embeds. If
	// a rule is embedded by another importable rule of the same language, only
	// the embedding rule will be indexed. The embedding rule will inherit
	// the imports of the embedded rule.
	Embeds(r *rule.Rule, from label.Label) []label.Label

In Go, nil and the empty slice are often used interchangeably, and changing the behavior for a nil slice is a code smell, so I'd like to get rid of that.

Looking at the language/go code, I'm having trouble convincing myself this behavior is actually necessary at all. It's only used for a couple special cased proto libraries, and I think we could work around that. So maybe we could drop the functionality altogether.

[jbedard]

WDYT about dropping WORKSPACE support? Is that an option in the rules_go ecosystem yet? I will be dropping (or not actively testing+supporting) WORKSPACE in all the js related rulesets as well as gazelle language implementations.

[jayconrod]

Dropping WORKSPACE support how?

We would be dropping Gazelle's ability to update WORKSPACE files with update-repos.

No plans yet to make Gazelle v2 not able to build with WORKSPACE... but I'd definitely consider that.

Bazel itself I think has had good enough module support since 7.0, but the rule sets didn't all support modules until maybe ~5 months ago (mainly I'm thinking of rules_scala 7.0.0). So I expect most users haven't migrated yet. That said, Bazel 9.0 is dropping WORKSPACE support, and Gazelle v1 will still be around.

[sluongng]

I think we can deprioritize support for WORKSPACE. But if there are incoming patches to fix Gazelle issue under WORKSPACE, and it does not break bzlmod code paths, then we should still accept it.

Bazel 7 will reach EOL in Dec 2026. That would be the appropriate time to drop WORKSPACE logic entirely.

[sluongng]

https://bazel.build/release for support matrix

[jayconrod]

That sounds reasonable. That gives us a horizon beyond which we can actually remove code that supports WORKSPACE.

[jbedard]

How about moving go specific things like bazel_gazelle_go_repository_config out of the core gazelle macro?

[jbedard]

Or things like filegroup and alias living in the go language. I've tried moving that before but it causes a lot of test failures: #2053

[fmeum]

How about moving go specific things like bazel_gazelle_go_repository_config out of the core gazelle macro?

That could be turned into data of the go language, so I think it's doable.

[jayconrod]

How about moving go specific things like bazel_gazelle_go_repository_config out of the core gazelle macro?

Hopefully that's not too user-visible, but sure.

Or things like filegroup and alias living in the go language. I've tried moving that before but it causes a lot of test failures: #2053

To generalize, I think multiple extensions ought to be able to return Kinds for the same rule kinds without conflicting. The merger could work differently depending on which extension generated a rule.

[sluongng]

Drop protobuf support. This never actually worked correctly. go_repository shall only generate Go targets.

Is there an alternative solution proposed here?

There are cases where folks would like to use the protos which came from the go_repository, but with their own proto compiler.

[jayconrod]

Could you elaborate, maybe give an example?

I don't think this works today without a lot of customization, providing directives, maybe patches, and a lot of # gazelle:resolve directives. I'm not aware of people using this successfully, so it seems better to drop protobuf support, since it makes the Go support harder to use. But maybe I'm wrong about current usage, or maybe there's a better way to automate it?

[dougthor42]

I personally haven't had any significant trouble with Gazelle making proto_library targets, but yes I've had to use map_kind:

# MODULE.bazel

# Rename to `com_google_protobuf` because the OG rules_proto is deprecated and the
# protocolbuffers/protobuf repository calls it `com_google_protobuf`.
# See https://github.com/protocolbuffers/protobuf/blob/main/MODULE.bazel and
# https://github.com/bazel-contrib/bazel-gazelle/issues/2001.
bazel_dep(name = "protobuf", version = "29.3", repo_name = "com_google_protobuf")

# in root BUILD.bazel

# One rule per proto
# gazelle:proto file

# Make sure we use the correct protobuf repo name.
# See https://github.com/bazel-contrib/bazel-gazelle/issues/2001
# gazelle:map_kind proto_library proto_library @com_google_protobuf//bazel:proto_library.bzl

# google/api/xzy isn't well supported by Gazelle (yet?)
# gazelle:resolve proto google/api/metric.proto @googleapis//google/api:metric_proto
# gazelle:resolve proto google/api/distribution.proto @googleapis//google/api:distribution_proto
# gazelle:resolve proto google/api/field_behavior.proto @googleapis//google/api:field_behavior_proto

gazelle_binary(
    name = "gazelle_multilang",
    languages = [
        "@bazel_gazelle//language/proto",
        # To generate py_proto_library, the python gazelle plugin must be listed
        # _after_ the core gazelle proto language.
        "@rules_python_gazelle_plugin//python",
    ],
    visibility = ["//visibility:public"],
)

gazelle(
    name = "gazelle",
    gazelle = "//:gazelle_multilang",
    visibility = ["//visibility:public"],
)

[jayconrod]

Gazelle 2.0 would still support proto_library: that's done through the //language/proto extension in this repo, and ideally the core of Gazelle wouldn't need any special casing for it. Perhaps we can fix the areas where that doesn't work well.

The proposal to drop support only applies to the go_deps module extension and the go_repository repo rule, which would then be Go-specific. go_repository uses a Gazelle binary that's built by a separate internal repository rule that only includes the Go and proto extensions.

[sluongng]

https://github.com/buildbuddy-io/buildbuddy/blob/59ab0c0c042b26d7875568f157d1280a54c39222/deps/go_deps.MODULE.bazel#L80-L85

Here is an example from our repo. We use a custom proto compiler to gen code with vtprotobuf.

Without the current support in go_repository, we will have to create custom proto_library and go_proto_library targets in our repo for each of these cases, then add custom resolve directives based on these new targets. So it's a bunch of extra work and code to maintain, thus it will be a downgrade overall.

I think a sane approach here would be to turn off proto support by default. But keep the logic there so folks can turn it on when they need it?

[guw]

For Gazelle 2.0 I would like to see go_deps start using a namespace (eg., @go_deps//:foo). That would decouple things and avoid problems and possible conflicts with whatever bzlmod graph has. I believe that Bazel dependencies should not be convoluted with Go (or language) dependencies.

Other languages do this as well (Java @maven//$dependency, JavaScript/TypeScript //:node_modules/$dependency).

[fmeum]

See #2219 (comment), it's definitely not obvious how that would work. We can change the API surface at any time, but the problems run deeper.

[guw]

The problem today is that one can have a transitive Bazel dependency (eg. @xsd) without knowing (because it's a transitive). This causes confusion and clashes. But it should be completely opaque to Go's dependencies specified via go.mod/go.work.

[jayconrod]

Aside from the discussion in #2219, I think this would be too disruptive of a change for end users of Bazel. The change in labels would immediately break existing build files. Most of this proposal is about changes to the extension interfaces, and I'm trying to make sure we can support both v1 and v2 extensions with both Gazelle v1 and v2, largely in a way that's invisible to users. There are some other disruptive changes of course, but nothing on this scale.

I don't think this solves the problem either. You would have a way to refer to the same logical package provided at different versions by a Bazel module and by a Go module, but what then? We'll get link-time or run-time conflicts. We actually want to pick one version. While the actual version we pick may differ depending on the situation, go_deps has no way to override the version or content of a Bazel module, so our hand is forced: we pick the Bazel module version.

[guw]

You would have a way to refer to the same logical package provided at different versions by a Bazel module and by a Go module.

I think this is the problem, which is created by Gazelle/go_deps in the first place. In Golang there is no such thing as a Bazel module. Hence you cannot depend on a Bazel module. You only to go get foo. Mixing Bazel modules and Go modules and treating them as the same things adds problems. As a Go developer, instead of one dependency system you now have to think in two layers (or dimensions?) of possible problems. I prefer to keep Bazel dependencies (bzlmod) separate and isolated from program dependency system.

[jayconrod]

I agree that it would be best if they were separate ecosystems. Right now, they are effectively separate ecosystems, but the choice on whether something is a Bazel module or Go module is up to the module author, not the user. Once the Bazel module is published, then it's a Bazel module. If the author decides not to publish new versions of the Bazel module anymore, users don't have an easy way to switch back.

[sluongng]

My take on the new extension interfaces:

The number one challenge in implementing a Gazelle extension is that the author would have to learn and maintain Go code. This is not addressed in the current proposal.

The extensions are usually created to support other language ecosystems: python, java, cc, swift, etc... and realistically, they often have to use a shim Go code that would eventually shell out to a side-car process that was written in a different language. So at the very least, I would expect that we can come up with a solution that makes this effort a bit easier to work with.

---

Nit: why are we adding context to the custom args structs?
According to https://pkg.go.dev/context

Do not store Contexts inside a struct type; instead, pass a Context explicitly to each function that needs it. This is discussed further in https://go.dev/blog/context-and-structs. The Context should be the first parameter, typically named ctx

[alexeagle]

@jayconrod I wrote the RFC for that RPC spec a few years ago: #938

However we found that customers weren't successful, at least with gRPC protocol:

• too complex to write a new gRPC server even in a language you're familiar with
• wrote enough extensions that the server processes use a lot of RAM

So @jbedard already wrote a bridge Language called "Orion" so that you can write Gazelle extensions in Starlark. We have several customers using this, one of them has over 20 extensions now for their JS codebase.

Code: https://github.com/aspect-build/aspect-gazelle/tree/main/language/orion
Example: https://github.com/bazel-starters/shell/blob/main/.aspect/gazelle/shell.axl
Webinar: https://portal.aspect.build/training/aspect-301

@sluongng I'd love to walk you through that for some use case you care about.

[jayconrod]

We might be looking at two problems with different solutions.

I think an RPC setup makes sense when you have a complicated language that's hard to extract imports from. For example, in Scala, you can't really tell at the syntax level if something is a fully qualified class name or just an object. You'd need to write a large chunk of a compiler to be sure, with type checking, not just a parser. So you'd probably want to call existing Scala code to do that: I'd expect that Starlark + TreeSitter wouldn't be powerful enough. Full gRPC would be very heavy, but JSON-RPC seems like a good choice (as used by LSP).

I think Starlark makes sense when you have a relatively simple language with a well-defined syntax, and you don't need to implement too much complication on top of that like package grouping or platform-specific imports.

What's the status of Orion and compatibility with Gazelle? My impression was that it's coupled with aspect-cli and so an extension wouldn't work with mainline Gazelle. Is that not the case?

[jbedard]

It's a standard gazelle extension in aspect-gazelle/languages/orion. Those extensions are not published to BCR yet but that is the plan, for now they need to be fetched with archive_override.

[dougthor42]

I personally don't mind having to write go code to make a Gazelle plugin, but I understand the concern.

I'd like to note that some languages are simply much slower than go: rules_python's gazelle plugin used to have the core parsing code written in python. When it was migrated to go and tree-sitter (bazel-contrib/rules_python#1895), Gazelle's speed increased significantly.

Having an RPC introduces at least some overhead and I'd like that Gazelle 2.0 continues to have the option of directly writing go code for the plugin. I don't want to be forced to do go -> RPC -> go - that just seems silly. Other people who don't want to use go can do $LANG -> RPC -> go if they want to.

[jayconrod]

I have to exercise some self-restraint to avoid evangelizing Go :) I think it's a good language for this application, and I think it's the best choice for extensions supporting major languages like C++, Python, or Java. But learning a language is a barrier to entry, and if a company is able to quickly write 20 extensions for internal tools in the language of their choice, that's a productivity win.

So I do think there should be support for some kind of RPC integration, but it should be optional: basically a toolkit to implement a Gazelle extension as an RPC client/server, transparent to the rest of Gazelle.

There's certainly going to be overhead by the way, but I think JSON-RPC can be sufficiently fast. It's the data layer underlying LSP, and many editors use that on every keystroke, so it's good enough latency for interactive work. The extension implementations still need to be fast though. I think Go's Starlark interpreter is reasonably fast, but it's not JIT compiled or anything.

[jayconrod]

From #2232, merger.MergeFile and rule.MergeRule should accept an options struct that modifies behavior and can be extended in the future. From that PR, we'd want a flag that deletes unnecessary # keep comments.

[tyler-french]

@sluongng mentioned this, but my biggest feedback is the context.Context should be passed directly in the function.

If the signature is Fix(ctx, FixArgs), I'm unlikely to forget context. If its Fix(FixArgs{Context: ctx}), I'm much more likely to forget it.

(cc @linzhp) We may want a way to make Indexer tell gazelle whether or not it needs to actually do any work, and actually look at the BUILD files to create the indexer. In other words: if Indexer only uses simple string conversion (similar to Uber's conventions), we don't need to Walk the directories at all.

[jayconrod]

my biggest feedback is the context.Context should be passed directly in the function.

That seems reasonable. Similarly, I think @linzhp also mentioned to me at BazelCon that errors should be returned as separate values not as part of the result structs. It makes the code cleaner both at call sites and return sites. I agree with both of those changes.

We may want a way to make Indexer tell gazelle whether or not it needs to actually do any work, and actually look at the BUILD files to create the indexer. In other words: if Indexer only uses simple string conversion (similar to Uber's conventions), we don't need to Walk the directories at all.

If lazy indexing is enabled by default, I think this is equivalent to simply not returning anything through GenerateResult.RelsToIndex.

[linzhp]

I'm unlikely to forget context.

I think this is a great point that also applied to returned errors. When I talked to Jay, I was thinking more about the convention, but now I realize that it's also a way to prevent people from forgetting to handle the errors.

[mateusz-olczyk]

@jayconrod, I have a suggestion regarding error handling, which came to my mind when working on EngFlow/gazelle_cc#157

rule.Merger.Merge() should allow returning errors. Currently, it's up to the extension implementation to come up with reasonable behavior, which is not obvious:

• Should one log the error message or silently ignore it?
• What should be returned as the bzl.Expr result? It probably should return the original expression without applying any changes, but what was the original expression? The function interface is Merge(other bzl.Expr) bzl.Expr. You cannot deduce from it whether the object should return an expression made from itself or return other. You have to dig into bazel-gazelle, find the calling side, which is rule.mergeAttrValues(), and then you know you should return other, because it stands for the "destination" attribute, which was the original value.

So instead of:

type Merger interface {
	Merge(other bzl.Expr) bzl.Expr
}

There should be:

type Merger interface {
	Merge(other bzl.Expr) (bzl.Expr, error)
}

Things are getting clear now. A potential error is passed up into the rule.MergeRules(), which is responsible for logging the error message and leaving the original attribute value unchanged. Especially, rule.mergePlatformStringsExprs() already allows returning errors and they're properly handled in rule.MergeRules().

[jayconrod]

I'm preparing the initial set of changes for v2, but I could use some feedback on how modules and releases should be structured, going forward. There are two major alternatives on how to accomplish this. I'm leaning toward the subdirectory alternative.

Subdirectory alternative

• Usage: A user depends on @gazelle, before and after upgrading to v2. We don't introduce a new BCR module.
• A repo may import packages from both v1 and v2. A large monorepo can migrate extensions incrementally without breaking.
• v1 and v2 packages are distinguished with Bazel package paths, for example, @gazelle//v2/rule instead of @gazelle//rule.
• v2 source code is stored in the /v2 subdirectory. v1 source code is stored in the repo root directory, where it is now.

This makes use of Go's module subdirectories feature, which was meant to smooth the GOPATH migration. It allowed go get in GOPATH mode to download a repository at a single commit, without needing to know about modules. Code for both major versions is stored at different paths in the same commit.

Branch alternative

• Usage: A user depends on @gazelle_v2. We introduce a new BCR module to match the new Go module.
• A repo may depend on both @gazelle and @gazelle_v2.
• v1 and v2 packages are distinguished by Bazel module name, for example @gazelle_v2//rule instead of @gazelle//rule.
• v2 source code is stored on a separate branch from v1 source code.

For both alternatives

• The new Go module is named github.com/bazel-contrib/bazel-gazelle/v2 (different org and /v2 suffix).
• v1 and v2 packages can co-exist in the same build.
• v1 packages are compatibility shims for v2 packages, with wrapper functions and type aliases.
• v2 packages do not import v1 packages except to refer to types like language.Language for compatibility.

Comparison

• Keeping the gazelle module name will make v2 more discoverable to Bazel users who aren't following this discussion. gazelle_v2 may otherwise be hard to find.
• It would be somewhat confusing if the gazelle BCR module contained something different than the Go module. I don't anticipate this will break anything; it's just a little weird.
• Contributors to Gazelle itself are not likely to be familiar with module subdirectories and may be confused where to put new code.
• Combining source code into a single BCR module would increase the download size a bit (perhaps 100 KiB). But this is not really different than needing to download modules for two separate versions.

[jbedard]

I would vote for the the minimal changes for gazelle users. Gazelle users should be able to update versions in MODULE without caring about the language API or "v2". I think this is the "subdirectory alternative" suggestion.

v2 packages do not import v1 packages except to refer to types like language.Language for compatibility.

This is the only thing I'm uncertain about. Shouldn't the v1 package import v2 since it is a "compatibility shim"? Ideally if a gazelle_binary() contains no languages still using the v1 API then only the "v2" go module is compiled...

It would be somewhat confusing if the gazelle BCR module contained something different than the Go module

IMO go.mod vs bzlmod is already weird. Today they are "the same", but with enough differences that they are not the same. Distinguishing them more might be good honestly. The gazelle bazel module is for configuring+running gazelle binaries via bzlmod, the gazelle go modules are for those writing go code 🤷

[guw]

Thinking of this from a user perspective, what is the expected goal?

As a user I run on Gazelle command and my BUILD files are updated/generated and I can mix and match Gazelle extensions of different major versions in the same workspace.

OR

As a user I need to ensure all extensions are compatible when updating to the new Gazelle version. I would need to run two commands when working with binaries for two different major versions of Gazelle in the same workspace at the same time.

I would be ok with the latter.

What I don't understand ... I thought the idea for a v2 version is to break things. Why is backwards compatibility all of a sudden a requirement? Is the extension ecosystem too large to break?

[fmeum]

I would also prefer the subdirectory approach and don't mind the very minor "bloat" of shipping both versions in one module. As it does seem feasible to keep v1 extensions working that's a small price to pay.

An alternative that is more daring would be to stop shipping the Gazelle Go targets as part of the BCR module and instead consume them via go_deps (well, they need to be shipped to bootstrap go_repository, but not as targets users depend on). This would help with the confusion that Jason brought up by trimming the Gazelle BCR module down to just rules plus aliases in place for the v1 Go targets to preserve compatibility.

[jayconrod]

Thanks all for your feedback! This is helpful. I'm more convinced we should go with a subdirectory rather than a branch. I wouldn't suggest it for a pure Go project, but it makes it easier to depend on Gazelle as a Bazel module.

This is the only thing I'm uncertain about. Shouldn't the v1 package import v2 since it is a "compatibility shim"? Ideally if a gazelle_binary() contains no languages still using the v1 API then only the "v2" go module is compiled...

@jbedard Your understanding is correct. Extensions that import the v1 packages will get compatibility shims over the v2 packages. If all extensions only import the v2 packages, the v1 packages shouldn't be compiled or linked, except possibly for v1 interfaces needed for compatibility.

What I don't understand ... I thought the idea for a v2 version is to break things. Why is backwards compatibility all of a sudden a requirement? Is the extension ecosystem too large to break?

@guw I'd like to avoid incompatibility as much as possible. It creates work for other engineers and creates fragmentation in the ecosystem. I do think it's sometimes necessary, so we aren't burdened by mistakes that only become clear after usage, growth, and experience.

With v2, I mainly want to fix Gazelle's extension interfaces to support error handling and better extensibility (without further incompatible changes). Extension authors will need to adjust their code to the new interfaces.

I want minimal disruption to the end users of Gazelle. Some extensions may take a long time to upgrade to v2 or may not upgrade at all. I think it's important that users are able to mix and match v1 and v2 extensions, and I don't think it will be too much effort to support compatibility with both. From the user's perspective, the distinction is really not important.

An alternative that is more daring would be to stop shipping the Gazelle Go targets as part of the BCR module and instead consume them via go_deps (well, they need to be shipped to bootstrap go_repository, but not as targets users depend on). This would help with the confusion that Jason brought up by trimming the Gazelle BCR module down to just rules plus aliases in place for the v1 Go targets to preserve compatibility.

@fmeum That's an interesting idea. I like it, but it may be difficult to execute. We'd need to leave behind alias targets for packages imported by v1 extensions so their build files still work. The go_repository bootstrapping is harder, though perhaps we could just include sources for fetch_repo and make that a little smarter. We don't need to build gazelle to generate build files for gazelle within go_repository since they're already present.