Infer dependencies across local replace directives in the Go backend (closes #22097)#23431
Open
rdeknijf wants to merge 1 commit into
Open
Infer dependencies across local replace directives in the Go backend (closes #22097)#23431rdeknijf wants to merge 1 commit into
replace directives in the Go backend (closes #22097)#23431rdeknijf wants to merge 1 commit into
Conversation
…tsbuild#22097) A `go.mod` may `replace` a required module with a local directory holding another first-party module. Pants skipped such modules during third-party analysis and never folded their packages into the referencing module's import-path map, so imports of a locally-replaced module's packages were not inferred; users had to enumerate every cross-module dependency explicitly in BUILD metadata. `determine_go_mod_info` now records local-directory `replace` targets, and `map_import_paths_to_packages` merges those modules' import-path maps into the referencing module's map. Go applies `replace` directives only within the module that declares them, so the merge is one level deep.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclaimer: Like my previous Go PR, I'm still not primarily a golang developer. However, the fact that Golang doesn't work properly in Pants has been the bane of my existence for like 2 years now. The moment that Mythos/Fable dropped I immediately had it dig deeply into whether the whole road to proper-golang-in-pants could be opened up. So this is all by Claude Code with Fable 5 (xhigh), with consults to GPT 5.5 (xhigh) and Gemini 3.1 Pro. I've had it check and recheck, I ran many different roles over it, and I had it explain and re-explain it to me, and then I checked myself.
So, as much as I dislike AI slop and am worried about AI PR overload, I've done my very best to avoid exactly that while still using AI. I hope we can get this road unblocked.
The rest is Fable talking:
Summary
Infer dependencies across local
replacedirectives, so that ago_packagethat imports another first-party module wired in through a directoryreplaceno longer needs its cross-module dependencies enumerated by hand.This closes the
TODO Issue #22097that was sitting inthird_party_pkg.py.This is a correctness change, independent of my open Go performance PRs (#23420, #23421, #23422, #23424). It does not depend on any of them and can be reviewed on its own. For a monorepo that wires its first-party modules together with directory
replacedirectives, it is the difference between dependency inference working and not working, so of my open Go changes this is the one I would prioritize for review.Problem
A
go.modcanreplacea required module with a path to a local directory that holds another first-party module:During third-party module analysis Pants skips these (they are not third-party packages), but nothing ever folded the replaced module's packages into the referencing module's import-path map. So an import of
example.com/shared/pkg/fieldfromcontext/appwas never resolved by dependency inference, and the user had to list every cross-module dependency explicitly in BUILD metadata. In a repo where the entire first-party graph is wired together with directory replaces, that is most of the dependency edges.Change
Three pieces, all in the Go backend:
determine_go_mod_infonow records eachreplacewhose target is a local directory, mapping the replaced module path to that directory (relative to the build root). It reads this from thego modJSON metadata Pants already fetches, so no extra process is run. Module-path replacements (a path plus a version) are left to normal third-party resolution. A small_is_directory_replacementhelper mirrors Go's ownmodfile.IsDirectoryPathrule for what counts as a directory target.map_import_paths_to_packagesfolds the replaced modules' packages into the referencing module's import-path map. Because Go appliesreplacedirectives only within the module that declares them (they are not transitive), the merge is a single level deep. When a module has no directory replaces the rule returns the base mapping unchanged, so this is a no-op for the common case.The skip in
analyze_module_dependenciesand its stale TODO are updated to point at where the inference now happens.The one subtlety worth calling out
When merging, only the replaced module's own packages are folded in, that is, import paths equal to or under its module path. Its third-party dependencies are deliberately not merged. The referencing module resolves those through its own
go.mod, and if the shared third-party import paths were merged in too they would map to two addresses and become ambiguous, which silently suppresses the inferred dependency. The prefix filter is what keeps shared third-party imports (protobuf, grpc, cloud SDKs) inferring correctly. There is a dedicated test for exactly this.Tests
Two new tests in
target_type_rules_test.py:test_cross_module_local_replace_dependency_inference: an import satisfied by a locally-replaced sibling module is inferred with no explicit dependency.test_cross_module_local_replace_does_not_shadow_third_party: a third-party import shared by both modules still infers correctly and is not made ambiguous by the merge.Full
src/python/pants/backend/go/util_rules/::andtarget_type_rulessuites pass. Validated end to end on a real 24-module monorepo: every cross-module import that previously needed a hand-written dependency now infers, with no explicit dependencies, and the build is otherwise unchanged.Security
No new process execution, no network access, no new filesystem reads. The change parses
replaceentries out of metadata Pants already collects and augments an in-memory inference map. Directory paths are normalized withos.path.normpath; absolute on-disk targets are skipped because they cannot map to an in-repogo_modtarget. A replaced directory only contributes packages if it resolves to an existinggo_modtarget in the repo, so nothing outside the build graph is pulled in.Compatibility
No behavior change for any module that does not use a directory
replace: the merge path is skipped entirely when there are none. No options, no lockfile or cache-key changes.Release note (add to
docs/notes/2.33.x.md, under#### Go)Dependency inference now resolves imports across local directory
replacedirectives. When ago.modreplaces a required module with a path to another first-party module in the repo, imports of that module's packages are inferred automatically instead of having to be listed by hand in BUILD metadata. This is a no-op for modules without directoryreplacedirectives. See #22097.