backend/go: deduplicate third-party module downloads across go.mods

[rdeknijf]

Disclaimer

I'm only a casual Go developer and this is my first Pants work. But this problem has been blocking us for so long that I decided that I'd try to see how far I could come with the best AI I could gather. So it's me plus every model and role I could get my hands on, dredging through possible solutions and taking the simplest one. And then testing it to death, and having yet more models and roles review it. Everything below is Claude talking:

Summary

Third-party Go module analysis is now deduplicated across go.mod files.
Previously, a module required by N go.mod files was downloaded and
analyzed N times, causing O(N*M) downloads and significant memory
overhead in monorepos with many overlapping go.mod files.

Partially addresses #20274.

Motivation

Issue #20274 reports Pants' Go backend OOM-killing machines on monorepos
with multiple go.mod files. Root cause: AnalyzeThirdPartyModuleRequest
included go_mod_address, go_mod_digest, and go_mod_path in its key,
so the engine treated the same module@version required by two
go.mod files as two independent requests. A monorepo with 24 go.mod
files sharing ~700 unique modules would download each shared module up
to 24 times.

Approach

• Replace AnalyzeThirdPartyModuleRequest with ModuleDownloadRequest,
  keyed only on content-addressable inputs:
  (name, version, minimum_go_version, build_opts, go_sum_entries).
• New download_and_analyze_module rule builds a synthetic sandbox
  (go.mod requiring just the one module, plus a synthetic go.sum
  carrying the consuming repo's real checksum lines for that module@version)
  and runs go mod download.
• The engine's built-in memoization then deduplicates: N consumers of
  the same module share a single download.
• The old per-go.mod analyze_go_third_party_module rule, its request
  type, and _check_go_sum_has_not_changed helper are removed.
• go.sum parsing is done once per go.mod into a dict for O(1) lookup
  per module (was O(N*L) with per-module linear scans).

Security

go.sum entries for a given module@version are content-addressable by
definition, so two well-formed consuming go.mods must agree on them.
Including go_sum_entries in the memoization key ensures:

• Happy path: all consumers share one download; the synthetic
  go.sum in the sandbox lets go mod download run its normal checksum
  verification (no GONOSUMCHECK override anywhere).
• Tampered path: disagreeing entries produce distinct requests, each
  verified independently — the tampered one fails with Go's usual
  SECURITY ERROR.
• Missing entries: when a consumer's go.sum lacks entries for a
  transitive module (discovered during MVS), the synthetic go.sum is
  omitted and Go falls back to GOSUMDB — matching effective pre-patch
  behavior.

This was validated by three independent AI code reviews (security
engineer, supply chain specialist, Go module expert) plus mutation
testing, none of which found a security regression.

Benchmarks

Measured on a reproducer with 3 go.mod files sharing grpc / protobuf /
uber-fx / cloud.google.com/go (206 unique modules, 272 total dep entries).
Pants main @ 2.32.0.dev7, isolated LMDB per run, peak RSS of the
process tree sampled at 0.5 s:

Scenario	Before	After	Memory Δ	Time Δ
list :: (3 go.mods)	91.34 GB / 48 s	32.08 GB / 33 s	−65%	−32%
list mod-a:: (1 go.mod)	28.36 GB / 26 s	25.62 GB / 25 s	−10%	−5%
package :: (3 go.mods)	3.02 GB / 133 s	1.51 GB / 95 s	−50%	−29%
package mod-a:: (1 go.mod)	2.88 GB / 103 s	1.26 GB / 77 s	−56%	−25%

Targets listed are byte-identical between before and after; all package
invocations produce the expected binaries.

A real 24-go.mod monorepo completes pants list :: with the patch in
2.58 GB / 112 s / 9,729 targets. The unpatched run was not attempted
— it is the long-standing OOM that motivated the fix.

This is a no-op for repos with a single go.mod.

Tests

• test_cross_go_mod_dedup_produces_identical_results — two go.mod
  files sharing a dep produce byte-identical analyses (including digest).
• test_parse_go_sum — unit test for the go.sum parser: grouping,
  prefix safety (v1.3 must not match v1.3.0), single-entry modules,
  CRLF handling, empty input.
• test_extract_go_sum_entries_for_module — unit test for the
  per-module extraction wrapper.
• test_invalid_go_sum (pre-existing) — tampered hashes trigger Go's
  SECURITY ERROR through the synthetic go.sum path.

Local run: 12 passed / 0 failed / 2 skipped (2 tests skipped are
pre-existing @pytest.mark.skip for #15824).

Known limitation: the test sandbox has network access to GOSUMDB, so
test_invalid_go_sum cannot distinguish "go.sum verification failed
locally" from "GOSUMDB verification failed remotely." Mutation testing
confirmed this gap. The security property (synthetic go.sum is written
and used) is covered by the combination of the integration tests
(test_invalid_go_sum + test_download_and_analyze_all_packages) plus
the _parse_go_sum / _extract_go_sum_entries_for_module unit tests,
but not by a single test in isolation.

Known trade-offs

• minimum_go_version fallback: when a module's GoVersion is
  None, the synthetic go.mod uses go 1.21 as a fallback. This is
  arbitrary; a more principled default (e.g., the module's own go
  directive, or 1.17 as the last major module semantics change) could
  be a follow-up. In practice None is rare and the go directive has
  minimal effect on go mod download -json module@version.
• Partial go.sum detection removed: the old
  _check_go_sum_has_not_changed would warn when go mod download
  added entries to go.sum. This check is removed because the synthetic
  sandbox cannot observe changes to the real go.sum. The user's
  go.sum must already be complete (enforceable by go mod tidy).

Release note

Added under #### Go in docs/notes/2.32.x.md.

Test plan

• ./pants test src/python/pants/backend/go/util_rules/third_party_pkg_test.py passes
• ./pants --changed-since=HEAD fmt clean
• build-support/githooks/pre-push clean
• A/B benchmarks on 3-go.mod reproducer (list and package)
• Patch applied to a real 24-go.mod monorepo completes successfully
• CI full matrix

AI disclosure

This change was developed with assistance from Claude (Anthropic). A
human reviewed every diff, authored the design decisions (synthetic
go.sum over GONOSUMCHECK, dedup key composition, soft go.sum
fallback), ran all benchmarks, and verified correctness. Independent code
reviews were solicited from Claude (Opus, Sonnet), OpenAI Codex
(GPT-5.4), and Google Gemini (2.5 Flash); their findings informed the
current state of the patch. Mutation testing was performed to validate
test coverage.

[rdeknijf]

Hey @tdyas @benjyw — this is ready for review. Happy to address any feedback.

[sureshjoshi]

Thanks for the contribution. We've just branched for 2.32.x, so merging this pull request now will come out in 2.33.x, please move the release notes updates to docs/notes/2.33.x.md if that's appropriate.

[rdeknijf]

Done — moved the release notes entry to docs/notes/2.33.x.md.