High memory use in Golang

[JettJones]

Describe the bug
Building our golang projects uses a lot of memory - leading to pantsd exits / OOM kills. Because of this, we get slow build and rebuilds due to a loss of caching, or other crashes from OOM.

Pants version
2.18.1

OS
My setup is on Linux (Ubuntu 22.04)

Additional info
Likely similar to #19053 from earlier pants releases.

I uploaded a simplified reproduction here: https://github.com/JettJones/go-include

• main.go does ~nothing, most of the issue comes from requiring modules.
• in this reproduction, temporal client / server are the largest module sets. In our internal work, more requirements increase the impact of this issue.

Here is the memory summary of pants --stats-memory-summary check ::
& debug logs gist

11:33:29.41 [INFO] Memory summary (total size in bytes, count, name):
  48            1               pants.backend.go.dependency_inference.AllGoModuleImportPathsMappings
  48            1               pants.backend.go.dependency_inference.GoModuleImportPathsMapping
  48            1               pants.backend.go.dependency_inference.GoModuleImportPathsMappings
  48            1               pants.backend.go.goals.check.GoCheckRequest
  48            1               pants.backend.go.subsystems.golang.GolangSubsystem
  48            1               pants.backend.go.subsystems.golang.GolangSubsystem.EnvironmentAware
  48            1               pants.backend.go.subsystems.gotest.GoTestSubsystem
  48            1               pants.backend.go.target_type_rules.FirstPartyGoModuleImportPathsMappingsHook
  48            1               pants.backend.go.target_type_rules.GenerateTargetsFromGoModRequest
  48		1		pants.backend.go.target_type_rules.GoImportPathMappingRequest
  48		1		pants.backend.go.target_type_rules.InferGoPackageDependenciesRequest
  48		1		pants.backend.go.util_rules.build_pkg.CheckForGolangAssemblyRequest
  48		1		pants.backend.go.util_rules.build_pkg.CheckForGolangAssemblyResult
  48		1		pants.backend.go.util_rules.build_pkg.SetupAsmCheckBinary
  48		1		pants.backend.go.util_rules.build_pkg_target._ResolveStdlibEmbedConfigRequest
  48		1		pants.backend.go.util_rules.build_pkg_target._ResolveStdlibEmbedConfigResult
  48		1		pants.backend.go.util_rules.cgo.CGoCompilerWrapperScript
  48		1		pants.backend.go.util_rules.cgo_binaries.CGoBinaryPathRequest
  48		1		pants.backend.go.util_rules.first_party_pkg.FallibleFirstPartyPkgAnalysis
  48		1		pants.backend.go.util_rules.first_party_pkg.FallibleFirstPartyPkgDigest
  48		1		pants.backend.go.util_rules.first_party_pkg.FirstPartyPkgAnalysisRequest
  48		1		pants.backend.go.util_rules.first_party_pkg.FirstPartyPkgDigestRequest
  48		1		pants.backend.go.util_rules.first_party_pkg.FirstPartyPkgImportPath
  48		1		pants.backend.go.util_rules.first_party_pkg.FirstPartyPkgImportPathRequest
  48		1		pants.backend.go.util_rules.go_bootstrap.GoBootstrap
  48		1		pants.backend.go.util_rules.go_mod.NearestAncestorGoModRequest
  48		1		pants.backend.go.util_rules.go_mod.NearestAncestorGoModResult
  48		1		pants.backend.go.util_rules.goroot.GoRoot
  48		1		pants.backend.go.util_rules.implicit_linker_deps.SdkImplicitLinkerDependenciesHook
  48		1		pants.backend.go.util_rules.link.LinkerSetup
  48		1		pants.backend.go.util_rules.link_defs.ImplicitLinkerDependencies
  48		1		pants.backend.go.util_rules.pkg_analyzer.PackageAnalyzerSetup
  48		1		pants.backend.go.util_rules.sdk.GoSdkRunSetup
  48		1		pants.backend.go.util_rules.third_party_pkg.AllThirdPartyPackages
  48		1		pants.backend.go.util_rules.third_party_pkg.AllThirdPartyPackagesRequest
  48		1		pants.backend.go.util_rules.third_party_pkg.ModuleDescriptors
  48		1		pants.backend.go.util_rules.third_party_pkg.ModuleDescriptorsRequest
  48		1		pants.backend.project_info.filter_targets.FilterSubsystem
  48		1		pants.backend.python.goals.lockfile.PythonSyntheticLockfileTargetsRequest
  48		1		pants.backend.python.subsystems.setup.PythonSetup
  48		1		pants.base.build_root.BuildRoot
  48		1		pants.build_graph.build_configuration.BuildConfiguration
  48		1		pants.core.goals.check.Check
  48		1		pants.core.goals.check.CheckResults
  48		1		pants.core.goals.check.CheckSubsystem
  48		1		pants.core.util_rules.distdir.DistDir
  48		1		pants.core.util_rules.environments.AllEnvironmentTargets
  48		1		pants.core.util_rules.environments.DockerPlatformFieldDefaultFactoryRequest
  48		1		pants.core.util_rules.environments.EnvironmentNameRequest
  48		1		pants.core.util_rules.environments.EnvironmentTarget
  48		1		pants.core.util_rules.environments.EnvironmentsSubsystem
  48		1		pants.core.util_rules.search_paths.ValidateSearchPathsRequest
  48		1		pants.core.util_rules.search_paths.ValidatedSearchPaths
  48		1		pants.core.util_rules.source_files.SourceFiles
  48		1		pants.core.util_rules.source_files.SourceFilesRequest
  48		1		pants.core.util_rules.system_binaries.BashBinary
  48		1		pants.core.util_rules.system_binaries.BinaryPath
  48		1		pants.engine.console.Console
  48		1		pants.engine.env_vars.PathEnvironmentVariable
  48		1		pants.engine.environment.ChosenLocalEnvironmentName
  48		1		pants.engine.fs.Workspace
  48		1		pants.engine.internals.build_files.BuildFileOptions
  48		1		pants.engine.internals.build_files.OptionalAddressFamily
  48		1		pants.engine.internals.dep_rules.MaybeBuildFileDependencyRulesImplementation
  48		1		pants.engine.internals.graph.ResolveTargetGeneratorRequests
  48		1		pants.engine.internals.graph.ResolvedTargetGeneratorRequests
  48		1		pants.engine.internals.graph.SubprojectRoots
  48		1		pants.engine.internals.mapper.AddressFamily
  48		1		pants.engine.internals.mapper.SpecsFilter
  48		1		pants.engine.internals.options_parsing._Options
  48		1		pants.engine.internals.parser.BuildFilePreludeSymbols
  48		1		pants.engine.internals.parser.Parser
  48		1		pants.engine.internals.synthetic_targets.AllSyntheticAddressMaps
  48		1		pants.engine.internals.synthetic_targets.SyntheticAddressMapsRequest
  48		1		pants.engine.platform.Platform
  48		1		pants.engine.streaming_workunit_handler.WorkunitsCallbackFactory
  48		1		pants.engine.target.FieldDefaultFactoryResult
  48		1		pants.engine.target.FieldDefaults
  48		1		pants.engine.target.GeneratedTargets
  48		1		pants.engine.target.RegisteredTargetTypes
  48		1		pants.engine.target.TargetTypesToGenerateTargetsRequests
  48		1		pants.engine.unions.UnionMembership
  48		1		pants.goal.stats_aggregator.StatsAggregatorCallbackFactoryRequest
  48		1		pants.goal.stats_aggregator.StatsAggregatorSubsystem
  48		1		pants.init.bootstrap_scheduler.BootstrapStatus
  48		1		pants.option.global_options.GlobalOptions
  48		1		pants.option.global_options.KeepSandboxes
  48		1		pants.option.global_options.UnmatchedBuildFileGlobs
  48		2		pants.engine.env_vars.CompleteEnvironmentVars
  48		2		pants.engine.internals.build_files.AddressFamilyDir
  48		5		pants.base.specs.Specs
  48		10859		pants.engine.environment.EnvironmentName
  56		1		pants.backend.go.util_rules.go_mod.AllGoModTargets
  56		1		pants.engine.streaming_workunit_handler.WorkunitsCallbackFactories
  88		1		builtins.ProcessExecutionEnvironment
  96		2		pants.backend.go.go_sources.load_go_binary.LoadedGoBinary
  96		2		pants.backend.go.go_sources.load_go_binary.LoadedGoBinaryRequest
  96		2		pants.backend.go.util_rules.build_opts.GoBuildOptions
  96		2		pants.backend.go.util_rules.build_opts.GoBuildOptionsFromTargetRequest
  96		2		pants.backend.go.util_rules.build_pkg.RenderEmbedConfigRequest
  96		2		pants.backend.go.util_rules.build_pkg.RenderedEmbedConfig
  96		2		pants.backend.go.util_rules.cgo.CGoCompileRequest
  96		2		pants.backend.go.util_rules.cgo.CGoCompileResult
  96		2		pants.backend.go.util_rules.cgo.SetupCompilerCmdRequest
  96		2		pants.backend.go.util_rules.cgo.SetupCompilerCmdResult
  96		2		pants.backend.go.util_rules.go_mod.GoModInfo
  96		2		pants.backend.go.util_rules.go_mod.GoModInfoRequest
  96		2		pants.backend.go.util_rules.import_analysis.GoStdLibPackages
  96		2		pants.backend.go.util_rules.import_analysis.GoStdLibPackagesRequest
  96		2		pants.backend.go.util_rules.link.LinkGoBinaryRequest
  96		2		pants.backend.go.util_rules.link.LinkedGoBinary
  96		2		pants.engine.internals.graph.OwnersRequest
  96		2		pants.engine.internals.synthetic_targets.SyntheticAddressMaps
  96		2		pants.engine.internals.synthetic_targets.SyntheticTargetsSpecPaths
  96		2		pants.engine.internals.synthetic_targets.SyntheticTargetsSpecPathsRequest
  96		5		pants.engine.internals.graph.Owners
  136		1		(native) session_values
  144		3		pants.backend.go.util_rules.build_pkg.BuiltGoPackage
  144		3		pants.backend.go.util_rules.cgo.CheckCompilerSupportsFlagRequest
  144		3		pants.backend.go.util_rules.cgo.CheckCompilerSupportsOptionResult
  144		3		pants.backend.go.util_rules.sdk.GoSdkToolIDRequest
  144		3		pants.backend.go.util_rules.sdk.GoSdkToolIDResult
  144		3		pants.core.util_rules.system_binaries.BinaryPathRequest
  144		3		pants.core.util_rules.system_binaries.BinaryPaths
  144		3		pants.engine.env_vars.EnvironmentVars
  144		3		pants.engine.env_vars.EnvironmentVarsRequest
  144		3		pants.engine.internals.graph._AdaptorAndType
  144		3		pants.engine.internals.graph._RequestAdaptorAndType
  144		3		pants.engine.internals.parametrize._TargetParametrizationsRequest
  144		3		pants.engine.internals.target_adaptor.TargetAdaptorRequest
  144		4		pants.base.specs.RawSpecs
  144		6		pants.base.specs.RawSpecsWithOnlyFileOwners
  168		3		pants.engine.internals.parametrize._TargetParametrizations
  192		3		pants.engine.internals.target_adaptor.TargetAdaptor
  240		5		pants.engine.target.HydrateSourcesRequest
  240		5		pants.engine.target.HydratedSources
  265		1		(native) construct_scope_
  265		1		(native) construct_scope_check
  265		1		(native) construct_scope_environments_preview
  265		1		(native) construct_scope_filter
  265		1		(native) construct_scope_go_test
  265		1		(native) construct_scope_golang
  265		1		(native) construct_scope_python
  265		1		(native) construct_scope_stats
  265		1		(native) pants.backend.go.util_rules.cgo.make_cgo_compile_wrapper_script
  265		1		(native) pants.backend.go.util_rules.go_mod.find_all_go_mod_targets
  265		1		(native) pants.core.util_rules.distdir.get_distdir
  265		1		(native) pants.core.util_rules.environments.determine_all_environments
  265		1		(native) pants.core.util_rules.environments.determine_local_environment
  265		1		(native) pants.engine.internals.build_files.evaluate_preludes
  265		1		(native) pants.engine.internals.build_files.extract_build_file_options
  265		1		(native) pants.engine.internals.dep_rules.get_build_file_dependency_rules_implementation
  265		1		(native) pants.engine.internals.graph.extract_subproject_roots
  265		1		(native) pants.engine.internals.graph.extract_unmatched_build_file_globs
  265		1		(native) pants.engine.internals.graph.field_defaults
  265		1		(native) pants.engine.internals.graph.find_all_targets
  265		1		(native) pants.engine.internals.graph.find_all_unexpanded_targets
  265		1		(native) pants.engine.internals.graph.target_types_to_generate_targets_requests
  265		1		(native) pants.engine.internals.options_parsing.extract_keep_sandboxes
  265		1		(native) pants.engine.internals.options_parsing.parse_options
  265		1		(native) pants.engine.internals.specs_rules.setup_specs_filter
  265		1		(native) pants.engine.internals.synthetic_targets.all_synthetic_targets
  265		1		(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.bootstrap_status
  265		1		(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.build_configuration_singleton
  265		1		(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.build_root_singleton
  265		1		(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.parser_singleton
  265		1		(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.registered_target_types_singleton
  265		1		(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.union_membership_singleton
  281		1		(native) construct_env_aware_scope_golang
  281		1		(native) pants.backend.go.target_type_rules.go_merge_import_paths_analysis
  281		1		(native) pants.backend.go.util_rules.build_pkg.setup_golang_asm_check_binary
  281		1		(native) pants.backend.go.util_rules.go_bootstrap.resolve_go_bootstrap
  281		1		(native) pants.backend.go.util_rules.go_mod.find_nearest_ancestor_go_mod
  281		1		(native) pants.backend.go.util_rules.goroot.setup_goroot
  281		1		(native) pants.backend.go.util_rules.link.setup_go_linker
  281		1		(native) pants.backend.go.util_rules.pkg_analyzer.setup_go_package_analyzer
  281		1		(native) pants.backend.go.util_rules.sdk.go_sdk_invoke_setup
  281		1		(native) pants.backend.python.goals.lockfile.python_lockfile_synthetic_targets
  281		1		(native) pants.core.util_rules.environments.docker_platform_field_default_factory
  281		1		(native) pants.core.util_rules.environments.extract_process_config_from_environment
  281		1		(native) pants.core.util_rules.environments.get_target_for_environment_name
  281		1		(native) pants.core.util_rules.environments.resolve_environment_name
  281		1		(native) pants.core.util_rules.search_paths.validate_search_paths
  281		1		(native) pants.core.util_rules.system_binaries.get_bash
  281		1		(native) pants.engine.internals.build_files.ensure_address_family
  281		1		(native) pants.engine.internals.build_files.parse_address_family
  281		1		(native) pants.engine.internals.graph.resolve_generator_target_requests
  281		1		(native) pants.engine.internals.platform_rules.complete_environment_vars
  281		1		(native) pants.engine.internals.platform_rules.current_platform
  281		1		(native) pants.engine.internals.platform_rules.environment_path_variable
  281		1		(native) pants.engine.internals.specs_rules.resolve_addresses_from_specs
  281		1		(native) pants.engine.internals.synthetic_targets.get_synthetic_address_maps
  281		1		(native) pants.engine.streaming_workunit_handler.construct_workunits_callback_factories
  281		1		(native) pants.goal.stats_aggregator.construct_callback
  297		1		(native) pants.backend.go.goals.check.check_go
  297		1		(native) pants.backend.go.target_type_rules.generate_targets_from_go_mod
  297		1		(native) pants.backend.go.target_type_rules.go_map_import_paths_by_module
  297		1		(native) pants.backend.go.target_type_rules.infer_go_dependencies
  297		1		(native) pants.backend.go.target_type_rules.map_import_paths_to_packages
  297		1		(native) pants.backend.go.util_rules.build_pkg.check_for_golang_assembly
  297		1		(native) pants.backend.go.util_rules.build_pkg_target.required_build_go_package_request
  297		1		(native) pants.backend.go.util_rules.build_pkg_target.resolve_go_stdlib_embed_config
  297		1		(native) pants.backend.go.util_rules.cgo_binaries.find_cgo_binary_path
  297		1		(native) pants.backend.go.util_rules.first_party_pkg.analyze_first_party_package
  297		1		(native) pants.backend.go.util_rules.first_party_pkg.compute_first_party_package_import_path
  297		1		(native) pants.backend.go.util_rules.first_party_pkg.setup_first_party_pkg_digest
  297		1		(native) pants.backend.go.util_rules.implicit_linker_deps.provide_sdk_implicit_linker_dependencies
  297		1		(native) pants.backend.go.util_rules.third_party_pkg.analyze_module_dependencies
  297		1		(native) pants.backend.go.util_rules.third_party_pkg.download_and_analyze_third_party_packages
  297		1		(native) pants.core.util_rules.source_files.determine_source_files
  312		6		pants.engine.internals.mapper.AddressFamilies
  313		1		(native) pants.core.goals.check.check
  384		8		pants.option.scope.Scope
  384		8		pants.option.scope.ScopedOptions
  432		19		pants.base.specs.RawSpecsWithoutFileOwners
  562		2		(native) pants.backend.go.util_rules.build_pkg.render_embed_config
  562		2		(native) pants.engine.internals.graph.find_owners
  562		2		(native) pants.engine.internals.specs_rules.filter_targets
  562		2		(native) pants.engine.internals.synthetic_targets.get_synthetic_targets_spec_paths
  594		2		(native) pants.backend.go.go_sources.load_go_binary.setup_go_binary
  594		2		(native) pants.backend.go.util_rules.build_opts.go_extract_build_options_from_target
  594		2		(native) pants.backend.go.util_rules.cgo.cgo_compile_request
  594		2		(native) pants.backend.go.util_rules.cgo.setup_compiler_cmd
  594		2		(native) pants.backend.go.util_rules.go_mod.determine_go_mod_info
  594		2		(native) pants.backend.go.util_rules.import_analysis.analyze_go_stdlib_packages
  594		2		(native) pants.backend.go.util_rules.link.link_go_binary
  653		1		(native) scandir
  812		6		(native) digest_file
  843		3		(native) pants.engine.internals.build_files.find_target_adaptor
  843		3		(native) pants.engine.internals.graph._determine_target_adaptor_and_type
  843		3		(native) pants.engine.internals.specs_rules.addresses_from_owners
  843		3		(native) pants.engine.internals.specs_rules.addresses_from_raw_specs_with_only_file_owners
  843		3		(native) pants.engine.internals.specs_rules.resolve_addresses_from_raw_specs
  891		3		(native) pants.backend.go.util_rules.build_pkg.required_built_go_package
  891		3		(native) pants.backend.go.util_rules.cgo.check_compiler_supports_flag
  891		3		(native) pants.backend.go.util_rules.sdk.compute_go_tool_id
  891		3		(native) pants.core.util_rules.system_binaries.find_binary
  891		3		(native) pants.engine.internals.graph.resolve_target_parametrizations
  891		3		(native) pants.engine.internals.platform_rules.environment_vars_subset
  1136		4		(native) select
  1449		5		(native) snapshot
  1485		5		(native) pants.engine.internals.graph.hydrate_sources
  1488		31		pants.backend.go.util_rules.assembly.AssembleGoAssemblyFilesRequest
  1488		31		pants.backend.go.util_rules.assembly.FallibleAssembleGoAssemblyFilesResult
  1488		31		pants.backend.go.util_rules.assembly.FallibleGenerateAssemblySymabisResult
  1488		31		pants.backend.go.util_rules.assembly.GenerateAssemblySymabisRequest
  1686		6		(native) pants.engine.internals.specs_rules.address_families_from_raw_specs_without_file_owners
  1686		6		(native) pants.engine.internals.specs_rules.addresses_from_raw_specs_without_file_owners
  2248		8		(native) pants.engine.internals.options_parsing.scope_options
  3408		71		pants.backend.go.go_sources.load_go_binary.NaiveBuildGoPackageRequestForStdlibPackageRequest
  8016		170		pants.backend.go.util_rules.build_pkg_target.BuildGoPackageRequestForStdlibRequest
  9207		31		(native) pants.backend.go.util_rules.assembly.assemble_go_assembly_files
  9207		31		(native) pants.backend.go.util_rules.assembly.generate_go_assembly_symabisfile
  9552		199		pants.backend.go.target_type_rules.InferGoThirdPartyPackageDependenciesRequest
  9552		199		pants.backend.go.util_rules.third_party_pkg.ThirdPartyPkgAnalysis
  9552		199		pants.backend.go.util_rules.third_party_pkg.ThirdPartyPkgAnalysisRequest
  9600		200		pants.backend.go.util_rules.build_pkg_target.BuildGoPackageTargetRequest
  9600		200		pants.engine.target.ExplicitlyProvidedDependencies
  9600		200		pants.engine.target.ExplicitlyProvidedDependenciesRequest
  9600		200		pants.engine.target.InferredDependencies
  9600		1000		pants.engine.target.DependenciesRequest
  15360		320		pants.backend.go.util_rules.import_config.ImportConfig
  15360		320		pants.backend.go.util_rules.import_config.ImportConfigRequest
  17616		367		pants.backend.go.util_rules.build_pkg.FallibleBuildGoPackageRequest
  18480		385		pants.backend.go.util_rules.third_party_pkg.AnalyzeThirdPartyModuleRequest
  18480		385		pants.backend.go.util_rules.third_party_pkg.AnalyzedThirdPartyModule
  19104		398		pants.build_graph.address.MaybeAddress
  21087		71		(native) pants.backend.go.go_sources.load_go_binary.naive_build_go_package_request_for_stdlib
  21120		440		pants.backend.go.util_rules.build_pkg.FallibleBuiltGoPackage
  21120		440		pants.backend.go.util_rules.build_pkg.GoCompileActionIdRequest
  21120		440		pants.backend.go.util_rules.build_pkg.GoCompileActionIdResult
  21120		513		pants.backend.go.util_rules.build_pkg.BuildGoPackageRequest
  40512		844		pants.engine.process.ProcessResult
  40512		844		pants.engine.process.ProductDescription
  42384		1745		pants.backend.go.util_rules.sdk.GoSdkProcess
  49599		167		(native) pants.backend.go.util_rules.build_pkg_target.setup_build_go_package_target_request_for_stdlib
  49984		1		pants.engine.target.AllTargets
  49992		1		pants.engine.target.AllUnexpandedTargets
  50032		2		pants.engine.target.FilteredTargets
  56200		200		(native) pants.engine.internals.graph.convert_dependencies_request_to_explicitly_provided_dependencies_request
  56200		200		(native) pants.engine.internals.graph.determine_explicitly_provided_dependencies
  56200		200		(native) pants.engine.internals.graph.resolve_dependencies
  57312		398		builtins.Address
  57324		204		(native) pants.engine.internals.graph.resolve_targets
  57605		205		(native) pants.engine.internals.graph.resolve_unexpanded_targets
  59103		199		(native) pants.backend.go.target_type_rules.infer_go_third_party_package_dependencies
  59103		199		(native) pants.backend.go.util_rules.third_party_pkg.extract_package_info
  59400		200		(native) pants.backend.go.util_rules.build_pkg_target.setup_build_go_package_target_request
  62208		1709		pants.engine.process.Process
  66864		796		builtins.AddressInput
  89920		320		(native) pants.backend.go.util_rules.import_config.generate_import_config
  111838		398		(native) pants.engine.internals.build_files.maybe_resolve_address
  111838		398		(native) pants.engine.internals.build_files.resolve_address
  114345		385		(native) pants.backend.go.util_rules.third_party_pkg.analyze_go_third_party_module
  118632		204		pants.engine.target.Targets
  130680		440		(native) pants.backend.go.util_rules.build_pkg.build_go_package
  130680		440		(native) pants.backend.go.util_rules.build_pkg.compute_compile_action_id
  168640		205		pants.engine.target.UnexpandedTargets
  244060		844		(native) pants.engine.process.get_multi_platform_request_description
  250668		844		(native) pants.engine.process.fallible_to_exec_result_or_raise
  262251		883		(native) pants.backend.go.util_rules.sdk.setup_go_sdk_process
  299520		6240		pants.backend.go.util_rules.go_mod.OwningGoMod
  299520		6240		pants.backend.go.util_rules.go_mod.OwningGoModRequest
  299664		6243		pants.engine.target.WrappedTarget
  299664		6243		pants.engine.target.WrappedTargetRequest
  310560		6470		pants.backend.go.util_rules.third_party_pkg.AnalyzeThirdPartyPackageRequest
  310560		6470		pants.backend.go.util_rules.third_party_pkg.FallibleThirdPartyPkgAnalysis
  318856		213		pants.engine.addresses.Addresses
  1753440		6240		(native) pants.backend.go.util_rules.go_mod.find_owning_go_mod
  1754283		6243		(native) pants.engine.internals.graph.resolve_target
  1921590		6470		(native) pants.backend.go.util_rules.third_party_pkg.analyze_go_third_party_package
  11513123		1346		(native) process

[benjyw]

Thanks for the repro! That will be very helpful in profiling this.

[tdyas]

I wonder if a lack of structure sharing in the Go build graph (BuildGoPackageRequest) contributes to the issue. If the buld graph for the standard library is not being reused, that could contribute greatly to the issue.

[benjyw]

That seems fairly likely. Tangentially similar: #20030

[benjyw]

Hmm, so looking at this, I don't actually see redundancy. Those 6470 AnalyzeThirdPartyPackageRequest objects correspond to 6470 distinct go packages. The trouble may be that this many pending Process invocations (with somewhat large inputs) is more than the engine can handle with reasonable memory consumption.

Since individual packages within a third-party module can't change given a module version, maybe we need to switch to one-process-per-module instead of one-process-per-package? Those 6470 packages belong to just 385 modules.

@tdyas thoughts?

[benjyw]

Note that I incidentally noticed and fixed #20332 while looking into this, but it is not the cause of this issue.

[tdyas]

Since individual packages within a third-party module can't change given a module version, maybe we need to switch to one-process-per-module instead of one-process-per-package? Those 6470 packages belong to just 385 modules.

That would be a reasonable way to batch the work.

[johansja]

Any update on this? Have just tried adding Pants as our build tool for our project since last week. Generally, it fits quite well with support of Python, Go, Docker and Helm as that is our pipeline as well.

However, high memory usage and constant redownloading of Go modules affects the user experience.

[benjyw]

I believe @JettJones is working on this?

[JettJones]

I believe @JettJones is working on this?

I tried a version of downloading go modules into named_cache - and while it solved the memory problems, that structure broke assumptions around cache lifetime and didn't have a path forward for remote execution. So it's largely back to the drawing board, and I don't have an active plan at the moment.

[benjyw]

@tdyas sounds like we really need to solve this. Any ideas?