Sharing mypy cache for the same project with different buildroots

[csqzhang]

Is your feature request related to a problem? Please describe.
I'd would like to report an issue when we migrate to pants 2.15 from 2.14. We use Pants in our Jenkins CI/CD pipeline. In our setup, Jenkins will work on each pipeline in different folders. For example, a branch name AB would be under /tmp/workspace/AB, and branch name CD would be under /tmp/workspace/CD. We notice that Pants will not be able to share mypy cache across branch after 2.15. I believe the issue comes from this pull request #18061. It seems that Pants now replies on buildroot (e.g. a path) to differentiate repos.

ref: https://pantsbuild.slack.com/archives/C046T6T9U/p1682039225688279

Describe the solution you'd like

1. Provide an option to disable the option of creating mypy cache per buildroot?
   or 2. use some pre-defined env variable to decide the repo?

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[huonw]

Sorry for the impact (I think I noticed/filed the original issue!).

Just confirming: is the only impact of this pants check ... in the different build roots is slower? Or, are there other non-performance impacts too?

[paiforsyth]

I am on Qiang's team, and so far we have only noticed performance impacts.

[csqzhang]

Hi team, I am wondering if there is a plan on this issue?

[thejcannon]

Since this has a potential impact for soundness of the operations, it'd have to be approached very carefully.

I don't think it's on anyone's radar that is a maintainer, so you'd likely have to be doing the work yourselves. Feel free to get a discussion going in #development on Slack if that's something you'd wanna pick up

[huonw]

Relevant code:

pants/src/python/pants/backend/python/typecheck/mypy/rules.py

Lines 76 to 105 in 0392d37

	async def _generate_argv(
	mypy: MyPy,
	*,
	pex: VenvPex,
	cache_dir: str,
	venv_python: str,
	file_list_path: str,
	python_version: Optional[str],
	) -> Tuple[str, ...]:
	args = [pex.pex.argv0, f"--python-executable={venv_python}", *mypy.args]
	if mypy.config:
	args.append(f"--config-file={mypy.config}")
	if python_version:
	args.append(f"--python-version={python_version}")
	mypy_pex_info = await Get(PexResolveInfo, VenvPex, pex)
	mypy_info = mypy_pex_info.find("mypy")
	assert mypy_info is not None
	if mypy_info.version > packaging.version.Version("0.700") and python_version is not None:
	# Skip mtime checks because we don't propogate mtime when materialzing the sandbox, so the
	# mtime checks will always fail otherwise.
	args.append("--skip-cache-mtime-check")
	# See "__run_wrapper.sh" below for explanation
	args.append("--sqlite-cache") # Added in v 0.660
	args.extend(("--cache-dir", cache_dir))
	else:
	# Don't bother caching
	args.append("--cache-dir=/dev/null")
	args.append(f"@{file_list_path}")
	return tuple(args)

pants/src/python/pants/backend/python/typecheck/mypy/rules.py

Line 224 in 0392d37

mypy_cache_dir = f"{named_cache_dir}/{sha256(build_root.path.encode()).hexdigest()}"

Maybe a path forward here would be to make the {sha256(build_root.path.encode()).hexdigest()} part of that second link customisable, for instance, a new cache_path_disambiguator option with default value default=lambda _: pants.base.build_environment.get_buildroot()

pants/src/python/pants/backend/python/typecheck/mypy/subsystem.py

Lines 66 to 120 in 0392d37

	class MyPy(PythonToolBase):
	options_scope = "mypy"
	name = "MyPy"
	help = "The MyPy Python type checker (http://mypy-lang.org/)."
	default_main = ConsoleScript("mypy")
	default_requirements = ["mypy>=0.961,<2"]
	# See `mypy/rules.py`. We only use these default constraints in some situations.
	register_interpreter_constraints = True
	default_lockfile_resource = ("pants.backend.python.typecheck.mypy", "mypy.lock")
	skip = SkipOption("check")
	args = ArgsListOption(example="--python-version 3.7 --disallow-any-expr")
	config = FileOption(
	default=None,
	advanced=True,
	help=lambda cls: softwrap(
	f"""
	Path to a config file understood by MyPy
	(https://mypy.readthedocs.io/en/stable/config_file.html).
	Setting this option will disable `[{cls.options_scope}].config_discovery`. Use
	this option if the config is located in a non-standard location.
	"""
	),
	)
	config_discovery = BoolOption(
	default=True,
	advanced=True,
	help=lambda cls: softwrap(
	f"""
	If true, Pants will include any relevant config files during runs
	(`mypy.ini`, `.mypy.ini`, and `setup.cfg`).
	Use `[{cls.options_scope}].config` instead if your config is in a non-standard location.
	"""
	),
	)
	_source_plugins = TargetListOption(
	advanced=True,
	help=softwrap(
	f"""
	An optional list of `python_sources` target addresses to load first-party plugins.
	You must also set `plugins = path.to.module` in your `mypy.ini`, and
	set the `[mypy].config` option in your `pants.toml`.
	To instead load third-party plugins, set the option `[mypy].install_from_resolve`
	to a resolve whose lockfile includes those plugins, and set the `plugins` option
	in `mypy.ini`. See {doc_url('python-check-goal')}.
	"""
	),
	)

This becomes a sharp "you better know what you're doing"/"if there's problem, it's your own fault" configuration option, but opens the door for more control.

(E.g. a user had a different-but-related request around customising the cache based on different mypy args https://pantsbuild.slack.com/archives/C046T6T9U/p1724987069966649?thread_ts=1724161347.765349&cid=C046T6T9U)