metadata resolve workstream

[cosmicexplorer]

What's the problem this feature will solve?

The 2020 resolver separated the resolution logic from the rest of pip, and made the resolver much easier to read and extend. With --use-feature=fast-deps, we began to investigate improved pip performance by avoiding the download of dependencies until we reach a complete resolution. With install --report, we enabled pip users to read the output of the resolver without needing to download dependencies. However, there remain a few blockers to achieving performance improvements, for multiple use cases:

Uncached resolves:

When pip is executed entirely from scratch (without an existing ~/.cache/pip directory), as is often the case in CI, we are unlikely to get too much faster than we are now (and notably, it's extremely unlikely a non-pip tool could go faster in this case without relying on some sort of remote resolve index). However, there are a couple improvements we can still make here:

• Make --use-feature=fast-deps default, to cover for wheels without PEP 658 metadata backfilled yet.
  • Importantly, this is often the case for internal corporate CI indices, which are unlikely to have performed the process of backfilling PEP 658 metadata, and may even be a --find-links repo instead of serving the PyPI simple repository API.
  • fast-deps is not as fast as it could be, which will be addressed further below.
• Establish a separate metadata cache directory, so that CI runs in e.g. github actions can retain the result of metadata resolution run-over-run, without having to store large binary wheel output.
  • We will discuss metadata caching further below--this currently does not exist in pip.
  • This should achieve similar performance to partially-cached resolves, discussed next.
• Batch downloading of wheels all at once after a shallow metadata resolve.
  • Pip already has the infrastructure to do this, it's just not implemented yet!

Partially-cached resolves with downloading

When pip is executed with a persistent ~/.cache/pip directory, we can take advantage of much more caching, and this is the bulk of the work here. In e.g. #11111 and other work, we (mostly) separated metadata resolution from downloading, and this has allowed us to consider how to cache not just downloaded artifacts, but parts of the resolution process itself. This is directly enabled by the clean separation and design of the 2020 resolver. We can cache the following:

• PEP 658 metadata for a particular wheel (this saves us a small download).
• fast-deps metadata for a particular wheel (this saves us a few HTTP range requests).
• Metadata extracted from an sdist (this saves us a medium-size download and a build process).

These alone may not seem like much, but over the course of an entire resolve, not having to make potentially multiple network requests per dependency and staying within our in-memory pip resolve logic adds up and produces a very significant performance improvement. These also reduce the number of requests we make against PyPI.

But wait! We can go even faster! Because in addition to the metadata cache (which is idempotent and not time-varying--the same wheel hash always maps to the same metadata), we can also cache the result of querying the simple repository API for the list of dists available for a given dependency name! This additional caching requires messing around with HTTP caching headers to see if a given page has changed, but it lets us cache:

• The raw content of simple repository index page, if unchanged (this saves us a medium-size download).
• The result of parsing a simple repository index page into Links, if unchanged (this saves us an HTML parser invocation).
• The result of filtering Links by interpreter compatibility, if unchanged (this saves us having to calculate interpreter compatibility using the tags logic).

Resolves without downloading

With install --report out.json --dry-run and the metadata resolve+caching discussed above, we should be able to avoid downloading the files associated with the resolved dists, enabling users to download those dependencies in a later phase (as I achieved at Twitter with pantsbuild/pants#8793). However, we currently don't do so (see #11512), because of a mistake I made in previous implementation work (sorry!). So for this, we just need:

• Avoid downloading dists when invoked from a command which only requests metadata (such as install --dry-run).

Describe the solution you'd like

I have created several PRs which achieve all of the above:

Batch downloading [0/2]

For batch downloading of metadata-only dists, we have two phases:

• improve pooled progress output for BatchDownloader #12925 produces extremely rich progress output for batched downloads.
• execute batch downloads in parallel worker threads #12923 finally makes the BatchDownloader download and prepare metadata-only dists in parallel. This produces a drastic performance improvement.

fast-deps fixes [0/1]

• perform 1-3 HTTP requests for each wheel using fast-deps #12208 fixes the fast-deps implementation to achieve excellent performance against the current iteration of PyPI behind fastly, as well as any other HTTP host supporting range requests.

Formalize "concrete" vs metadata-only dists [0/3]

To avoid downloading dists for metadata-only commands, we have several phases:

• cache "concrete" dists by Distribution instead of InstallRequirement #12863 introduces .is_concrete to our Distribution wrappers to codify the concept of "metadata-only" dists.
• refactor requirement preparer to remove duplicated code paths for metadata-only dists #12871 is a refactoring change to decouple the caching of metadata-only dists from the rest of the RequirementPreparer logic.
• pull preparer logic out of the resolver to consume metadata-only dists in commands #12186 finally fixes use .metadata distribution info when possible #11512, so install --dry-run doesn't download any dists.

Metadata caching [0/1]

• cache metadata lookups for sdists and lazy wheels #12256 introduces the first "metadata cache", which is separate from and much smaller than the cache used for downloaded or built wheels. This produces the most drastic performance improvement to the resolve process.

Caching index pages [0/2]

To optimize the process of obtaining Links to resolve against, we have at least two phases:

• send HTTP caching headers for index pages to further reduce bandwidth usage #12257 attaches HTTP caching headers to requests against index pages, which allows our CacheControl to implicitly retrieve the cached response after a very fast 304 Not Modified from PyPI.
• persistent cache for link parsing and interpreter compatibility #12258 checks whether the index page was updated (e.g. by a new upload) since last downloaded, and if not, retrieves the result of Link parsing and interpreter compatibility filtering from the metadata cache.

Each of these PRs demonstrate some nontrivial performance improvement in their description. All together, the result is quite significant, and never produces a slowdown.

Alternative Solutions

• None of these changes modify any external APIs. They do introduce new subdirectories to ~/.cache/pip, which can be tracked separately from the wheel cache to speed up resolution without ballooning in size.
• I expect the Link parsing and interpreter compatibility caching in persistent cache for link parsing and interpreter compatibility #12258 to involve more discussion, as they take up more cache space than the idempotent metadata cache, produce less of a performance improvement, and are more complex to implement. However, nothing else depends on them to work, and they can safely be discussed later after the preceding caching work is done.

Additional context

In writing this, I realized we may be able to modify the approach of #12257 to work with --find-links repos as well. Those are expected to change much more frequently than index pages using the simple repository API, but may be worth considering after the other work is done.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pelson]

I personally am excited to see metadata being used (without downloading the full dist) to optimise the --dry-run case. This looks like a really comprehensive suite of improvements 👍.

When it comes to testing, I wanted to make you aware of simple-repository-server, which can give you a very functional package index (PEP-503) complete with PEP-658 support. All you need to provide are the wheels, and it can compute the metadata on the fly (perhaps the problem with this though would be that there are no wheels without metadata for testing...). More generally, I gave a presentation last year at EuroPython 2023 on the simple-repository infrastructure that we have developed - whilst not perfect yet, I am convinced that a common simple-repository interface can serve both client and server usage. Imagine on the pip side to be able to request package metadata from a single interface, and behind that interface it can choose to compute the metadata and cache the dist if there is no metadata on the index being used. It would normalize the functionality within pip (you can assume that there is always a metadata file), and move the (http/dist) caching logic into its own layer. I don't want to derail this meta-issue, so can open a discussion elsewhere if you are interested to explore the idea more.

Back on the topic of metadata resolve optimisation, the big one which would speed-up the no-cache resolve is parallel requests. Is this something that you have explored as a possibility? (FWIW, there is an issue for parallel downloads in #825, which mostly pre-dates PEP-658 metadata).

[cosmicexplorer]

@pelson responding to the optimization part first as that's what I have more paged in right now:

Back on the topic of metadata resolve optimisation, the big one which would speed-up the no-cache resolve is parallel requests.

Do you have a benchmark of sorts to demonstrate this? Parallel downloads of dists is actually something we have already established a foundation for; see the use of BatchDownloader in the preparer:

pip/src/pip/_internal/operations/prepare.py

Lines 465 to 484 in 858a515

	batch_download = self._batch_download(
	links_to_fully_download.keys(),
	temp_dir,
	)
	for link, (filepath, _) in batch_download:
	logger.debug("Downloading link %s to %s", link, filepath)
	req = links_to_fully_download[link]
	# Record the downloaded file path so wheel reqs can extract a Distribution
	# in .get_dist().
	req.local_file_path = filepath
	# Record that the file is downloaded so we don't do it again in
	# _prepare_linked_requirement().
	self._downloaded[req.link.url] = filepath
	# If this is an sdist, we need to unpack it after downloading, but the
	# .source_dir won't be set up until we are in _prepare_linked_requirement().
	# Add the downloaded archive to the install requirement to unpack after
	# preparing the source dir.
	if not req.is_wheel:
	req.needs_unpacked_archive(Path(filepath))

I avoided addressing that because it would require figuring out how to represent download progress for parallel downloads, which I wasn't sure about, and I also didn't see an incredible performance improvement when I tested it a few years ago in #7819 (but I'm not sure I was doing the right thing).

I was hesitant at first but now that I recall we already have a basis for parallel downloads it would probably make perfect sense to investigate it. Thanks so much for mentioning it! I'll add it to this issue.

[cosmicexplorer]

When it comes to testing, I wanted to make you aware of simple-repository-server

Thanks so much for linking this!! ^_^ So for e.g. #12208 and #12256, a lot of the testing we're doing here is to cover the variety of ways an index may respond to pip requests, so instead of the conformant behavior it looks like simple-repository-server provides, we're trying to test a whole bunch of arbitrary types of responses so pip can handle them all. See the very very wonky local server logic for HTTP range requests in #12208:

pip/tests/conftest.py

Lines 1175 to 1305 in b06d73b

	class ContentRangeDownloadHandler(
	http.server.SimpleHTTPRequestHandler, metaclass=abc.ABCMeta
	):
	"""Extend the basic ``http.server`` to support content ranges."""
	@abc.abstractproperty
	def range_handler(self) -> RangeHandler: ...
	# NB: Needs to be set on per-function subclasses.
	get_request_counts: ClassVar[Dict[str, int]] = {}
	positive_range_request_paths: ClassVar[Set[str]] = set()
	negative_range_request_paths: ClassVar[Set[str]] = set()
	head_request_paths: ClassVar[Set[str]] = set()
	ok_response_counts: ClassVar[Dict[str, int]] = {}
	@contextmanager
	def _translate_path(self) -> Iterator[Optional[Tuple[BinaryIO, str, int]]]:
	# Only test fast-deps, not PEP 658.
	if self.path.endswith(".metadata"):
	self.send_error(http.HTTPStatus.NOT_FOUND, "File not found")
	yield None
	return
	path = self.translate_path(self.path)
	if os.path.isdir(path):
	path = os.path.join(path, "index.html")
	ctype = self.guess_type(path)
	try:
	with open(path, "rb") as f:
	fs = os.fstat(f.fileno())
	full_file_length = fs[6]
	yield (f, ctype, full_file_length)
	except OSError:
	self.send_error(http.HTTPStatus.NOT_FOUND, "File not found")
	yield None
	return
	def _send_basic_headers(self, ctype: str) -> None:
	self.send_header("Content-Type", ctype)
	if self.range_handler.supports_range():
	self.send_header("Accept-Ranges", "bytes")
	# NB: callers must call self.end_headers()!
	def _send_full_file_headers(self, ctype: str, full_file_length: int) -> None:
	self.send_response(http.HTTPStatus.OK)
	self.ok_response_counts.setdefault(self.path, 0)
	self.ok_response_counts[self.path] += 1
	self._send_basic_headers(ctype)
	self.send_header("Content-Length", str(full_file_length))
	self.end_headers()
	def do_HEAD(self) -> None:
	self.head_request_paths.add(self.path)
	with self._translate_path() as x:
	if x is None:
	return
	(_, ctype, full_file_length) = x
	self._send_full_file_headers(ctype, full_file_length)
	def do_GET(self) -> None:
	self.get_request_counts.setdefault(self.path, 0)
	self.get_request_counts[self.path] += 1
	with self._translate_path() as x:
	if x is None:
	return
	(f, ctype, full_file_length) = x
	range_arg = self.headers.get("Range", None)
	if range_arg is not None:
	m = re.match(r"bytes=([0-9]+)?-([0-9]+)", range_arg)
	if m is not None:
	if m.group(1) is None:
	self.negative_range_request_paths.add(self.path)
	else:
	self.positive_range_request_paths.add(self.path)
	# If no range given, return the whole file.
	if range_arg is None or not self.range_handler.supports_range():
	self._send_full_file_headers(ctype, full_file_length)
	self.copyfile(f, self.wfile) # type: ignore[misc]
	return
	# Otherwise, return the requested contents.
	assert m is not None
	# This is a "start-end" range.
	if m.group(1) is not None:
	start = int(m.group(1))
	end = int(m.group(2))
	assert start <= end
	was_out_of_bounds = (end + 1) > full_file_length
	else:
	# This is a "-end" range.
	if self.range_handler.sneakily_coerces_negative_range():
	end = int(m.group(2))
	self.send_response(http.HTTPStatus.PARTIAL_CONTENT)
	self._send_basic_headers(ctype)
	self.send_header("Content-Length", str(end + 1))
	self.send_header(
	"Content-Range", f"bytes 0-{end}/{full_file_length}"
	)
	self.end_headers()
	f.seek(0)
	self.wfile.write(f.read(end + 1))
	return
	if not self.range_handler.supports_negative_range():
	self.send_response(http.HTTPStatus.NOT_IMPLEMENTED)
	self._send_basic_headers(ctype)
	self.end_headers()
	return
	end = full_file_length - 1
	start = end - int(m.group(2)) + 1
	was_out_of_bounds = start < 0
	if was_out_of_bounds:
	if self.range_handler.overflows_negative_range():
	self._send_full_file_headers(ctype, full_file_length)
	self.copyfile(f, self.wfile) # type: ignore[misc]
	return
	self.send_response(http.HTTPStatus.REQUESTED_RANGE_NOT_SATISFIABLE)
	self._send_basic_headers(ctype)
	self.send_header("Content-Range", f"bytes */{full_file_length}")
	self.end_headers()
	return
	sent_length = end - start + 1
	self.send_response(http.HTTPStatus.PARTIAL_CONTENT)
	self._send_basic_headers(ctype)
	self.send_header("Content-Length", str(sent_length))
	self.send_header("Content-Range", f"bytes {start}-{end}/{full_file_length}")
	self.end_headers()
	f.seek(start)
	self.wfile.write(f.read(sent_length))

However, it was absolutely appropriate for you to mention this, especially in response to e.g. this bit of the issue:

Importantly, this is often the case for internal corporate CI indices, which are unlikely to have performed the process of backfilling PEP 658 metadata, and may even be a --find-links repo instead of serving the PyPI simple repository API.

I was speaking from my knowledge of Twitter from 2017-2020, which initially served a --find-links repo from SVN (hence my change in #7729), then later generated a simple repository index, but for many reasons I suspect wouldn't have caught up to PEP 658 metadata yet. Lazily generating metadata like simple-repository-server sounds like absolutely the right move! While I think pip's own testing requires the ability to generate nonstandard/erroneous responses so we can be sure to handle them correctly, it would definitely absolutely be useful to advertise the use of something like simple-repository-server so e.g. internal corporate repos don't have to rely on workarounds like fast-deps (which is actually very impressively robust now thanks to @dholth's work in #12208, but still a workaround which PEP 658 was created specifically to address).

[cosmicexplorer]

@pelson thanks so much for mentioning parallel downloads! It ended up being much less effort than I expected: see #12923. It's still drafted because I whipped it up very quickly, but please take a look at that and ask others to review! One part I'm not sure how to implement is the parallel progress bar, especially because I believe we may not know Content-Length in advance. However, I believe for servers which do provide Content-Length, we should be able to get the full number of bytes we need to download, and produce a progress bar based on that.