Skip to content

Commit 6044c3f

Browse files
An experimental uv-based PEX builder (#23197)
Add an experimental `[python].pex_builder` option that allows using [uv](https://github.com/astral-sh/uv) to install dependencies when building PEX files via `pants package`. When set to `"uv"`, Pants: 1. Downloads the `uv` binary (as an `ExternalTool`). 2. Creates a virtual environment with `uv venv`. 3. Installs dependencies with `uv pip install`. 4. Passes the pre-populated venv to PEX via `--venv-repository`. When a PEX-native lockfile is available, uv installs the exact pinned versions with `--no-deps`, preserving reproducibility. Otherwise it falls back to transitive resolution from requirement strings. Builds that cannot use uv (internal-only, cross-platform, no local interpreter) silently fall back to the default pip path. ## Benchmark Raw `pip install` vs `uv pip install` for 11 packages (requests, boto3, cryptography, aiohttp, sqlalchemy, pillow, etc.) measured with [hyperfine](https://github.com/sharkdp/hyperfine): | Condition | pip | uv | Speedup | |-----------|-----|-----|---------| | Cold cache | 6.3s | 4.1s | **1.6x faster** | | Warm cache | 6.9s | 0.14s | **51x faster** | Within Pants, the end-to-end improvement is smaller because scheduler and bootstrap overhead dominates, but the dependency installation step itself is significantly faster — especially with warm caches on repeated builds. ## LLM Disclosure Code was written by the author. Claude was used for code review, catching edge cases, and verifying test coverage.
1 parent bebf4a0 commit 6044c3f

6 files changed

Lines changed: 482 additions & 5 deletions

File tree

docs/notes/2.32.x.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,14 @@ Copying the `mypy` cache back to the ["named cache"](https://www.pantsbuild.org/
111111

112112
The mypy subsystem now supports a new `cache_mode="none"` to disable mypy's caching entirely. This is slower and intended as an "escape valve". It is hoped that on the latest mypy with the above Pants side fixes it will not be necessary.
113113

114+
A new **experimental** `[python].pex_builder` option allows using [uv](https://github.com/astral-sh/uv) to install
115+
dependencies when building PEX binaries via `pants package`. When set to `"uv"`, Pants creates a virtual environment
116+
with uv, then passes it to PEX via `--venv-repository` so PEX packages from the pre-populated venv instead of
117+
resolving with pip. When a PEX-native lockfile is available, uv installs the exact pinned versions from the lockfile
118+
with `--no-deps`, preserving reproducibility. This only applies to non-internal, non-cross-platform PEX builds with
119+
explicit requirement strings and a local Python interpreter; other builds silently fall back to pip.
120+
See [#20679](https://github.com/pantsbuild/pants/issues/20679) for background.
121+
114122
The `runtime` field of [`aws_python_lambda_layer`](https://www.pantsbuild.org/2.32/reference/targets/python_aws_lambda_layer#runtime) or [`aws_python_lambda_function`](https://www.pantsbuild.org/2.32/reference/targets/python_aws_lambda_function#runtime) now has built-in complete platform configurations for x86-64 and arm64 Python 3.14. This provides stable support for Python 3.14 lambdas out of the box, allowing deleting manual `complete_platforms` configuration if any.
115123

116124
The `grpc-python-plugin` tool now uses an updated `v1.73.1` plugin built from <https://github.com/nhurden/protoc-gen-grpc-python-prebuilt]. This also brings `macos_arm64` support.

src/python/pants/backend/python/register.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@
3232
from pants.backend.python.macros.python_artifact import PythonArtifact
3333
from pants.backend.python.macros.python_requirements import PythonRequirementsTargetGenerator
3434
from pants.backend.python.macros.uv_requirements import UvRequirementsTargetGenerator
35-
from pants.backend.python.subsystems import debugpy
35+
from pants.backend.python.subsystems import debugpy, uv
3636
from pants.backend.python.target_types import (
3737
PexBinariesGeneratorTarget,
3838
PexBinary,
@@ -70,6 +70,7 @@ def rules():
7070
# Subsystems
7171
*coverage_py.rules(),
7272
*debugpy.rules(),
73+
*uv.rules(),
7374
# Util rules
7475
*ancestor_files.rules(),
7576
*dependency_inference_rules.rules(),

src/python/pants/backend/python/subsystems/setup.py

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,12 @@ class LockfileGenerator(enum.Enum):
4242
POETRY = "poetry"
4343

4444

45+
@enum.unique
46+
class PexBuilder(enum.Enum):
47+
pex = "pex"
48+
uv = "uv"
49+
50+
4551
RESOLVE_OPTION_KEY__DEFAULT = "__default__"
4652

4753
_T = TypeVar("_T")
@@ -312,6 +318,23 @@ def default_to_resolve_interpreter_constraints(self) -> bool:
312318
),
313319
advanced=True,
314320
)
321+
pex_builder = EnumOption(
322+
default=PexBuilder.pex,
323+
help=softwrap(
324+
"""
325+
Which tool to use for installing dependencies when building PEX files.
326+
327+
- `pex` (default): Use pip via PEX.
328+
- `uv` (experimental): Pre-install dependencies into a uv venv, then pass it
329+
to PEX via `--venv-repository`. When a PEX-native lockfile is available,
330+
uv installs the exact pinned versions with `--no-deps`.
331+
332+
Only applies to non-internal, non-cross-platform PEX builds. Other builds
333+
silently fall back to pip.
334+
"""
335+
),
336+
advanced=True,
337+
)
315338
_resolves_to_interpreter_constraints = DictOption[list[str]](
316339
help=softwrap(
317340
"""
Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Copyright 2026 Pants project contributors (see CONTRIBUTORS.md).
2+
# Licensed under the Apache License, Version 2.0 (see LICENSE).
3+
4+
from __future__ import annotations
5+
6+
from dataclasses import dataclass
7+
8+
from pants.core.util_rules.external_tool import (
9+
TemplatedExternalTool,
10+
download_external_tool,
11+
)
12+
from pants.engine.fs import Digest
13+
from pants.engine.platform import Platform
14+
from pants.engine.rules import collect_rules, rule
15+
from pants.option.option_types import ArgsListOption
16+
from pants.util.strutil import softwrap
17+
18+
19+
class Uv(TemplatedExternalTool):
20+
options_scope = "uv"
21+
name = "uv"
22+
help = "The uv Python package manager (https://github.com/astral-sh/uv)."
23+
24+
default_version = "0.6.14"
25+
default_known_versions = [
26+
"0.6.14|macos_x86_64|1d8ecb2eb3b68fb50e4249dc96ac9d2458dc24068848f04f4c5b42af2fd26552|16276555",
27+
"0.6.14|macos_arm64|4ea4731010fbd1bc8e790e07f199f55a5c7c2c732e9b77f85e302b0bee61b756|15138933",
28+
"0.6.14|linux_x86_64|0cac4df0cb3457b154f2039ae471e89cd4e15f3bd790bbb3cb0b8b40d940b93e|17032361",
29+
"0.6.14|linux_arm64|94e22c4be44d205def456427639ca5ca1c1a9e29acc31808a7b28fdd5dcf7f17|15577079",
30+
]
31+
version_constraints = ">=0.6.0,<1.0"
32+
33+
default_url_template = (
34+
"https://github.com/astral-sh/uv/releases/download/{version}/uv-{platform}.tar.gz"
35+
)
36+
default_url_platform_mapping = {
37+
"linux_arm64": "aarch64-unknown-linux-musl",
38+
"linux_x86_64": "x86_64-unknown-linux-musl",
39+
"macos_arm64": "aarch64-apple-darwin",
40+
"macos_x86_64": "x86_64-apple-darwin",
41+
}
42+
43+
def generate_exe(self, plat: Platform) -> str:
44+
platform = self.default_url_platform_mapping[plat.value]
45+
return f"./uv-{platform}/uv"
46+
47+
args_for_uv_pip_install = ArgsListOption(
48+
tool_name="uv",
49+
example="--index-strategy unsafe-first-match",
50+
extra_help=softwrap(
51+
"""
52+
Additional arguments to pass to `uv pip install` invocations.
53+
54+
Used when `[python].pex_builder = "uv"` to pass extra flags to the
55+
`uv pip install` step (e.g. `--index-url`, `--extra-index-url`).
56+
These are NOT passed to the `uv venv` step.
57+
"""
58+
),
59+
)
60+
61+
62+
@dataclass(frozen=True)
63+
class DownloadedUv:
64+
"""The downloaded uv binary with user-configured args."""
65+
66+
digest: Digest
67+
exe: str
68+
args_for_uv_pip_install: tuple[str, ...]
69+
70+
71+
@rule
72+
async def download_uv_binary(uv: Uv, platform: Platform) -> DownloadedUv:
73+
downloaded = await download_external_tool(uv.get_request(platform))
74+
return DownloadedUv(
75+
digest=downloaded.digest,
76+
exe=downloaded.exe,
77+
args_for_uv_pip_install=tuple(uv.args_for_uv_pip_install),
78+
)
79+
80+
81+
def rules():
82+
return collect_rules()

0 commit comments

Comments
 (0)