Support universal requirements for multiple python versions

[keith]

🐞 bug report

Affected Rule

pip.parse

Is this a regression?

I don't think so

Description

Some dependencies can vary based on the current python version. In this case for requirements.txt files generated by uv or similar tools, the python version is encoded in the requirements.txt:

numpy==2.0.2 ; python_full_version < '3.10'
    # via -r requirements.in
numpy==2.2.5 ; python_full_version >= '3.10'
    # via -r requirements.in

In this case if you attempt to use this requirements file with pip.parse you get this error:

ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/extension.bzl:224:25: Traceback (most recent call last):
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/extension.bzl", line 607, column 25, in _pip_impl
                mods = parse_modules(module_ctx)
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/extension.bzl", line 478, column 36, in parse_modules
                out = _create_whl_repos(
        File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/extension.bzl", line 224, column 25, in _create_whl_repos
                fail("Attempting to creating a duplicate library {} for {}".format(
Error in fail: Attempting to creating a duplicate library pip_311_numpy_linux_aarch64_linux_arm_linux_ppc_linux_s390x_linux_x86_64_osx_aarch64_osx_x86_64_windows_x86_64 for numpy

Ideally rules_python would know that there should only be a single repo here since these constraints are mutually exclusive

🔬 Minimal Reproduction

bazel build ... in #2798

🌍 Your Environment

Operating System:

  
linux x86_64
  

Output of bazel version:

  
8.2.1
  

Rules_python version:

c981569

[aignas]

For experimental_index_url code path where the sha256 is included in the repo name, this will not be an issue.

[baxelrod-bdai]

I am also getting this error.

Minimal reproduction

requirements.in has just:

ipython >= 5

and BUILD.bazel has:

load("@rules_uv//uv:pip.bzl", "pip_compile")

pip_compile(
    name = "generate_requirements",
    extra_args = [
        "--universal",
    ],
    requirements_in = "requirements.in",
    requirements_txt = "requirements.lock",
)

bazel run //:generate_requirements runs fine, and creates a lock file with:

ipython==8.35.0 ; python_full_version < '3.11' \
    --hash=sha256:d200b7d93c3f5883fc36ab9ce28a18249c7706e51347681f80a0aef9895f2520 \
    --hash=sha256:e6b7470468ba6f1f0a7b116bb688a3ece2f13e2f94138e508201fad677a788ba
    # via -r requirements.in
ipython==9.1.0 ; python_full_version >= '3.11' \
    --hash=sha256:2df07257ec2f84a6b346b8d83100bcf8fa501c6e01ab75cd3799b0bb253b3d2a \
    --hash=sha256:a47e13a5e05e02f3b8e1e7a0f9db372199fe8c3763532fe7a1e0379e4e135f16
    # via -r requirements.in

But when you try to bazel run anything else, I get the crash:

ERROR: /home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python+/python/private/pypi/extension.bzl:231:25: Traceback (most recent call last):
	File "/home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python+/python/private/pypi/extension.bzl", line 611, column 25, in _pip_impl
		mods = parse_modules(module_ctx)
	File "/home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python+/python/private/pypi/extension.bzl", line 482, column 36, in parse_modules
		out = _create_whl_repos(
	File "/home/baxelrod/.cache/bazel/_bazel_baxelrod/223f490fa0c2f2ec38707ddac3dd6272/external/rules_python+/python/private/pypi/extension.bzl", line 231, column 25, in _create_whl_repos
		fail("Attempting to creating a duplicate library {} for {}".format(
Error in fail: Attempting to creating a duplicate library pip_310_ipython_linux_aarch64_linux_arm_linux_ppc_linux_s390x_linux_x86_64_osx_aarch64_osx_x86_64_windows_x86_64 for ipython
ERROR: error evaluating module extension @@rules_python+//python/extensions:pip.bzl%pip

Versions

Bazelisk version: v1.19.0
Build label: 8.1.1

bazel_dep(name = "rules_python", version = "1.4.0-rc1")
bazel_dep(name = "rules_uv", version = "0.68.0")

Linux baxelrod-ThinkPad 6.8.0-57-generic #59~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Mar 19 17:07:41 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Notes

I tried adding this to my .bazelrc to force things to use a single Python version, but no luck.

build --action_env=PYTHON_VERSION=3.10
run --action_env=PYTHON_VERSION=3.10

The problem goes away if I don't specify --universal in the uv pip compile arguments. Because then no markers are added in the lock file.

[baxelrod-bdai]

@aignas can you explain your comment? Is this a workaround?

For experimental_index_url code path where the sha256 is included in the repo name, this will not be an issue.

[keith]

with your requirements file, since it has hashes, if you set experimental_index_url = "https://pypi.org/simple", on pip.parse it will not fail with this error.

In my case it seems using experimental_index_url causes a 50% regression in dependency download time, but I plan to file a separate issue for that.

I am also curious if the plan is to require this codepath for this fix, in which case maybe we should add a more clear error