bootstrap_impl=script blocks creation of tar with rules_pkg/tar.bzl

[TvdW]

🐞 bug report

Affected Rule

py_binary

Is this a regression?

Yes, in that prior to 8ff4386 packaging was successful (but I haven't confirmed whether the output was usable)

Description

rules_python creates a venv directory containing symlinks to a Python interpreter, and expects to use this venv from the runfiles directory.

However, the venv is declared as both output and runfiles, so it shows up twice under bazel-bin:

# ls -al bazel-bin/_test.venv/bin/
total 0
python3@ -> ../../../rules_python++python+python_3_11_aarch64-apple-darwin/bin/python3

# ls -al bazel-bin/test.runfiles/_main/_test.venv/bin/
total 0
python3@ -> ../../../rules_python++python+python_3_11_aarch64-apple-darwin/bin/python3

While the second symlink is correct, the first one points at a binary that does not exist (../../../ goes up to the project root, which does not contain rules_python++python+python_3_11_aarch64-apple-darwin).

When the TAR library (either rules_pkg or tar.bzl) then attempts to store these, it fails for two reasons:

1. rules_pkg (latest from HEAD, not just from the registry!) complains about storing the same file twice (DEBUG: .../external/rules_pkg+/pkg/mappings.bzl:307:34: same source mapped to different locations <generated file _test.venv/bin/python3> test.runfiles/_main/_test.venv/bin/python3 test.py.runfiles/_main/_test.venv/bin/python3). It then fails trying to add the file: FileNotFoundError: [Errno 2] No such file or directory: 'bazel-out/darwin_arm64-fastbuild/bin/_test.venv/bin/python3'
2. tar.bzl (libarchive) fails for the same reason: tar: Error reading archive bazel-out/darwin_arm64-fastbuild/bin/test_tar_mtree.txt: Can't open bazel-out/darwin_arm64-fastbuild/bin/_test.venv/bin/python3

Now, to be fair, those two libraries should probably recognize that they're adding symlinks and not try to read the contents, but I do think the symlinks should at least be valid.

I suspect that the fix here should be to make sure that the venv is only considered as runfiles, so that the bazel-bin/_test.venv/ is not created at all.

🔬 Minimal Reproduction

# MODULE.bazel
module(
    name = "test",
    version = "1.0",
)

bazel_dep(name = "rules_python", version = "1.4.0")
# local_path_override(module_name = "rules_python", path = "...")

bazel_dep(name = "tar.bzl", version = "0.2.0")

# BUILD.bazel
load("@rules_python//python:defs.bzl", "py_binary")

py_binary(
    name = "test",
    srcs = [":test.py"],
)

tar(
    name = "test_tar",
    srcs = [":test"],
)

# test.py
# empty file

🔥 Exception or Error

tar.bzl

Tar test_tar.tar failed: (Exit 1): tar failed: error executing Tar command (from target //:test_tar) external/aspect_bazel_lib++toolchains+bsd_tar_darwin_arm64/tar --create --file bazel-out/darwin_arm64-fastbuild/bin/test_tar.tar @bazel-out/darwin_arm64-fastbuild/bin/test_tar_mtree.txt

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
tar: Error reading archive bazel-out/darwin_arm64-fastbuild/bin/test_tar_mtree.txt: Can't open bazel-out/darwin_arm64-fastbuild/bin/_test.venv/bin/python3
tar: Error exit delayed from previous errors.
Target //:test_tar failed to build

rules_pkg // tar

  File "............./bin/external/rules_pkg+/pkg/private/tar/build_tar.runfiles/rules_pkg+/pkg/private/tar/tar_writer.py", line 277, in add_file
    with open(file_content, 'rb') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'bazel-out/darwin_arm64-fastbuild/bin/_test.venv/bin/python3'

🌍 Your Environment

Operating System:

Mac sequoia

Output of bazel version:

  
Bazelisk version: 1.26.0
Build label: 8.2.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Thu Apr 17 18:31:45 2025 (1744914705)
Build timestamp: 1744914705
Build timestamp as int: 1744914705
  

Rules_* version:

tested with rules_pkg and rules_python at HEAD

Anything else relevant?

I think this needs a section of "why this is maybe not covered by existing issues":

• --bootstrap_impl=script breaks pkg_tar, bazel-lib tar and py_binary with py_package + py_wheel #2489 has a comment that identified the problem but the resolution was different ("The problem as I see it, is that the venv is created both in the runfiles directory, but also in the directory containing the runfiles directory").
• recreating the venv and build_python_zip are problematic due to having a read-only filesystem with noexec on /tmp. fix: fixes to prepare for making bootstrap=script the default for Linux #2760 briefly touches on this in the comments. Is it possible the answer here is just "yeah, don't do that" ?

[rickeylev]

However, the venv is declared as both output and runfiles
bazel-bin/_test.venv/bin

I've seen this showing up, but I don't know why. The venv files aren't part of the default outputs and should only be showing up under runfiles. Hm, maybe it's because they're generated files? usually everything in runfiles are symlinks pointing back to the actual file (which would be in bazel-{bin/out} or bazel-genfiles, though I think the latter is "deprecated"). 🤷

In any case, it's not usable, nor is it intended to be (the leading "_" on its name is a hint that it's a private impl detail) -- the created symlinks are relative within the runfiles tree, so can only be used within that tree.

not working with pkg_tar at head

Hrm. I put a fix in to rules_pkg upstream, so this should work.

---

The escape hatch here is --venvs_use_declare_symlink: https://rules-python.readthedocs.io/en/latest/api/rules_python/python/config_settings/index.html#venvs_use_declare_symlink

For the read-only temp situation, there's RULES_PYTHON_EXTRACT_ROOT: https://rules-python.readthedocs.io/en/latest/environment-variables.html#envvar-RULES_PYTHON_EXTRACT_ROOT