pants export-codegen fails with "Error copying bytes" when run across entire repo

[kdowney-talos]

Discussed in #21713

This particular repo has a number of modules, but only one has code to generate: a repo with Protobuf files. However, if I widen it and run pants export-codegen :: for every module, I get what looks like a race condition related to copying a CSV file. I isolated it to just one module, serenity.risk -- it fails even if I just do pants export-codegen serenity.risk::

(1) It's unclear to me why this file would be in scope for code generation
(2) Regardless, the error itself seems like a red herring, an error changing file permissions

I tried further narrowing to sub-directories, etc., and it does seem to be this particular file.

The only notable thing about this file is its size, more than 500K:

-rw-r--r--  1 serenitydev serenitydev 574136 May 13  2024  DAR_prices_data.csv

Additional information:

• Observed with pants 2.23.0 and 2.25.0.dev1
• Running on Ubuntu jammy image
• Inside Docker container, running in Docker Desktop 4.37 on MacOS 15.1.1

(.venv) serenitydev@adb319d55c1a:/workspaces/pms$ pants export-codegen ::
16:11:29.08 [INFO] Writing generated files to dist/codegen
16:11:29.09 [ERROR] 1 Exception encountered:

Engine traceback:
  in `export-codegen` goal

IntrinsicError: Error copying bytes from /home/serenitydev/.cache/pants/lmdb_store/immutable/files/2d/2d522219da490246fd3842165d3ebb7362314e218a510c16daeb09bc8d5fc5a6 to /workspaces/pms/dist/codegen/serenity.risk/experiments/dar-wilshire/DAR_prices_data.csv: Permission denied (os error 13)

Output of uname -a:

Linux c774de47c447 6.10.14-linuxkit #1 SMP Fri Nov 29 17:22:03 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Severity: very low, I can easily work around this.

[huonw]

Sorry for the trouble.

When you see this error, can you manipulate that file directly (in the same shell running Pants)? For instance cp /home/serenitydev/.cache/pants/lmdb_store/immutable/files/2d/2d522219da490246fd3842165d3ebb7362314e218a510c16daeb09bc8d5fc5a6 ./dist/testing

#19826 is potentially slightly related?

[huonw]

Also, can you work out what targets are generating the file? If so, can you share their definitions here (and pants.toml and any relevant other targets), in as much detail as you are comfortable?

[kdowney-talos]

Yes, I can copy it:

(.venv) serenitydev@c774de47c447:/workspaces/pms$ ls -la dist
total 564
drwxr-xr-x  5 serenitydev serenitydev    160 Dec 19 00:00 .
drwxr-xr-x 34 serenitydev serenitydev   1088 Dec 18 18:38 ..
-r-xr-xr-x  1 serenitydev serenitydev 574136 Dec 19 00:00 2d522219da490246fd3842165d3ebb7362314e218a510c16daeb09bc8d5fc5a6

It's hard to identify the targets because as I said, the module that's causing problems doesn't have any codegen targets, and if I specify just the one that has protobuf targets, the bug does not appear.

I cannot directly attach *.toml files, so I have provided pants.toml inline here:

[GLOBAL]
pants_version = "2.23.0"
backend_packages = [
  "pants.backend.build_files.fmt.ruff",
  "pants.backend.codegen.protobuf.lint.buf",
  "pants.backend.codegen.protobuf.python",
  "pants.backend.docker",
  "pants.backend.docker.lint.hadolint",
  "pants.backend.experimental.adhoc",
  "pants.backend.experimental.java",
  "pants.backend.experimental.openapi",
  "pants.backend.experimental.openapi.lint.openapi_format",
  "pants.backend.experimental.python",
  "pants.backend.experimental.python.lint.ruff.check",
  "pants.backend.experimental.python.lint.ruff.format",
  "pants.backend.experimental.terraform",
  "pants.backend.experimental.terraform.lint.tfsec",
  "pants.backend.experimental.tools.yamllint",
  "pants.backend.python",
  "pants.backend.python.typecheck.mypy",
  "pants.backend.shell.lint.shfmt",
  "pants.backend.tools.taplo",
]

[buf]
config = "build-support/buf.yaml"
known_versions = [
  "v1.47.2|linux_arm64 |e7188833039d4e7736de517eba6141b9306f4b60b00974392dac7ce38627321e|22700012",
  "v1.47.2|linux_x86_64|39716cfe0185df3cba21f66ec739620ffb6876c48b2da4338a8c68c290c9b116|24719887",
]
version = "v1.47.2"

[export]
py_editable_in_resolve = [
  "serenity-base",
  "serenity-base-services",
  "serenity-data-client",
  "serenity-data-pipelines",
  "serenity-middleware",
  "serenity-risk",
  "serenity-specifications",
]
resolve = [
  "serenity-deps"
]

[mypy]
config = "build-support/mypy.ini"
install_from_resolve = "mypy"

[pytest]
install_from_resolve = "pytest"
requirements = ["//3rdparty/python:pytest"]

[python]
default_resolve = "serenity-deps"
enable_resolves = true
interpreter_constraints = ['==3.12.*']

[python.resolves]

# tools
black = "3rdparty/python/black.lock"
flake8 = "3rdparty/python/flake8.lock"
isort = "3rdparty/python/isort.lock"
pytest = "3rdparty/python/pytest.lock"
mypy = "3rdparty/python/mypy.lock"
ruff = "3rdparty/python/ruff.lock"

# frameworks
dagster = "3rdparty/python/dagster.lock"

# modules
serenity-deps = "3rdparty/python/serenity-deps.lock"
serenity_data_pipelines-deps = "3rdparty/python/serenity_data_pipelines-deps.lock"

[python-protobuf]
infer_runtime_dependency = false

[ruff]
config = "build-support/ruff.toml"

[source]
root_patterns = [
  "src/protos",
  "src/python",
  "tests",
  "/serenity.analytics",
  "/serenity.base",
  "/serenity.base.services",
  "/serenity.data.client",
  "/serenity.data.pipelines",
  "/serenity.labs",
  "/serenity.middleware",
  "/serenity.specifications",
  "/serenity.risk",
  "/serenity.seal"
]

[taplo]
glob_pattern = ["**/*.toml", "!pyproject.toml", "!.taplo.toml", "!taplo.toml"]

[test]
attempts_default = 3

[yamllint]
config_file_name = "build-support/yamllint.yml"
exclude = ["**/k8s/**/*.yaml", "**/k8s/**/*.yml"]

[huonw]

Thanks for the info.

Re the target, from the error message looks like the cached file is the contents of what is attempting to be serenity.risk/experiments/dar-wilshire/DAR_prices_data.csv. Are there any targets that interact with a file with that name?

[kdowney-talos]

Hi @huonw, thanks for picking this up. Here's the full BUILD under serenity.risk:

python_requirements(name="deps", source="pyproject.toml")

resources(name="metadata", sources=["**/*.csv"])
resource(name="pyproject", source="pyproject.toml")
resource(name="version", source="VERSION.txt")

python_distribution(
    name="dist",
    dependencies=["//serenity.risk/src/python:lib", ":pyproject", ":version"],
    provides=python_artifact(name="serenity-risk"),
    generate_setup=False,
)

As you can see, the place the CSV file gets picked up is in the metadata resource, which then gets used here, in serenity.risk/src/python/BUILD's python_sources target:

resources(name="config", sources=["**/*.cfg", "**/*.json"])

python_sources(
    name="lib",
    sources=["**/*.py"],
    dependencies=[
        "//serenity.risk:deps",
        "//serenity.base/src/python:lib",
        "//serenity.data.client/src/python:lib",
        "//serenity.specifications/src/python:lib",
        "//serenity.risk:metadata",
        ":config",
    ],
)

Nothing code generation related. I could certainly refine that glob to avoid picking up files under the experiments directory without breaking out build as a workaround.

[benjyw]

Hi @kdowney-talos . I assume that you can still reproduce this?

If so, does it still happen if you remove serenity-risk from the py_editable_in_resolve option?

And, am I understanding correctly that that DAR_prices_data.csv is just a regular source file in your repo, and not the output of a code generator?

I'm looking at why this file would be exported as part of the export-codegen goal, when it seems unrelated.

If there is any way you could share a reproduction with us, that would be great.

[kdowney-talos]

Hi @benjyw, I was able to reproduce it with 2.24.0 just now. I can confirm both:

1. DAR_prices_data.csv is just a simple data extract for testing, nothing to do with code generation
2. removing serenity-risk from py_editable_in_resolve made no difference

I tried to sub-set this before in a way to help reproduce but other than narrowing it down the CSV tripping it, no joy. I understand that makes it more challenging for you.

[benjyw]

Hi @benjyw, I was able to reproduce it with 2.24.0 just now. I can confirm both:

1. DAR_prices_data.csv is just a simple data extract for testing, nothing to do with code generation
2. removing serenity-risk from py_editable_in_resolve made no difference

I tried to sub-set this before in a way to help reproduce but other than narrowing it down the CSV tripping it, no joy. I understand that makes it more challenging for you.

Any chance you can create a repo that reproduces this, and that you can give me access to? If making this public is not feasible then even a private one that you grant me temporary access to would be helpful (I can sign an NDA if necessary).

[kdowney-talos]

I think the most practical thing to do here -- though I might need some time, am about to leave on a business trip -- is for me to create a public repo or gist or some other bare-minimum reproducible use case and share with you.

[benjyw]

That would be excellent, thanks. This is bugging me...

[benjyw]

A public GH repo would be ideal, rather than a gist, so that it's easy to be sure I'm looking at a complete case. Thanks!

[kdowney-talos]

Yeah, leave this with me, maybe I can do it on the plane if there's good WiFi, no promises. 😆

[kdowney-talos]

Hi @benjyw I managed to reproduce this error (in export) in a bare-minimum repo today. I need to clear internally (ETA Monday), but then will add you as collaborator so you can see the full example. I've given a complete VSCode devcontainer as part of it, so you should be able to just open it and see what I'm seeing.