How To Run a Python Source That Uses Files In Repo Subdirectory

[justicefreed]

I'm likely missing something, but more or less, no matter what I do, pants seems to be running my scripts from repo root, without relocated_files targets present.

Repo Structure (Monorepo):

repo-name
      pants.toml
      -- projects
                -- myservice
                          BUILD
                          config.json
                          -- src
                                    main.py   # imports mylibrary.foo, usually run from myservice dir
                -- myotherservice
                          BUILD
                          otherconfig.json
                          -- src
                                    othermain.py   # also imports mylibrary.foo
      -- library
                -- mylibrary
                          BUILD
                          -- mylibrary
                                   foo.py

The key idea is that config.json contains necessary data for main.py's service, but is NOT bundled/deployed with the service and will be different for every deployment instance. This data is loaded via python, using open("config.json"). In the deployment context, each project/service lives in it's own somewhat isolated directory, and does not have any sense of the repo structure other than below the myservice directory. The goal is to deploy/distribute each service using PEX files. This all generally seems to work.

However, I want to be able to use pants to run main.py during development/testing, without breaking the script's access to config.json. This is what I can't seem to make work. I've tried:

• Using files and relocated_files to specify that config.json should be moved to root at runtime, along with run_goal_use_sandbox=True. The file gets successfully added to the sandbox, but running os.listdir() and os.getcwd() within my script show that the script's working directory is project root, NOT the sandbox as expected.
• Same thing except using pex_binary instead of python_source. Same result.
• Using adhoc_tool -> run_shell_command as a hack to try to execute the script using myservice as the working directory, but really doesn't seem to be the right approach (though it's nice to be able pass in the default required args that way).

At this point, my main question is - why doesn't the python script's working directory use the sandbox directory when that setting is applied? It seems this would resolve my issue. Secondarily, is there any way to run a target with the working directory equal to the BUILD file location for that target? This would be so, so much simpler for my use case I think, but I could not find any way to change the working directory in a way that doesn't break normal, deployed operation.

I feel it's likely I'm missing something silly here, so any guidance on how to allow a simple relative file import to work as expected from a script & file with paths relative to their common subdirectory instead of root would be greatly appreciated. I've spent probably 6 hours with the docs trying to get this one thing working 😅 so I've been as thorough as I could.

Note: this repo is all on python 3.12, and would not install on the latest stable release, hence the alpha.

pants.toml

[GLOBAL]
# pants_version = "2.18.0"
pants_version = "2.19.0a0"
backend_packages = [
    "pants.backend.shell",
    "pants.backend.python",
    # "pants.backend.experimental.python.lint.ruff",
    "pants.backend.python.typecheck.mypy",
    "pants.backend.experimental.adhoc"
]
pants_ignore=["!__pycache__/","!**/__pycache__/","!config.json"]

[source]
root_patterns = [
    '/',
    '/components/*',
    '/library/*',
    '/library-stubs/*',
    '/tools'
]

[python]
interpreter_constraints = ['==3.12']
enable_resolves = true
default_run_goal_use_sandbox = true

myservice -> BUILD

python_requirements(
    name="reqs",
    source="requirements.txt",
)

files(
    name="files_needed_at_cwd",
    sources=["dynamic_config_overrides.json"]
)

relocated_files(
    name="cwd_files",
    src="components/MTConnectClient",
    dest="",
    files_targets=[":files_needed_at_cwd"]
)

pex_binary(
    name="pex_binary",
    entry_point="main",
    dependencies=[":cwd_files"]
)

python_source(
    name="main",
    source="src/main.py",
    dependencies=[":reqs",":cwd_files"],
    run_goal_use_sandbox=True
)

[huonw]

I don't have time to really dig in yet, but on first glance, I wonder if this is a resources vs. files problem (it's confusing and subtle). Things will only be included in a pex_binary if they're part of a package, which is what resources gives. https://www.pantsbuild.org/docs/assets has some more info.

Does switching from files to resources and/or wrapping the result of relocated_files in https://www.pantsbuild.org/docs/reference-experimental_wrap_as_resources help?

[justicefreed]

Thanks for the initial response! I tried everything you suggested, (and various permutations) and saw no observable difference whatsoever. My intention has been to use files specifically because I need it to NOT be bundled/packaged with the code for deployment but instead use the file that will available independently within each deployment.

I could just leave it alone and it would work fine in deployment, but I cannot currently use pants to correctly run the code for local development/testing because that required file is in a subdirectory relative to the repository root and thus isn't accessible with the normal relative path (and it has to be relative).

The file target and relocated_files target both seem to do exactly what they are supposed to - the file shows up in the sandbox at the expected, relocated, location at sandbox root. However, what seems to NOT work is that even with run_goal_use_sandbox=True for my python_source target, the python script executes with repo root as it's working directory, instead of the sandbox root directory.

I'd happily avoid running the goal in the sandbox if I could instead run the goal in the subdirectory for that specific project, but I could not find a way to get pants to change the effective working directory when running a goal. Any recommendations would certainly be appreciated!

[huonw]

Ah, well. Thanks for trying things.

I'd happily avoid running the goal in the sandbox if I could instead run the goal in the subdirectory for that specific project, but I could not find a way to get pants to change the effective working directory when running a goal. Any recommendations would certainly be appreciated!

I don't know of one... except for having the code that is being run itself do this.

For instance, could you have a "local" version of the target itself that varies slightly from the production code? For instance, does a few set-up things like switching directories and/or loading the config before calling into the real main(). (For my work codebase, this is effectively what we do: have a "wrapper" entrypoint for local dev for things like API servers, that does the special set-up required when running locally.)

[justicefreed]

To more directly reference the docs for clarity, the below from the docs is exactly what I want to do, EXCEPT that I need the path to NOT be the "full path", as the deployment itself will only contain the equivalent of project/* in the example below and will live in a different parent directory. Hence why I tried to use relocated_files + run_goal_use_sandbox=True to compensate for this. Docs ref, emphasis mine.

files
A file target is for loose files that are copied into the chroot where Pants runs your code. You can then load these files through direct mechanisms like Python's open() or Java's FileInputStream. The files are not associated with a code package, and must be extracted out of a deployed archive file before they can be loaded.
To reduce boilerplate, the files target generates a file target per file in the sources field.
For example, to load loose files in Python:
def test_open_file():
with open("src/python/project/config.json") as f:
content = f.read().decode()
assert content == '{"k1": "v", "k2": "v"}'
Note that we open the file with its full path, including the src/python prefix.

It's possible that the issue is that I'm completely misunderstanding how file targets are supposed to be used. In this part of the docs on the same page, it specifies the "use with" only with archive targets and tests. Is that supposed to be an example or a comprehensive list of possibilities?

If file targets are truly only used for archive target purposes, then perhaps what I am trying to do is not possible (though that feels unlikely to me). I assumed it would be a relatively common thing for someone to want to do in a monorepo, but I must be missing something.

[justicefreed]

Sorry for the parallel reply.

For instance, could you have a "local" version of the target itself that varies slightly from the production code? For instance, does a few set-up things like switching directories and/or loading the config before calling into the real main(). (For my work codebase, this is effectively what we do: have a "wrapper" entrypoint for local dev for things like API servers, that does the special set-up required when running locally.)

Yeah, I could make a pants-specific python script that just calls the other one I suppose. Maybe it's the simplest solution and therefore the best here. I'll give that a go!

[justicefreed]

Wrapping my service inside another python script that also changes the working directory seems to satisfactorily do everything I need to here. Thanks @huonw for the assistance. For others, what I now have is the below files:

repo_name/projects/myservice/dev-main-wrapper.py

"""Simple wrapper function to run src/main.py in the right context"""
import os
from src.main import main

os.chdir("./components/MTConnectClient")

main()

repo_name/projects/myservice/BUILD:

python_requirements(
    name="reqs",
    source="requirements.txt",
)

pex_binary(
    name="pex_binary",
    entry_point="src/main.py",
    dependencies=[":reqs"]
)

python_source(
    name="runner",
    source="dev-main-wrapper.py",
    dependencies=[":reqs"]
)

This works well it seems, and packaging via PEX loads the file as expected too, and all the dependencies get loaded correctly.

[justicefreed]

Only final thing I'll note is it seems the run_goal_use_sandbox=True setting for python_source targets is a little misleading, since it DOES copy the code and run the sandbox location of the code, but it does not actually run it USING the sandbox working directory. As far as I can tell, this does nothing different at all unless you are loading resource targets and doing so using something like importlib.resources to load.

Not sure if this is intended behavior or not - personally I would prefer if run_goal_use_sandbox=True would cause pants to change the working directory for the python_source target's script to be run from to be the sandbox directory. Workaround works for my purposes though.