Resources not available while running a python_source

[thelittlebug]

link to the example repo: https://github.com/thelittlebug/pants_resources_possible_bug

Possible Bug? Feature? Undocumented Behavior?

Preamble

I’m facing an issue with generated resources and Python.

I already discussed this on Slack: Slack thread here.
However, as I’ve learned: always create a minimal working project to showcase the issue. So, here we are.

What’s Working

1. The generated file is included in the loose layout of the PEX
   • Command: pants package ::
   • You can inspect the output at dist/src.python.example/loose.pex/example, the generated.txt is right there.
2. It works in the loose layout of the PEX
   • Command: pants run src/python/example:loose
   • Output: found_generated_txt=True
3. It works in the zipapp layout of the PEX
   • Command: pants run src/python/example:zipapp
   • Output: found_generated_txt=True

What’s Not Working

1. The generated file is missing when running the python_sources target directly.
   This would be a huge speed boost for Django projects since it avoids waiting for a regenerated PEX file.
   • Command: pants run src/python/example/main.py
   • Output: found_generated_txt=False

Questions

1. Is this behavior intended?
2. Is there a workaround?

Thanks

Thanks to everyone on the team for your hard work on such a massive tool. It gets better with every release. Keep up the great work! 🎉

[lilatomic]

This turns out to be mostly due to silly path things. relocated_files moves the files into the ":/example" directory. package bundles the "pex" starting from the [source].root_patterns, so ":/src/python/example"->":/example". this means that the relocated_files end up in the same directory as the python_sources in the pex_binary, and the importlib machinery can import it. run, on the other hand, tries to emulate running in the local directory. This isn't actually true, it still creates a sandbox. But it does preserve the path to the python sources, which are still at ":/src/example/python". The relocated files are still at ":/example", so the importlib load fails.

You can inspect the sandboxes by setting the [GLOBAL].keep_sandboxes option. if you do, you'll see almost nothing when running the pex_binary, since it's all been bundled into a pex and unzipped elsewhere on disk. If you follow where it's been unzipped, you'll see your resources in the same dir as the python sources, like you want. Contrasting, for the run invocation, the sources are at ":/example/generated.txt" but the python is at ":/src/python/example/main.py".