Dependency inference for `resources` loaded via `importlib.resources.files`

[grihabor]

The problem

I'm using importlib.resources.files to load resources from python. I'd like to have dependency inference support for resource targets I load.

The solution

It would be nice to have the support implemented in rust.

Specifically, I want the anchor argument to work with an import:

import my_package
from importlib.resources import files 

files(my_package).joinpath('path/to/my_resource.txt')

and __package__:

from importlib.resources import files 

files(__package__).joinpath('path/to/my_resource.txt')

Alternatives

The support could be added in python, but it will be slow and it would duplicate the AST parsing and analysis.

[huonw]

Interesting idea!

This seems like it has conceptual overlap with the existing [python-infer].string_imports option (https://www.pantsbuild.org/2.21/reference/subsystems/python-infer#string_imports), although maybe understanding all the importlib.resources magic to join packages means it'd be significantly more complicated.

[grihabor]

Yeah, I checked string_imports and assets options, but they don't work in this case because the package is not included in the string

[lilatomic]

I'm a bit surprised that __package__ doesn't work. For assets we have inference tested for pkgutil.get_data(__file__, "foo/bar.txt"). We compute the expected path by taking the dir of the current python file and adding the string import code ref. I'd think that would be the same for importlib.resources using __package__, since we don't actually look for that. It's a bit loose.

The other 2 cases seem much harder to me:

1. an import without from (import a.b.c) : isn't actually simple. We need to find the path on disk for the python target providing that import, and then find the relative file. We need to identify the path of the python target because it might exist in a different source root.
2. an import with from : we would have to dereference the value in files to find the import, and then do the same as 2. This requires us to actually understand the python source file and keep track of references and imports and such. (we do some of that in the @rule decorator's impl, but it's not pretty. We end up actually running python, effectively.)

But the __package__ one should be easy enough to support.

[lilatomic]

It looks like __package__ works for me in this test, although it's pretty basic. Can you outline a scenario where it fails?