Running python tests in a multi project workspace results in an import error

[amircodota]

Describe the bug
I have a multi-project repo with inter-dependencies between python packages.
I'm running into issues running tests where importing dependencies fails when ran from pants test.
When I run pants run dependencies are resolved correctly nd everything works

Pants version
2.23.1

OS
Linux

Additional info

I created a small repro project, thac can be found here
In the README you'll find more details on the project structure and how to reproduce the issue I'm seeing.

I suspect it has to do with pythonpath not being correctly set for tests, but I could be completely wrong.

[huonw]

Hello! Sorry for the trouble.

You're thinking along approximately correct lines about PYTHONPATH, although not in the specifics. The key here is that Pants works out the right way to arrange modules based on each target's relationship to a source roots.

The key problem is that the python_sources targets are outside the desired src source root. Thus, when Pants looks for the closest parent source root for a target, it finds /. For instance, for https://github.com/amircodota/pants-pythonpath-issue/blob/54af467adc761ffc21a7b9d874546bfa2667b2a0/services/service1/BUILD, the python_sources there is connected to the / source root, so Pants thinks the code will need to be imported as services.service1.src...., and thus dependency inference falls over.

Defining the python_sources targets within BUILD files the corresponding src/ directory works better, for me. Here's the diff, note how python_sources targets moved to .../src/BUILD:

modified   services/service1/BUILD
@@ -1,21 +1,13 @@
-python_sources(
-    name="service1", 
-    sources=["src/**/*.py"], 
-    dependencies=[
-        "//shared/library1:library1"
-    ]
-)
-
 python_tests(
     name="tests",
     sources=["tests/**/*.py"],
-    dependencies=[":service1"]
+    dependencies=["./src:service1"]
 )
 
 pex_binary(
     name="binary",
     entry_point="service1.main",
     dependencies=[
-        ":service1",
+        "./src:service1",
     ],
 )
\ No newline at end of file
new file   services/service1/src/BUILD
@@ -0,0 +1,5 @@
+python_sources(
+    name="service1",
+    sources=["**/*.py"],
+    dependencies=["//shared/library1/src:library1"],
+)
modified   shared/library1/BUILD
@@ -1,4 +0,0 @@
-python_sources(
-    name="library1", 
-    sources=["src/**/*.py"]
-)
\ No newline at end of file
new file   shared/library1/src/BUILD
@@ -0,0 +1 @@
+python_sources(name="library1", sources=["**/*.py"])

Going one step further, Pants' tailor and dependency inference seems to be another route that works. Instead of maintaining recursive targets in the right place and specifying dependencies manually, Pants can manage the python_sources and python_tests targets automatically.

I ran:

# clear all existing BUILD files
find . -name BUILD -delete

# let pants create them
pants tailor ::

# restore the pex target
echo 'pex_binary(name="binary", entry_point="service1.main")' > services/service1/BUILD

# tests now work:
pants test ::

# still runs
pants run services/service1:binary

(Don't just do a blanket delete for all BUILD files, a more targeted approach would be deleting just the python_sources and python_tests targets.)

Does that help?