Explicitly importing `types_...` modules leads to non-obvious errors

[huonw]

Describe the bug

Importing from a package with name like types_... seems to not work, as dependency inference cannot find it automatically.

Importing from a type-stub is peculiar, but seems to be necessary when interacting with the AWS ecosystem, and particularly the async aioboto3. For instance, as far as I know, this is the best way to annotate this function:

from typing import TYPE_CHECKING
import aioboto3

if TYPE_CHECKING:
    from types_aiobotocore_s3.client import S3Client  # importing from a type stub!

async def make_client(session: aioboto3.Session) -> Iterator[S3Client]:
    async with aioboto3.client("s3") as client:
        yield client

It looks like a work around is to set modules (potentially via python_requiremnts(module_mapping=...)) for the type stub target, to make it behave more like a non-stub. There's no evidence of this workaround from the diagnostics.

Reproducer:

cd $(mktemp -d)

cat > pants.toml <<EOF
[GLOBAL]
pants_version = "2.24.2"
backend_packages = [
  "pants.backend.python",
  "pants.backend.python.typecheck.mypy",
]

[python]
interpreter_constraints = ["CPython==3.10.*"]
EOF

# start pantsd
pants version

cat > x.py <<EOF
from types_aiobotocore_s3 import S3Client
EOF

echo "***** Just dependency inference"

cat > BUILD <<EOF
python_requirement(name="types-aiobotocore-s3", requirements=["types-aiobotocore-s3"])

python_source(name="x", source="./x.py")
EOF


echo "BUG: no dependency on types_aiobotocore_s3"
pants dependencies x.py
echo "BUG: fails with 'cannot find ... types_aiobotocore_s3'"
pants check x.py

echo "***** Workaround: set modules"

cat > BUILD <<EOF
python_requirement(name="types-aiobotocore-s3", requirements=["types-aiobotocore-s3"], modules=["types_aiobotocore_s3"])

python_source(name="x", source="./x.py")
EOF

echo "OKAY: dependency listed types_aiobotocore_s3"
pants dependencies x.py

echo "OKAY: checking passes"
pants check x.py

Output:

***** Just dependency inference
BUG: no dependency on types_aiobotocore_s3
18:01:22.32 [WARN] Pants cannot infer owners for the following imports in the target //:x:

  * types_aiobotocore_s3.S3Client (line: 1)

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see https://www.pantsbuild.org/2.24/docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies for common problems.


BUG: fails with 'cannot find ... types_aiobotocore_s3'
18:01:23.54 [INFO] Starting: Building requirements_venv.pex
18:01:24.61 [INFO] Completed: Building requirements_venv.pex
18:01:25.73 [ERROR] Completed: Typecheck using MyPy - mypy - mypy failed (exit code 1).
x.py:1: error: Cannot find implementation or library stub for module named "types_aiobotocore_s3"  [import-not-found]
x.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)



✕ mypy failed.

***** Workaround: set modules
OKAY: dependency listed types_aiobotocore_s3
//:types-aiobotocore-s3
OKAY: checking passes
18:07:53.33 [INFO] Completed: Typecheck using MyPy - mypy - mypy succeeded.
Success: no issues found in 1 source file



✓ mypy succeeded.

Pants version
2.24.2
OS
macOS

Additional info
N/A