Support package projects with src-layout

[lemonyte]

I have an ASGI app project with the following src-layout file structure:

project_name
├── config.toml
├── pyproject.toml
├── wrangler.toml
└── src
    ├── package_name
    │   ├── __init__.py
    │   ├── server.py
    │   ├── ...
    │   ├── subpackage_name
    │   │   └── __init__.py
    │   ├── data
    │   │   └── data.json
    │   ├── static
    │   │   └── styles.css
    │   └── templates
    │       └── index.html
    ├── js-stubs
    │   └── __init__.pyi
    └── worker.py

During runtime, it needs to read files in subdirectories of the package (e.g. data, templates), and files in the current working directory (e.g. config.toml).

If I put worker.py inside the package and import the ASGI app, it fails because package_name is not installed as a package, so Python doesn't know how to import neighbouring files.

If I put worker.py at the top level, I need to import from src, which defeats the point of the src-layout.

If I put worker.py under src, the imports work fine, but none of the top-level files are accessible.

Is there a way to make pywrangler build and install the project along with its dependencies (e.g. when a build system is defined)?
What would be the best place to put worker.py in this case?

[hoodmane]

Just to clarify, by top-level files you mean pyproject.toml?

[lemonyte]

Yes, and others at the same level. Although access to top-level files isn't as important as installing the project, so maybe it's best to leave it for another issue.

Here's another example of code that fails when the project is not installed:

# src/package_name/__init__.py
from importlib.metadata import version
__version__ = version("package_name")
print(f"{__version__=}")

Raises:

importlib.metadata.PackageNotFoundError: No package metadata was found for package_name

[hoodmane]

I don't think we're planning to build the project like that. Probably what you ought to do is build the wheel yourself and then add the wheel path as a dependency in pyproject.toml for the application. Note also that if we did that the resulting wheel would still not contain the top level files like pyproject.toml...

[hoodmane]

On the other hand, I agree it is a confusing limitation that you can't include the top level files in the worker.

[lemonyte]

build the wheel yourself and then add the wheel path as a dependency in pyproject.toml for the application

This creates a chicken-and-egg dependency problem, as the package would then require a built wheel of itself in order to build a new wheel. uv simply fails to resolve the dependencies in this case.

It might be possible to have worker.py live in a completely separate package that depends on the application package, but that would require some major reshuffling of the repo and introduces an unnecessary decoupling and build step between the simple worker entrypoint and the actual ASGI app.

the resulting wheel would still not contain the top level files like pyproject.toml

It's not expected to. Access to top-level files is a nice-to-have at runtime but at present there's already an alternative for my use-case in the form of worker env vars, so I'm retracting this suggestion.

Perhaps pywrangler could use uv to install the project and its dependencies using pyproject.toml directly, instead of parsing it and generating a temporary requirements.txt?