Common package managers buildpack

[sgaist]

Hi,

The python packaging world is pretty diverse and has multiple tools at disposal as shown by the various buildpacks that are out there to provide support for them.

Currently there is:

• pip
• pipenv
• conda
• poetry

In the mean time, new tools have emerged such as mamba, pixi, uv, just to name the most recent.

At the same time, the pyproject.toml file has gain traction and is not as attached to poetry as it was at the start.

Adding support for a new package manager usually consist of creating two buildpacks:

• One that installs the tool in question
• One that does the detection and then uses the tool to do the setup

With pyproject.toml being more in use but supporting different backends, the current detection does not work since it only checks for the presence of the file.

With all of that in mind, I was wondering whether it would not be beneficial to create a new buildpack that would be responsible for all the detection and use of selected package manager rather than having a pair for each tool.

The goal would be to unify the various detection under one roof so that we could reduce the number of buildpacks to maintain for the same functionality.

This would mean a more complex buildpack but, at the same time, alleviate adding support for a new tool.

What do you think ?

[dmikusa]

100% +1 on consolidating these. It is less overhead to have a more complicated buildpack, than 10 less complicated buildpacks (IMHO).

@jericop - What do you think about this?

[jericop]

I also like this idea. Perhaps we could make the new common buildpack require a pyproject.toml with a supported backend (as part of the detection) so that we are focused on supporting the future direction of python projects? I imagine that the separate buildpacks will remain operational for the foreseeable future, but we could focus future buildpack development on the common buildpack and steer users towards modern projects samples so that we eventually can stop supporting the individual buildpacks.

I'm glad to hear that pyproject.toml has gained some traction. There are so many ways to build python apps that it's all too easy for a user to find an outdated example project. Thanks for bringing this up.

[sgaist]

I would put it as a documented recommendation rather than a requirement. Maybe have an environment variable that would trigger a detection failure if one wants to ensure that pyproject.toml is present for their use cases.

I think this would allow for a smoother adoption of this new buildpack.

I don't have a poc available yet but before diving into it, what would be the procedure to make it happen ? I can see two options:

• Start in my own repo and later on transfer it to the paketo-buildpacks organisation
• Start a new repo directly in the paketo-buildpacks organisation.

[dmikusa]

First thing, let me say that this is fantastic! Thanks for putting this together. If you can share a link. Then folks can take a look and try it out.

Second, we have some sample apps. If you haven't done so already, a good compatibility check is to run it against those and make sure things work. In general, if you can expand on what you've tested with also, that'd be helpful. Testing is important, especially when the goal is to replace existing buildpacks. We want to ensure compatibility as best we can. Ideally, user's won't see a difference.

Finally, to make a change like this official. We'd need to do an RFC. This is the project's way of formalizing and voting on large changes, like adding or replacing buildpacks. Don't worry, it's a pretty easy process. There's a markdown doc you can submit as a PR to log the proposal. Then we use Github comments to review & discuss. This should be a Python team RFC, so you can add the proposal here. The sections in the doc will walk you through what you're proposing and the steps to make that happen. You can also check out other RFCs as examples or to copy & paste. Then the @paketo-buildpacks/python-maintainers can review, provide feedback and ultimately vote on the proposal.

If you have questions on the RFC process, feel free to reach out. Happy to help with that.

[jericop]

Great news! I was going to create an rfc to consolidate package managers into a single buildpack, but have not had a chance yet. I will create the rfc tonight. Do you have this in your own repo that we could take a look at?

[jericop]

Just noticed Dan’s reply. Feel free to create the rfc if you feel inclined. Just let me know so we don’t duplicate effort.

[sgaist]

@dmikusa Thanks for the pointers !
@jericop I'll write the RFC since I started this.

I have pushed my proposal at https://github.com/idiap/python-packagers.
I have tested it locally against projects from each package manager either from the examples or from the test data.

Please note that the imports have been written with the goal to have the buildpack moved over to the paketo-buildpacks organisation.

[sgaist]

RFC PR available at paketo-buildpacks/rfcs#321