Dependency management

[matheusfelipeog]

Hi ;)

This project uses requirements.txt as a dependency manager file. Wouldn't it be useful to use Pipenv + Pipfile or Poetry + Pyproject.toml as a substitute?

With any of the indicated you can manage all dependencies in 1 file and with features like locking dependencies with hash and the like...

---

More info:

Pipenv: https://pipenv.pypa.io/en/latest/
Pipfile: https://github.com/pypa/pipfile

Poetry: https://python-poetry.org/
Pyproject.toml: https://python-poetry.org/docs/pyproject/

[dieggoluis]

Pipenv was almost a dead project for a long time (pypa/packaging.python.org#701) and I've seen some people having quite a few issues with it.

I've been using poetry for about six months and it looks promising. The only caveat from my experience is that it takes more time to generate the lock file and you will need to generate the equivalent requirements file if you do not want to install poetry in your docker images or servers. If you do want to use poetry with docker (https://stackoverflow.com/questions/53835198/integrating-python-poetry-with-docker).

Pip-tools can also generate hashed dependencies, but I believe they chose not to use it. Poetry looks promising and would definitely replace some scripts of this project, but if maintainers are happy the way it is, I don't see any reason to change. I'm curious though if there is any other reason to stick with pip-tools. 🤔 @caique-lima

[matheusfelipeog]

Pipenv was almost a dead project for a long time (pypa/packaging.python.org#701) and I've seen some people having quite a few issues with it.

This is really a point, but apparently the project is back on track, according to the release history. This don't deny having extra attention with this tool.

you will need to generate the equivalent requirements file if you do not want to install poetry in your docker images or servers

It looks like Pypa will create a new command line option for pip to support Pipfile format.

I didn't find much similar information for pyproject.toml format. The closest discussion was this: pypa/pip#8049

But, because both follow the toml standard, I believe support for both files will be implemented in the pip in the near future.

[gcbeltramini]

There are indeed many ways to manage dependencies in Python. We tested Pipfile and poetry. Both didn't seem to add any special advantage and their usage was not widespread. We wanted to:

1. Keep it simple: adding a new file format like Pipfile+Pipfile.lock or pyproject.toml would add complexity to the file format we were used to (requirements.txt)
2. Use a framework that is stable and implemented by the libraries we use: at the time that we explored the alternatives, requirements.txt or adding the dependencies directly to setup.py were the most common approaches
3. Pin the versions of all direct and indirect dependencies: read below
4. Have an easy way to generate the pinned dependencies: read below

In other internal libraries, we use pip-tools. In summary:

1. The direct dependencies are in requirements.in
2. Install pip-tools in a Python environment (we use venv, virtualenv or conda)
3. Run pip-compile to create the pinned list of all direct and indirect dependencies in requirements.txt (same goal as Pipfile.lock)
4. To run tests, prepare the environment with pip-sync

This choice involves a lot of personal preferences, but solved our problems with dependencies (sometimes there are conflicts or unexpected behaviors, so we simply adjust requirements.in, and run the steps above).

[gcbeltramini]

And thanks for raising this discussion. Things change, new formats are adopted, and we want to keep up with the community. So we are always open to change.

pip-tools is not being used by fklearn, and I think it could.

[erickisos]

Sorry, I know this is an old thread, but I want to be sure that I understand what could happen here.

What do you think about the advantage of having a tool which allow us to manage virtualenvs and dependencies within the same config file, also, using the pyproject.toml file we can keep track of some tools configuration in order to have an unopinionated codestyle, everything, centralized in the same point.

In the past I used to have some problems when sharing a project if some of my pals didn't really know what a venv was or how can that help'em to isolate everything, idk if this makes sense for this project.

And, if it makes sense, maybe we can start trying(?), would be valuable?