Document Python dependencies management (#165)

* Document Python dependencies management

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: cristianonicolai

.gitignore

@@ -130,3 +130,5 @@ dmypy.json
 .DS_Store
 .envrc
 .vault
+
+_readthedocs/

docs/guides/python/dependencies.md

@@ -0,0 +1,97 @@
+# Managing Python dependencies
+
+1. All pip dependency files stored in `.config`
+2. All dependency file names must match this `requirements.(txt|in)`. Needed for dependabot compatibility.
+
+### Recommended filenames:
+
+- `.config/requirements.in` - runtime deps
+- `.config/requirements-test.in` - test requirements
+- `.config/requirements-docs.in` - docs requirements
+- `.config/requirements-lock.txt` - locked (pinned) runtime requirements for projects having `lock` extra.
+- `.config/constraints.txt` - unified testing constraint file to use as `PIP_CONSTRAINTS`. Is named like this for Dependabot compatibility. It also pins all extras.
+
+### Upgrading dependencies
+
+To upgrade dependencies, it's recommended to use `pip-tools` as part of the `pre-commit` hook and invoke manually via a tox profile named `deps`.
+
+Example `.pre-commit-config.yaml`
+
+```
+- repo: https://github.com/jazzband/pip-tools
+    rev: 7.3.0
+    hooks:
+      - id: pip-compile
+        name: deps
+        alias: deps
+        stages: [manual]
+        entry: pip-compile .config/requirements.in --upgrade --all-extras --no-annotate --strip-extras --output-file=.config/constraints.txt pyproject.toml
+        files: ^.config\/.*requirements.*$
+        language_version: "3.10" # minimal we support officially
+```
+
+Example `tox.ini`
+
+```
+[testenv:deps]
+description = Bump all dependencies
+base_python = python3.10
+skip_install = true
+deps =
+    {[testenv:lint]deps}
+extras =
+set_env =
+    PIP_CONSTRAINT = /dev/null
+commands_pre =
+commands =
+    -pre-commit run --all-files --show-diff-on-failure --hook-stage manual deps
+    -pre-commit autoupdate
+    git diff --exit-code
+env_dir = {toxworkdir}/lint
+```
+
+To upgrade dependencies, execute `tox -e deps` in the local project.
+
+### Dependabot Github configuration
+
+To minimise the amount of PRs Dependabot would create, it is recommended to group all dependencies updates together. This can be accomplish with the following config file:
+
+Example `.github/dependabot.yml`
+
+```
+---
+version: 2
+updates:
+  - package-ecosystem: pip
+    directory: /.config/
+    schedule:
+      day: sunday
+      interval: weekly
+    labels:
+      - dependabot-deps-updates
+      - skip-changelog
+    groups:
+      dependencies:
+        patterns:
+          - "*"
+```
+
+### Dependabot quirks
+
+As Dependabot has very limited configurability, filenames matter and we can only make it work well if they match.
+
+If you have a pair of `requirements.in` and `requirements.txt` in a folder, dependabot always rewrite the `.txt` file as being the lock for the `.in` file. This means that if we use the txt as a constraint and we also have some extras, the lock file will not be correct. Dependabot will attempt to mess the file.
+
+Dependabot parses requirements files and tries executing the same command specified in the header as a comment. Please take a look at an example header below.
+
+```
+#
+# This file is autogenerated by pip-compile with Python 3.11
+# by the following command:
+#
+#    pip-compile --no-annotate --output-file=.config/requirements.txt --strip-extras .config/requirements.in pyproject.toml
+#
+```
+
+The actual `pip-compile` command executed by Dependbadot won't be the same as it parses arguments and only uses known ones, giving possible different results.
+Dependabot does not support `--extra` when running pip-compile based on requirements files. See https://github.com/dependabot/dependabot-core/issues/6406

mkdocs.yml

@@ -25,6 +25,7 @@ nav:
           - Permissions: guides/ansible/permissions.md
       - Python:
           - Python Packaging: guides/python/packaging.md
+          - Python Dependencies: guides/python/dependencies.md
           - PyTest: guides/python/pytest.md
           - Tox: guides/python/tox.md
           - Release: guides/python/release.md