Publish EvaP to PyPI

[niklasmohrin]

Closes #2303

Since we don't want to use nix in production (for example because security updates might take a long time to propagate through nixpkgs / through our github release cycle), pip install . or pip install evap seems to be the most canonical way. This allows us to install security patches via pip install --upgrade evap.

If we really need to override a dependency version beyond the allowed range in pyproject.toml, we can install EvaP from sources outside of PyPI. The script to build the release artifacts is available as nix run .#build-dist. It is possible to run this on a development machine and copy the .whl file over to the server.

The release wheel includes compiled TypeScript, SCSS and translation files, so there is no need to install NodeJS or gettext on the production server.

The new release workflow is:

1. Add translations and commit them.
2. Bump version in pyproject.toml, run nix develop .#impure --command uv lock, commit it, tag it, push it.
3. After the CI is done on main, dispatch a release workflow run to publish the wheel to PyPI.
4. Create Github Release.

The versioning scheme we agreed on is year-month-patch, for example "2024.11.0". I suppose we should take care that we don't necessarily push breaking changes like updating the Python version in a new patch. Following the Python Packaging User Guide, we don't include leading zeros in release versions like "2025.4.0".

Since upgrading to the next stable revision of EvaP is handled by pip, there is no need for the release branch anymore.

See

• https://packaging.python.org/en/latest/tutorials/packaging-projects/
• https://docs.pypi.org/trusted-publishers/using-a-publisher/
• https://packaging.python.org/en/latest/specifications/version-specifiers/

[niklasmohrin]

See https://test.pypi.org/project/evap/ and https://pypi.org/project/evap/

[niklasmohrin]

@richardebeling Thoughts? Furthermore, if you create a PyPI account, I can add you as a co-owner of the package. (I also requested the creation of an e-valuation organization on PyPI, not sure if it will be nicer / needed, we'll see once it's processed)

[niklasmohrin]

One thing we discussed in person was that we now don't really need a full checkout of the EvaP repository on production. In fact, this is a chance to move most of the (apache-)deployment files out of the main repository and into a separate repository in the e-valuation organization. @janno42 agreed that this would be a nice (while not really needed) change. The only drawback seems to be that testing the backup process in the EvaP repository will be more complicated. Currently, I believe that we would want to pin a commit of the deployment repository, for example through flake.lock, and run the tests here. I believe that we could add the backup process test logic through nix in the deployment repository and then import this test in the EvaP repository to use the updated source of EvaP. In particular, this would allow us to share the test definition and "easily" run the same tests in both repos. I am not too settled on this approach though.

Thoughts @richardebeling ?

[richardebeling]

@richardebeling Thoughts? Furthermore, if you create a PyPI account, I can add you as a co-owner of the package. (I also requested the creation of an e-valuation organization on PyPI, not sure if it will be nicer / needed, we'll see once it's processed)

While certainly unexpected, I think it's a funny and efficient solution, and I don't have any reasons against doing this.

One thing we discussed in person was that we now don't really need a full checkout of the EvaP repository on production. In fact, this is a chance to move most of the (apache-)deployment files out of the main repository and into a separate repository in the e-valuation organization. janno42 agreed that this would be a nice (while not really needed) change. The only drawback seems to be that testing the backup process in the EvaP repository will be more complicated.

Why? "Backup" essentially is "Dump database via dumpdata", isn't it? This would be separate from apache and all the other deployment aspects. I can see that the "Upgrade" part of the script wouldn't be testable -- we can compromise on that, I think. It's important that backup and especially restore is tested.

---

Re (separate repo, version pinning): Adds complexity, but if there is a need for this or it helps in any way, I guess we won't die 🤷‍♂️

A proper release procedure with some github magic action could also be used to deliver compiled TS and SCSS files automatically, something that I dislike a bit about our current approach (necessity to have npm somewhere). (We should keep in mind though: Automation is always nice up to the point where it stops working)

[niklasmohrin]

I updated the PR description to fit the current state of things. I tested the currently published release with the following suggested setup in a podman container (podman run --rm -p 8888:80 -it ubuntu):

apt update && apt upgrade -y && apt install -y sudo redis postgresql apache2 python3 python3-venv libapache2-mod-wsgi-py3

service redis-server start
service postgresql start
service apache2 start

sudo -u postgres psql
create user evap password 'evap' createdb;
create database evap owner evap;
exit

useradd -m evap
sudo -i -u evap
mkdir evap && cd evap
python3 -m venv venv
. venv/bin/activate
pip install evap[psycopg-binary]
# set up productionsettings.py and wsgi.py with template
python -m evap collectstatic
python -m evap migrate
# collectmessages not needed, already included in release

exit # venv
exit # sudo -i
chown -R evap:www-data /home/evap

cd /etc/apache2
# set up sites-available/100-evap.conf with template, replace /evap with /home/evap/evap
a2enmod headers
a2enmod wsgi
a2dissite 000-default
a2ensite 100-evap
service apache2 restart

Afterwards, evap should be available at the given port. To log in, one should also run python -m evap reload_testdata after collectstatic.

[niklasmohrin]

We made https://github.com/e-valuation/evap-deployment as a place to move the deployment files, because production will not have a checkout of the EvaP repository anymore

[richardebeling]

Code changes look fine to me, my biggest concern is complicated release process adding dependencies/complexity that add maintenance and security burden.

One future bug I expect to happen sooner or later is forgetting to add files to the release.

[niklasmohrin]

One future bug I expect to happen sooner or later is forgetting to add files to the release.

Yeah, that could be a problem. Maybe once we have #1854, we can let the browser fail on unhandled non-200 responses for any secondary requests like javascript files etc and run it on an instance from the wheel.

[niklasmohrin]

I now decided that I will try to split the changes here into multiple PRs, with the one here probably being the final one in the end

[niklasmohrin]

@janno42 @richardebeling Please review my updated PR description to ensure we are on the same page.

[Kakadus]

3. After the CI is done on main, dispatch a release workflow run to publish the wheel to PyPI.
4. Create Github Release.

we could merge these steps using the release trigger: https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#release

[niklasmohrin]

Yeah, I had something like this earlier, but I thought that it would be easier for now to do these steps separately. It's important that the PyPI release CI runs after the main CI is completely finished, otherwise the artifact is not available (and I wanted to reuse the wheel from the main CI run). There isn't really a good way to have a dependency between different workflow files, so I figured that a manual dispatch would be best, in particular because it can be re-run if it was triggered to early.

Notably, it is recommended to have the release in a separate workflow, because then the normal CI doesn't have access to the release environment, which is used to authenticate to PyPI.

We could automatically create a Github release as part of the PyPI release workflow, but again, I wanted to be a bit cautious with this automation for now.

[richardebeling]

Updated description sounds good to me 👍

[janno42]