State policy for supporting end-of-life (C)Python versions

[SnoopJ]

Inspired by #2683 and discussion on IRC around it.

More specifically, @half-duplex asked about when Python 3.8 will be retired, which made me realize there is no stated policy on sopel support for CPython versions that have entered end-of-life. @dgw points out that the EOL+1y mark for Python 3.8 is coming soon, and is expected before Sopel 8.1.0, so possibly 3.8 will be removed in the lead-up to this. Since the milestone was mentioned, I'm adding this issue to it, but of course it should be re-assigned if this should be punted on.

In the absence of stated policy you usually start at "whatever it is that we do" but Sopel's last deprecation was in 8.0.0 for a wide range of versions which is not very tenable as an ongoing policy.

It would be useful to users if Sopel had some stated policy on this, possibly in the form of resolution of this issue.

Personal opinion

CPython's release cadence is now annual, so I propose that a lower-bound support window would be EOL + 1y subject to the discretion of the maintainers. More than that is generous, but indefinite support is not tenable, so there must be a cutoff somewhere. +1.5y, +2y would not be unreasonable either, but they increase the likelihood of the dependency pain of the type seen in #2683.

[dgw]

The upgrade guides for 7.0 and 8.0 are the nearest things we have to existing "policy" re: Python compatibility, AFAIK. For 7.0:

For the life of Sopel 7.x, we still plan to maintain compatibility with Python 2.7 unless it becomes absolutely necessary to drop it. For example, if a severe bug is found in one of Sopel’s dependencies, and the fix is only released for Python 3, we would consider dropping Python 2 support in the next minor version.

[…]

Whatever minimum Python version is required for Sopel 8.0.0 will remain supported (by best effort) throughout the 8.x series, just as has been done during the life of Sopel 7.x.

Then for 8.0:

During the lifecycle of Sopel 8.x, we will do our best to test against new Python releases as they come. We will continue to test against Python versions that reach end-of-life unless it becomes impractical to do so—in which case the next maintenance release of Sopel will officially drop support for the Python version(s) that can no longer be tested.

We're missing a formal definition of "impractical", at least. I might not think that "inability to use --groups" qualifies, for example, but another contributor could. (This isn't a great example since it's not what the above language was written to consider, but it's late and that's all I have right now. 😅)

I'm hesitant to define a specific lower time-bound because we do need an escape hatch if too many upstream dependencies too eagerly drop an EOL Python release. Ideally we don't drop any Pythons in a minor (second field) bump, but other projects certainly do. (Some of our dependency pins are the result of third-field drops, even…)

[SnoopJ]

Oh yes, agreed. I would have been better off saying "I would personally aim for no less than EOL+1y unless there are exceptional circumstances"

We're missing a formal definition of "impractical", at least. I might not think that "inability to use --groups" qualifies, for example, but another contributor could.

Or a user! A decision to drop an EOL Python that all the maintainers agree on could still upset someone who has a spacebar-overheating workflow. It's hard to be perfect here, but it is straightforward to be predictable at least.

I think this part of it can be swallowed up by leaving room for maintainer discretion and the rest is shrugged off to informal consensus and the particular context and impact of each decision. Not looking for rules-lawyer grade policy here, just giving a user some expectation for support past EOL.

[dgw]

Right now the expectation is "EOL will be supported until the next Sopel major, unless running the tests becomes unreasonably hard".

I know it's late for both of us and you're signing off—I should too, tbh—but the question for this issue to address is: "What's more predictable for user expectations, Sopel's next major release or EOL+Xyr(s)?"

Presumably the time-based one is, until/unless we suddenly get a lot more predictable with our own majors, since (C)Python has a regular cadence now.

[half-duplex]

I think it would be more meaningful to think about what distro releases we want to support. Repology shows:

Stuck on 3.8

• Amazon Linux 1 (EOL 2023)
• GoboLinux
• Mageia 8
• SliTaz Cooking
• Ubuntu 18.04 (EOL 2023)
• UnitedRPMs Fedora 33 and 34 (f34 EOL 2022)

Stuck on 3.9

• Alma/etc 9 without appstream
• Alpine 3.15 (EOL 2023)
• ALT Linux p10
• Apertis v2023
• CentOS 8 (non-Stream) (?? Alma 8 has 3.12 in appstream)
• Crux 3.6
• Debian 11 (oldstable->oldoldstable, EOL 2024-08, EOLTS 2026-08)
• Devuan 4 (oldstable)
• OpenIndiana
• OpenEuler 22.03
• OpenMandriva 4.3 (EOL 2022)
• Pardus 21 (EOL 2025-05)
• PureOS byzantium
• Raspbian oldstable (probably ->oldoldstable)
• Slackware{,64,ARM} 15 (latest, but pretty dead distro)
• SliTaz Current
• Trisquel 10
• Ubuntu 20.04 (EOL 2025-05)
• UnitedRPMs Fedora 32 (idk why this is newer than 33/34)

Stuck on 3.10

• Alpine 3.16 and 3.17 (EOL 2024-11)
• Crux 37
• Mageia 9
• OpenEuler 23.03
• PCLinuxOS

Stuck on 3.11

• Adelie current
• Alpine 3.18 (EOL 2025-05) and 3.19 (EOL 2025-11)
• Arch Linux 32 (as in the i386 SIG one)
• Debian 12 (stable->oldstable)
• Guix current
• HP-UX 11.31
• Raspbian stable (probably ->oldstable)
• Ubuntu 22.04 LTS (EOL 2027)
• Others I haven't heard of

So we don't start losing recent versions of widespread distros until we require 3.12. At 3.11, the most impactful affected distro releases are probably Debian 11, Ubuntu 20.04, and Raspbian oldstable, and each of those had a new LTS release 3 years ago (and Ubuntu a second LTS last year).

Based on this, my preference would be something along the lines of:

We will support Python versions old enough to allow use of any release of Alma, Alpine, Debian, or Ubuntu (any others? telemetry would be handy...) that had at least 3 months of mainstream support left at the time of the Sopel major (x.0) release. We will continue to support the latest Python available on the previous LTS release on a best-effort basis, and when convenient we will avoid breaking the latest Python on the oldest release still receiving free first-party security updates.

Including appstream, if we released 9.0 today that would be promising Python 3.11 (Alma 9: 3.12; Alpine 3.19: 3.11; Debian 12: 3.11; Ubuntu 22.04: 3.11). For Sopel 8.0, released Jun 2, 2024, that would be promising Python 3.10 (Alma 9: 3.12; Alpine 3.17: 3.10; Debian 12: 3.11; Ubuntu 22.04: 3.11).

The best-effort support timeline with that would be: 3.8 gone; 3.9 until Deb13 (2025); 3.10 until Deb13 (2025); 3.11 until Deb14 (2027).
Convenience support would be: 3.8 gone; 3.9 until Deb11 EOL (2026); 3.10 until same; 3.11 until Deb12 EOL (2028)

n.b. Alma 8 without appstream has 3.6, Alma 9 without appstream has 3.9. I don't know why you wouldn't leave it enabled, especially when you can install e.g. python3.12 alongside python3.9.

[dgw]

IMO, which release ships as "system Python" on any given distro is pretty meaningless when tools like pyenv exist. You yourself pointed out that multiple versions of Python can be installed alongside each other. It is slightly more convenient for users of older distros if Sopel can run directly on their system Python version, but it's trivial enough to get a newer version if needed (in most cases) that I don't think we need to worry about it as part of deciding when to drop support for a Python release.

For fun, I just did a pyenv install 3.7 on my Ubuntu-22.04 shell. It spent a few minutes compiling, and now I can install Sopel 7.1 with legacy plugins in a new virtualenv if I want to. (I don't want to, but the point is that I could.)

I can likewise run pyenv install 3.13 or even pyenv install 3.14.0a4 and get newer Python releases. (n.b.: this does stop working at some point on old-enough distros, if the latest versions of Python aren't compatible with the older compilation toolchain. We're not trying to enable running current Sopels on ancient OSes forever, though!)

To me, the main reason* for dropping older Pythons is still: Can still we do everything we want, code-wise, consistently across the whole Python release range we claim to support? And can we still test it?

As of the Sopel 8 upgrade guide, as quoted above, we committed to supporting any Pythons it launched with that become EOL until Sopel 9, unless it becomes "impractical". We can change that for 9.0 to allow dropping support in a minor release (e.g. 9.1), but my reading of the commitments already published is that we need to keep py3.8+ around in 8.x unless it actually breaks.

---

* Availability of security fixes — i.e. do we want to make it easier for Sopel users to run our internet-facing daemon on outdated platforms? — is worth thinking about at some point. But to really achieve that, we'd have to start getting a lot stricter about dependency versions beyond just Python itself, track upstream vulnerabilities and pin >= known-fixed versions, etc. That's a much bigger conversation about dependency management than just "When do we drop an EOL Python?"

[half-duplex]

I strongly prefer distro python because every additional tool I need to install/update/maintain adds friction and makes it more annoying, repeated every time I need to touch any of the pieces.
Great: apt install acme -> apt update; apt upgrade.
Okay: virtualenv acme; source acme/bin/activate; pip install acme -> source acme/bin/activate; pip install -U acme.
Really annoying: apt install pyenv; echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc; echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc; echo 'eval "$(pyenv init - bash)"' >> ~/.bashrc; (repeat for .profile and .bash_profile??); exec bash; apt install [a heap of python build dependencies]; pyenv install 3.13; ???? (no pyenv-virtualenv on debian); pip install acme -> ???? (no pyenv-update on debian); pip install -U acme
Really annoying: The above, but with a dirty install of pyenv instead of apt, so I might get pyenv-virtualenv and pyenv-update, but have to accept some bash script dumping stuff into my system unsupervised.

A lot of people are already annoyed by even venv+pip, so to me the reason for supporting versions of python other than current±1 is primarily to avoid adding that annoyance whenever possible.

[Exirel]

That's an interesting discussion! I like several arguments from different people, so I'll try to cherry-pick what I agree with and what I would like to add.

What He Said

First and foremost, I'd like to say that the current policy is based on vibes - in a good way. The vibe being @dgw advocating for bot owner's ease of installation and maintenance, and me advocating for bot core devs and plugin authors' experience of using new shinny toys.

Can still we do everything we want, code-wise, consistently across the whole Python release range we claim to support? And can we still test it?

We say, and have been saying, either in migration guide or in various conversations over IRC and other written support, that a major version of Sopel doesn't drop support for a Python version in a minor version unless it becomes unbearable for us to maintain. Again, unbearable being a vibe, because we know what we can bear and what we can't. And it's a good thing, because we tend to be very generous with bot owners, and we bear with a lot of shenanigans.

As a result, and @SnoopJ said it well:

Not looking for rules-lawyer grade policy here, just giving a user some expectation for support past EOL.

Packaged Python Version

Which leads me to my second point: how to choose which version of Python to support? While I understand @dgw (pyenv is rather convenient, ngl), I tend to prefer @half-duplex's approach.

I strongly prefer distro python because every additional tool I need to install/update/maintain adds friction and makes it more annoying, repeated every time I need to touch any of the pieces.

Because, I, too, is a lazy dev who hate on sysadmin tasks and I'd rather play video games or code some stupid 8ball plugin than deal with "how did I install that again?" and "oh yeah I need to upgrade this 5 years old server that host 12 websites supported by a spacebar heating workflow". In that regard, I like the idea of trying to support the LTS distro at the time of release of a major version: Ubuntu 22.04 for Sopel 8 for example.

Pip? A Short Tangent

A lot of people are already annoyed by even venv+pip

Which I understand, and that's why I can't really use pyenv as an argument to support any version. For that, however, I think Sopel should recommend using pipx - which solve most of the problem of using the packaged Python of the distribution, and it's an official tool too!

It's a double-edge sword, because:

• using pipx solve the issue of custom install (yay)
• but it still uses the packaged Python of the distribution (hm...)

So the solution justifies itself, that's neat, but maybe not very honest in a way. Maybe you'll pardon me for I am transparent on the matter!

The Release Cadence: The Harder, Better, Faster, Stronger Tangent

Why do we have a problem with the version of Python we have to support?

I'm quoting @SnoopJ so if it's not true that isn't on me:

CPython's release cadence is now annual.

That's a good thing while revealing an issue with Sopel's own release cycle: it's slow, it's very slow. For various reason, we don't release often. Also, we have been putting a lot of deprecation warning "to be removed in" Sopel 9, so we have to fix/replace things in Sopel 8.x... which takes a lot of time and maybe... maybe we have too much?

There is a great effort of removing/replacing/moving out builtin plugins, which have been the main sources of many hotfixes, while the core is kinda... living its own slow paced life. Maybe it's time to change that a bit? Since there isn't "more time" in our life, the only option is to reduce what's next in Sopel 8.1, and then we can either rush for Sopel 9 or try to do more in a Sopel 8.2...

Ok, that was a bit of a tangent, but I feel like it's important to consider not only Python's release cycle, but also Sopel's release cycle, because it's where we allow ourselves to make big changes. To cite @dgw:

What's more predictable for user expectations [...]?

User Expectations?

I don't pretend to know what our bot owners want. I know what I want: when I install Sopel, I want it to be able to support the latest Ubuntu LTS's packaged version of Python, and to give me enough room to support the previous Ubuntu LTS (like, it's 2025, so support Ubuntu 22.04 and 24.04). That's what I want, me, personally. My wants are not universal.

I also want more versions of Sopel, I want a reliable software, not with more feature, but with more predictable behaviors and more configuration options. I also want more plugins on the side that I can upgrade whenever and not worry too much about Sopel's compatibility with said plugins.

Everything else doesn't matter to me, as a bot owner. I don't really care if it's Python EOL+1 year, as long as I can install it on my Ubuntu 22.04 I setup 3 years ago.

The Revenge Of The Release Cadence

I said that Sopel's release cycle is maybe too slow, and having so many deprecation warnings to be removed in Sopel 9 put a lot of effort on having a stable public API. However, if I want more version (as a bot owner, a core dev, and a plugin author), I certainly don't want to upgrade all my code all the time: I want to be able to be as slow as it's acceptable.

On that note: popular Linux distributions can be a good benchmark of what is an acceptable slow pace.

Taking that into consideration, it is certain to me, at that point, that whatever Python version range we want to support, we must make the public API more stable, with less deprecation warning. To me, predictability is achieved more with a stable API than by any policy on which version of Python we support.