Python PIP completely broken

[felixlabrot]

Describe the bug

Today all Photon OS 5 servers have their Python 3.11 PIP completely destroyed.
Every attempt to install any package with pip install ... (even reinstalling pip) crashes with this error:

ERROR: Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
    status = _inner_run()
             ^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run
    return self.run(options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 363, in run
    resolver = self.make_resolver(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 175, in make_resolver
    import pip._internal.resolution.resolvelib.resolver
  File "/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 8, in <module>
    from pip._vendor.resolvelib import BaseReporter, ResolutionImpossible
  File "/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/__init__.py", line 19, in <module>
    from .resolvers import (
  File "/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers/__init__.py", line 1, in <module>
    from ..structs import RequirementInformation
ImportError: cannot import name 'RequirementInformation' from 'pip._vendor.resolvelib.structs' (/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/structs.py)

This is because a recent update is bugged and the pip version 24 package isn't compatible with itself anymore. Upgrading pip to version 25 fixes the bug. So a upgrade to the python3-pip package needs to be made in the tdnf repos, so every server gets the new and repaired version.

Reproduction steps

1. run pip install ...

Expected behavior

pip working and not exploding

Additional context

No response

[prashant1221]

pip install is working on my system. Can you give which version of python3-pip is installed on your system rpm -qa | grep python3-pip

See if downgrading python3-pip to lower release fixes the issue. We are trying to reproduce the issue from our end.

[felixlabrot]

An original install from the ISO comes with Python 3.11.0. After the first tdnf update it goes to 3.11.13. This is a version from 2022 with an update from June 2025. There is already a new update within 3.11 from over a month ago and the main Python package is 3 years old.

Installing the python3-pip package installs pip version 24.3.1 (pip 24.3.1 from /usr/lib/python3.11/site-packages/pip (python 3.11)). This version is from 13 months ago. There are already 6 newer releases.

The command rpm -qa | grep python3-pip gives me:

python3-pip-wheel-24.3.1-5.ph5.noarch
python3-pip-24.3.1-5.ph5.noarch

The Python package resolvelib comes in version 1.0.1. This version is not compatible and causes the issue. Manually upgrading pip to the current version upgrades resolvelib to 1.2.1 which is fully compatible again.

Basically the root cause of the issue is Photon OS holding on to years old packages which cause compatibility problems. This is not the first time this strategy caused a Python breakdown or a security vulnerability.

On a fresh Photon OS install with all tdnf updates installed those Python libraries are outdated:

Package            Version    Latest     Type
------------------ ---------- ---------- -----
attrs              22.1.0     25.4.0     wheel
certifi            2023.11.17 2025.11.12 wheel
cffi               1.15.1     2.0.0      wheel
chardet            5.0.0      5.2.0      wheel
charset-normalizer 2.1.1      3.4.4      wheel
configobj          5.0.6      5.0.9      wheel
cryptography       41.0.7     46.0.3     wheel
idna               3.3        3.11       wheel
jsonpatch          1.32       1.33       wheel
jsonpointer        2.3        3.0.0      wheel
jsonschema         4.16.0     4.25.1     wheel
MarkupSafe         2.1.1      3.0.3      wheel
oauthlib           3.2.2      3.3.1      wheel
pip                24.3.1     25.3       wheel
prettytable        3.3.0      3.17.0     wheel
pyasn1             0.4.8      0.6.1      wheel
pycparser          2.21       2.23       wheel
pyOpenSSL          23.3.0     25.3.0     wheel
pyparsing          3.0.9      3.2.5      wheel
pyrsistent         0.18.1     0.20.0     wheel
PyYAML             5.4.1      6.0.3      wheel
requests           2.28.1     2.32.5     wheel
setuptools         None       80.9.0     wheel
six                1.16.0     1.17.0     wheel
urllib3            1.26.19    2.5.0      wheel

Getting the latest pip version upgrades all of it's deps to compatible ones and this fixes the issue permanently:

curl -sS https://bootstrap.pypa.io/get-pip.py -o /root/get-pip.py
python3 /root/get-pip.py "pip>=25.0"

The issue occured in all 11 Photon OS servers I have at the same time. We run tdnf update automatically once a week so I don't have a history of which version of all the packages was before and after. I just know it broke after the update on the evening of 22. Nov.

[oliverkurth]

We run tdnf update automatically once a week so I don't have a history of which version of all the packages was before and after.

You can look at the history with tdnf history --info. See https://github.com/vmware/tdnf/wiki/Commands#history-command- .

[felixlabrot]

This is the update which broke everything:

  48 (unknown)                                                                                                                                                                                            Sat Nov 22 2025 21:21 +24   / -22
added: Linux-PAM-1.5.3-10.ph5.x86_64, systemd-libs-253.19-17.ph5.x86_64, shadow-libs-4.13-12.ph5.x86_64, openssh-clients-9.3p2-17.ph5.x86_64, systemd-rpm-macros-253.19-17.ph5.noarch, dbus-libs-1.15.4-8.ph5.x86_64, libpwquality-1.4.4-7.ph5.x86_64, shadow-tools-4.13-12.ph5.x86_64, shadow-4.13-12.ph5.x86_64, openssh-server-9.3p2-17.ph5.x86_64, docker-cli-28.2.2-3.ph5.x86_64, python3-pip-wheel-24.3.1-4.ph5.noarch, kmod-binary-34.1-3.ph5.x86_64, kmod-34.1-3.ph5.x86_64, systemd-pam-253.19-17.ph5.x86_64, systemd-253.19-17.ph5.x86_64, containerd-2.1.5-1.ph5.x86_64, containerd-extras-2.1.5-1.ph5.x86_64, docker-28.2.2-3.ph5.x86_64, systemd-udev-253.19-17.ph5.x86_64, openssh-9.3p2-17.ph5.x86_64, linux-esx-6.1.158-3.ph5.x86_64, python3-pip-24.3.1-4.ph5.noarch, dbus-1.15.4-8.ph5.x86_64
removed: dbus-1.15.4-7.ph5.x86_64, python3-pip-wheel-24.3.1-3.ph5.noarch, python3-pip-24.3.1-3.ph5.noarch, kmod-34.1-2.ph5.x86_64, systemd-libs-253.19-15.ph5.x86_64, openssh-clients-9.3p2-16.ph5.x86_64, systemd-rpm-macros-253.19-15.ph5.noarch, systemd-pam-253.19-15.ph5.x86_64, systemd-253.19-15.ph5.x86_64, libpwquality-1.4.4-6.ph5.x86_64, openssh-server-9.3p2-16.ph5.x86_64, openssh-9.3p2-16.ph5.x86_64, systemd-udev-253.19-15.ph5.x86_64, Linux-PAM-1.5.3-9.ph5.x86_64, shadow-libs-4.13-11.ph5.x86_64, shadow-4.13-11.ph5.x86_64, shadow-tools-4.13-11.ph5.x86_64, docker-cli-28.2.2-2.ph5.x86_64, containerd-2.1.3-2.ph5.x86_64, containerd-extras-2.1.3-2.ph5.x86_64, linux-esx-6.1.158-2.ph5.x86_64, docker-28.2.2-2.ph5.x86_64

[bhllamoreaux]

Can you try running rpm -qf /usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers/__init__.py? This file doesn't seem to exist in python3-pip-24.3.1 as far as I can tell (it's not present on my system in general either). I'm curious where it's coming from for you?

[felixlabrot]

file /usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers/__init__.py is not owned by any package
No one has installed this. The only change to Python ever was tdnf install python3-pip and pip install docker, no change was made otherwise.

[bhllamoreaux]

As best as I can tell, this file was never packaged in any version of python3-pip provided by Photon. The upstream change happened in pypa/pip@1838962, which went in to v25.1. Photon is still on 24.3.1.

I was able to reproduce your bug by manually installing pip3==25.3 via get_pip.py, and then reinstalling our Photon version with tdnf. This causes the conflict between the two installed pip versions. Is it possible this is what happened in your case?

Photon PIP is not broken by itself, you can check it yourself on a clean install (e.g docker container):

root@photon-blam [ ~ ]# docker run -it --network=host photon:5.0 /bin/bash 
root [ / ]# tdnf install -y python3-pip &> /dev/null
root [ / ]# pip3 install requests
Collecting requests
  Downloading requests-2.32.5-py3-none-any.whl.metadata (4.9 kB)
Collecting charset_normalizer<4,>=2 (from requests)
  Downloading charset_normalizer-3.4.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (37 kB)
Collecting idna<4,>=2.5 (from requests)
  Downloading idna-3.11-py3-none-any.whl.metadata (8.4 kB)
Collecting urllib3<3,>=1.21.1 (from requests)
  Downloading urllib3-2.5.0-py3-none-any.whl.metadata (6.5 kB)
Collecting certifi>=2017.4.17 (from requests)
  Downloading certifi-2025.11.12-py3-none-any.whl.metadata (2.5 kB)
Downloading requests-2.32.5-py3-none-any.whl (64 kB)
Downloading certifi-2025.11.12-py3-none-any.whl (159 kB)
Downloading charset_normalizer-3.4.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (151 kB)
Downloading idna-3.11-py3-none-any.whl (71 kB)
Downloading urllib3-2.5.0-py3-none-any.whl (129 kB)
Installing collected packages: urllib3, idna, charset_normalizer, certifi, requests
Successfully installed certifi-2025.11.12 charset_normalizer-3.4.4 idna-3.11 requests-2.32.5 urllib3-2.5.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

And just to be safe, I tested both the upgrade and downgrade paths, and I'm not seeing any issues.

[felixlabrot]

get_pip.py was never used on any system prior to the total failure of pip. I only tried this as a resolution method after pip was broken and had success with it.