Docker Socket API broken after Python update

[felixlabrot]

Describe the bug

Since the latest update to some Python packages it is not possible anymore to access the Docker API via unix://var/run/docker.sock. All connection attempts fail with Not supported URL scheme http+docker universally on all Photon OS servers. This entirely breaks monitoring with all Python based monitoring solutions (like Check_MK) and likely much more.

The bug is in one of those packages:

python3-jinja2-3.1.2-5.ph5.noarch
python3-requests-2.28.1-6.ph5.noarch
python3-xml-3.11.9-7.ph5.x86_64
python3-libs-3.11.9-7.ph5.x86_64
python3-3.11.9-7.ph5.x86_64
expat-2.6.0-6.ph5.x86_64
expat-libs-2.6.0-6.ph5.x86_64

Reproduction steps

1. run Photon OS with all updates installed
2. enable and start Docker
3. install package python3-pip
4. run pip install docker==7.0.0 (with latest version 7.1.0 the library is broken as well)
5. run this script: https://gist.github.com/felixlabrot/c96a6d2018b7ee7dcb004a032f5c323a

Expected behavior

It should be possible to connect to the Docker API like it was before.

Additional context

No response

[dcasota]

Your script indicates a custom Photon OS compilation due to the fact that check_mk has specific libraries.
On a Workstation 17.x vm deployed by a recently compiled Photon OS 5.0 iso, an api test seems to be successful. Here the sample from that machine (with a custom docker plexserver).

curl --unix-socket /var/run/docker.sock http://api/containers/json | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2157    0  2157    0     0   262k      0 --:--:-- --:--:-- --:--:--  263k
[
  {
    "Id": "bec3e975976aeea0abb3e2b72420d78d4ccbba9b79b8661510ac7a0c812ca4dd",
    "Names": [
      "/plex"
    ],
    "Image": "lscr.io/linuxserver/plex:latest",
    "ImageID": "sha256:7c39c1281ab063d0e483b1a74f3e6be89e94e8f03b7162962933c474f734bb9f",
    "Command": "/init",
    "Created": 1735066503,
    "Ports": [],
    "Labels": {
      "build_version": "Linuxserver.io version:- 1.41.3.9314-a0bfb8370-ls250 Build-date:- 2024-12-23T09:23:54+00:00",
      "maintainer": "thelamer",
      "org.opencontainers.image.authors": "linuxserver.io",
...

Docker version

root@photon-f2966979acd9 [ ~ ]# docker version
Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.21.13
 Git commit:        3ab4256
 Built:             Tue Dec 24 04:06:55 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Tue Dec 24 04:08:04 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.14
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
root@photon-f2966979acd9 [ ~ ]#

From your script, did you use the RedHat check-mk-raw-2.3.0p23-el8-38.x86_64.rpm version of 2.3.0p23 ? I'm asking because it needs quite a few dependencies during installation.

rpm -i ./check-mk-raw-2.3.0p23-el8-38.x86_64.rpm
warning: ./check-mk-raw-2.3.0p23-el8-38.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID c4503261: NOKEY
error: Failed dependencies:
        bind-utils is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        binutils is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        cronie is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        dialog is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        freeradius-utils is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        glib2 is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        graphviz is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        graphviz-gd is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        httpd is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        libevent is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        libgsf is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        libpq is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        libtool-ltdl is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        logrotate is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        pango is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        perl-IO-Zlib is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        perl-Locale-Maketext-Simple is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        perl-Net-Ping is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php-cli is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php-gd is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php-json is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php-mbstring is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php-pdo is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        php-xml is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        poppler-utils is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        rpcbind is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        rpm-build is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        rsync is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        time is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        traceroute is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        uuid is needed by check-mk-raw-2.3.0p23-el8-38.x86_64
        xinetd is needed by check-mk-raw-2.3.0p23-el8-38.x86_64

Or did you use the docker version [1] as described in the [2]?

Accordingly to the latest commits in the official Photon OS 5.0 repository, see [3], afaik most changes were not related to python. There was a cve related fix for python virtualenv [4]. Can you describe ~"broken"?

edited:
The Not supported URL scheme http+docker issue seems be docker-py related only, e.g. [5]. A few users downgraded/pinned the requests module pip install requests==2.31 but that was before docker-py 7.1.0.

[1] docker container run -dit -p 8080:5000 -p 8000:8000 --tmpfs /opt/omd/sites/cmk/tmp:uid=1000,gid=1000 -v monitoring:/omd/sites --name monitoring -v /etc/localtime:/etc/localtime:ro --restart always checkmk/check-mk-raw:2.3.0p23
[2] https://checkmk.com/download?method=docker&edition=cre&version=2.3.0p23
[3] https://github.com/vmware/photon/commits/5.0/
[4] 4730351
[5] docker/docker-py#3256

[felixlabrot]

You are entirely wrong with those packages. You thought about the Check_MK server, which is not compatible with Photon OS. I am talking about the monitoring agent, which is not even needed to replicate the issue. The monitoring agent just calls the mentioned Python script and processes the text output it creates. To replicate the problem, there isn't even a need to install the agent. Just follow exactly my replication steps and do nothing more than that. That means your [1] and [2] are entirely not needed and are about something completely different.

The PIP package "docker-py" must not be used as per Check_MK documentation.
But your hint with the "requests" package brought me onto something.
The general problem with PIP is, that there isn't any command to update all packages, but that is the key to fix the problem. There is apparently something entirely broken when updating python3-requests with tdnf but then not updating it with pip as well.

This command is the magic fix to the issue:
pip list --outdated | cut -d" " -f1 | tail -n +3 | xargs -L1 pip install --upgrade

[dcasota]

So your enhancement suggestion is that eligible 'python3-*'-packages are updating their 'pip'-version as well? Good catch.

@prashant1221 Could you discuss this enhancement suggestion internally?

[felixlabrot]

Yes, because my issue has proven that a version mismatch can lead to broken packages. When both things are automatically kept in sync that can prevent such issues in the future.
In my case this has caused an outage for all Docker monitoring in our datacenter. A very unpleasant thing to have.

[dcasota]

Please re-open the issue, so the Photon OS team can have a look at it. Be patient. Thank you for your help. Thankful.

[sshedi]

pip list --outdated | cut -d" " -f1 | tail -n +3 | xargs -L1 pip install --upgrade
this can easily lead to an unusable python subsystem in the machine. It will probably mess up the dependency chain of the python packages.

We recommend using docker-py3 which is supplied by Photon.
tdnf install --refresh -y docker-py3 should make things come back to normal.

[dcasota]

Hi Shreenidhi,

An tdnf package docker-py 7.1.0 would help for this issue ticket.
Actually, tdnf list docker-py3 shows up the 6.0.0-5 bits, but pip list docker | grep docker shows up docker 7.1.0.

It will probably mess up the dependency chain of the python packages.
Yes, this is indeed always the case with non-event driven autonomous version updates.
That is today's claim: Is there an update e.g. in a Docker component? Then after a few hours the update should also be in an Automatic Version Bump branch. And many enthusiasts will then publish it with fancy texts {Boom. This is huge. Now we are talking. Gamechanger. It's so over. look at me - we are the media now, breaking news, it's crazy}... same boat :-/

---

Here for the sake of completeness, reproduction steps as described from Felix. See attached logfile.

1. run Photon OS with all updates installed: line 18: done
2. enable and start Docker: line 20: done
3. install package python3-pip: line 22: done
4. run pip install docker==7.0.0 (with latest version 7.1.0 the library is broken as well): line 36: done
5. run this script: https://gist.github.com/felixlabrot/c96a6d2018b7ee7dcb004a032f5c323a: line 59 : not okay

Then, your suggestion:
line 64: tdnf install --refresh -y docker-py3: done
line 97: rerun the script: not okay

Then, Felix' suggestion:
line 102: pip list --outdated | cut -d" " -f1 | tail -n +3 | xargs -L1 pip install --upgrade
line 515: rerun the script: okay

[felixlabrot]

The issue was initially caused by a version mismatch between the tdnf package and the pip package.
My suggestion would be that every tdnf update of a Python package automatically should trigger the pip update of corresponding package as pointed out by @dcasota to prevent mismatches, which can lead to broken functionality.

[sshedi]

@felixlabrot if you let us know the mismatched pips, we will try to fix it.
pip list --outdated | cut -d" " -f1 | tail -n +3 | xargs -L1 pip install --upgrade is not really helpful.

[dcasota]

:-/ Sounds more like unleash hell on earth... Pypi doesn't do any vulnerability scanning or audits to packages published.

Built-in scanned Photon OS subcomponent' sources have their origin repositories in VMware internal repos, Kernel.org, Github.com, Sourgeforge.net, Fedoraproject.org, Rubygems.org, Freedesktop.org, Cpan.org, ftp-based sources e.g. gnu, mozilla, Gitlab-based, and more. "Built-in" sometimes still is handmade, but as I've understood, the team is heavily working on devsecops automation.
In addition, the SRP initiative is really promising.

For python packages, there are promising projects e.g. https://github.com/pyupio/safety but status quo, I trust tdnf-curated packages more than python packages with all those input validation failures, exposed debug information, web application vulnerabilities, outdated dependencies, compromised temporary files, malicious packages, etc.

The issue was initially caused by a version mismatch between the tdnf package and the pip package.
Yes to faster secure update cycles!