shutdown(SHUT_WR) does not work for SSLSocket, causing docker_api connection and docker_container_exec module to have problems with TCP TLS sockets

[christophert]

SUMMARY

Hosts targeted via the docker_containers inventory module fail facts gathering if TCP TLS socket is in use.

ISSUE TYPE

• Bug Report

COMPONENT NAME

community.docker.docker_containers.docker_api connection

ANSIBLE VERSION

ansible [core 2.14.4]
  config file = /home/user/Documents/Projects/ansible/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/user/Documents/Projects/ansible/.venv/lib64/python3.11/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/user/Documents/Projects/ansible/.venv/bin/ansible
  python version = 3.11.2 (main, Feb  8 2023, 00:00:00) [GCC 12.2.1 20221121 (Red Hat 12.2.1-4)] (/home/user/Documents/Projects/ansible/.venv/bin/python)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /home/user/.ansible/collections/ansible_collections
Collection       Version
---------------- -------
community.docker 3.4.3  

# /home/user/Documents/Projects/ansible/.venv/lib64/python3.11/site-packages/ansible_collections
Collection       Version
---------------- -------
community.docker 3.4.3  

# /home/user/Documents/Projects/ansible/.venv/lib/python3.11/site-packages/ansible_collections
Collection       Version
---------------- -------
community.docker 3.4.3  

CONFIGURATION

CONFIG_FILE() = /home/user/Documents/Projects/ansible/ansible.cfg
DEFAULT_ROLES_PATH(/home/user/Documents/Projects/ansible/ansible.cfg) = ['/home/user/Documents/Projects/ansible/roles']

OS / ENVIRONMENT

Linux vm-dev 6.2.9-200.fc37.aarch64 #1 SMP PREEMPT_DYNAMIC Thu Mar 30 22:54:14 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Docker host:

Linux kali 6.1.0-kali7-arm64 #1 SMP Debian 6.1.20-1kali1 (2023-03-22) aarch64 GNU/Linux

Docker version:

Client: Docker Engine - Community
 Version:           23.0.3
 API version:       1.42
 Go version:        go1.19.7
 Git commit:        3e7cbfd
 Built:             Tue Apr  4 22:02:03 2023
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.3
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.7
  Git commit:       59118bf
  Built:            Tue Apr  4 22:02:03 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

STEPS TO REPRODUCE

Setup a Docker host that exposes a TLS-protected TCP socket for control, deploy a container, and attempt to gather facts from the container/use async against the container.

Inventory files:
inventory/kali.yml:

---

docker_hosts:
  hosts:
    kali:
      ansible_host: 192.168.200.133
      ansible_connection: ssh
      ansible_user: mgmt
      ansible_password: password
      ansible_ssh_common_args: '-o StrictHostKeyChecking=no'

inventory/remote-docker.yml:

plugin: community.docker.docker_containers
docker_host: tcp://192.168.200.133:2376
tls: true
tls_hostname: server
validate_certs: true
ca_cert: credentials/ca.pem
client_cert: credentials/client.pem
client_key: credentials/client.key
groups:
  docker_containers: true

Container deployment and bootstrap:

---

- hosts: docker_hosts
  tasks:
    - name: deploy container
      community.docker.docker_container:
        name: test
        image: ubuntu:22.04
        command: sleep infinity
        detach: true
        auto_remove: true

    - meta: refresh_inventory

- hosts: docker_containers
  gather_facts: false
  roles:
    - bootstrap-install-python
  tasks:
    - name: gather facts
      gather_facts:

bootstrap-install-python/tasks/main.yml:

---

- name: Install Python
  ansible.builtin.raw: |
    if [ -x "$(command -v apt-get)" ]; then apt-get update && apt-get install -y python3.11-full python-is-python3
    elif [ -x "$(command -v dnf)" ]; then dnf install -y python3
    else echo "Failed to install package."; fi

EXPECTED RESULTS

Gathering facts should complete with any unhandled errors being thrown by Ansible.

ACTUAL RESULTS

The actual result is that ansible will hang for about 30-45 seconds when gathering facts, then return

[WARNING]: Unhandled error in Python interpreter discovery for host test:
Expecting value: line 1 column 1 (char 0)

This is not present when executing on a Docker TCP socket without TLS or with the local Unix socket.

ansible-playbook [core 2.14.4]
  config file = /home/user/Documents/Projects/ansible/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/user/Documents/Projects/ansible/.venv/lib64/python3.11/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/user/Documents/Projects/ansible/.venv/bin/ansible-playbook
  python version = 3.11.2 (main, Feb  8 2023, 00:00:00) [GCC 12.2.1 20221121 (Red Hat 12.2.1-4)] (/home/user/Documents/Projects/ansible/.venv/bin/python)
  jinja version = 3.1.2
  libyaml = True
Using /home/user/Documents/Projects/ansible/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /home/user/Documents/Projects/ansible/inventory/kali.yml as it did not pass its verify_file() method
script declined parsing /home/user/Documents/Projects/ansible/inventory/kali.yml as it did not pass its verify_file() method
Parsed /home/user/Documents/Projects/ansible/inventory/kali.yml inventory source with yaml plugin
setting up inventory plugins
host_list declined parsing /home/user/Documents/Projects/ansible/inventory/remote-docker.yml as it did not pass its verify_file() method
script declined parsing /home/user/Documents/Projects/ansible/inventory/remote-docker.yml as it did not pass its verify_file() method
Loading collection community.docker from /home/user/.ansible/collections/ansible_collections/community/docker
Using inventory plugin 'ansible_collections.community.docker.plugins.inventory.docker_containers' to process inventory source '/home/user/Documents/Projects/ansible/inventory/remote-docker.yml'
Parsed /home/user/Documents/Projects/ansible/inventory/remote-docker.yml inventory source with auto plugin
Loading callback plugin default of type stdout, v2.0 from /home/user/Documents/Projects/ansible/.venv/lib64/python3.11/site-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: test.yml *************************************************************
Positional arguments: playbooks/test.yml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/home/user/Documents/Projects/ansible/inventory/kali.yml', '/home/user/Documents/Projects/ansible/inventory/remote-docker.yml')
forks: 5
1 plays in playbooks/test.yml

PLAY [docker_containers] *******************************************************

TASK [Gathering Facts] *********************************************************
task path: /home/user/Documents/Projects/ansible/playbooks/test.yml:3
[WARNING]: Unhandled error in Python interpreter discovery for host test:
Expecting value: line 1 column 1 (char 0)
[WARNING]: Platform linux on host test is using the discovered Python
interpreter at /usr/bin/python3.11, but future installation of another Python
interpreter could change the meaning of that path. See
https://docs.ansible.com/ansible-
core/2.14/reference_appendices/interpreter_discovery.html for more information.
<test> ESTABLISH DOCKER CONNECTION FOR USER: ?
Trying to determine actual user
Actual user is ''
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c 'echo ~ && sleep 0'"]
<test> EXEC ['/bin/sh', '-c', '/bin/sh -c \'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1681167100.4895165-43204-22990817581602 `" && echo ansible-tmp-1681167100.4895165-43204-22990817581602="` echo /root/.ansible/tmp/ansible-tmp-1681167100.4895165-43204-22990817581602 `" ) && sleep 0\'']
<test> Attempting python interpreter discovery
<test> EXEC ['/bin/sh', '-c', '/bin/sh -c \'echo PLATFORM; uname; echo FOUND; command -v \'"\'"\'python3.11\'"\'"\'; command -v \'"\'"\'python3.10\'"\'"\'; command -v \'"\'"\'python3.9\'"\'"\'; command -v \'"\'"\'python3.8\'"\'"\'; command -v \'"\'"\'python3.7\'"\'"\'; command -v \'"\'"\'python3.6\'"\'"\'; command -v \'"\'"\'python3.5\'"\'"\'; command -v \'"\'"\'/usr/bin/python3\'"\'"\'; command -v \'"\'"\'/usr/libexec/platform-python\'"\'"\'; command -v \'"\'"\'python2.7\'"\'"\'; command -v \'"\'"\'/usr/bin/python\'"\'"\'; command -v \'"\'"\'python\'"\'"\'; echo ENDFOUND && sleep 0\'']
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c '/usr/bin/python3.11 && sleep 0'"], with stdin (1234 bytes)
<test> wrote 1234 bytes, 0 are left
<test> Shutting socket down for writing
<test> select... (None)
<test> select event read:True write:False
<test> read 498 bytes
<test> select... (None)
<test> select event read:True write:False
<test> read 24 bytes
<test> select... (None)
<test> select event read:True write:False
<test> read 0 bytes
Using module file /home/user/Documents/Projects/ansible/.venv/lib64/python3.11/site-packages/ansible/modules/setup.py
<test> PUT /home/user/.ansible/tmp/ansible-local-431996hdrm4ms/tmpkj4e_tms TO /root/.ansible/tmp/ansible-tmp-1681167100.4895165-43204-22990817581602/AnsiballZ_setup.py
<test> EXEC ['/bin/sh', '-c', 'id -u && id -g']
<test> PUT: Determined uid=b'0' and gid=b'0' for user ""
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1681167100.4895165-43204-22990817581602/ /root/.ansible/tmp/ansible-tmp-1681167100.4895165-43204-22990817581602/AnsiballZ_setup.py && sleep 0'"]
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c '/usr/bin/python3.11 /root/.ansible/tmp/ansible-tmp-1681167100.4895165-43204-22990817581602/AnsiballZ_setup.py && sleep 0'"]
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1681167100.4895165-43204-22990817581602/ > /dev/null 2>&1 && sleep 0'"]
ok: [test]

TASK [test] ********************************************************************
task path: /home/user/Documents/Projects/ansible/playbooks/test.yml:6
<test> ESTABLISH DOCKER CONNECTION FOR USER: ?
Trying to determine actual user
Actual user is ''
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c 'echo ~ && sleep 0'"]
<test> EXEC ['/bin/sh', '-c', '/bin/sh -c \'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1681167188.2423344-43258-266621346864131 `" && echo ansible-tmp-1681167188.2423344-43258-266621346864131="` echo /root/.ansible/tmp/ansible-tmp-1681167188.2423344-43258-266621346864131 `" ) && sleep 0\'']
Using module file /home/user/Documents/Projects/ansible/.venv/lib64/python3.11/site-packages/ansible/modules/command.py
<test> PUT /home/user/.ansible/tmp/ansible-local-431996hdrm4ms/tmpw9yzpqfk TO /root/.ansible/tmp/ansible-tmp-1681167188.2423344-43258-266621346864131/AnsiballZ_command.py
<test> EXEC ['/bin/sh', '-c', 'id -u && id -g']
<test> PUT: Determined uid=b'0' and gid=b'0' for user ""
<test> PUT /home/user/.ansible/tmp/ansible-local-431996hdrm4ms/tmpa5b3u4zx TO /root/.ansible/tmp/ansible-tmp-1681167188.2423344-43258-266621346864131/async_wrapper.py
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1681167188.2423344-43258-266621346864131/ /root/.ansible/tmp/ansible-tmp-1681167188.2423344-43258-266621346864131/AnsiballZ_command.py /root/.ansible/tmp/ansible-tmp-1681167188.2423344-43258-266621346864131/async_wrapper.py && sleep 0'"]
<test> EXEC ['/bin/sh', '-c', '/bin/sh -c \'ANSIBLE_ASYNC_DIR=\'"\'"\'~/.ansible_async\'"\'"\' /usr/bin/python3.11 /root/.ansible/tmp/ansible-tmp-1681167188.2423344-43258-266621346864131/async_wrapper.py 170992352143 15 /root/.ansible/tmp/ansible-tmp-1681167188.2423344-43258-266621346864131/AnsiballZ_command.py _ && sleep 0\'']
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c 'echo ~ && sleep 0'"]
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c 'echo ~ && sleep 0'"]
<test> EXEC ['/bin/sh', '-c', '/bin/sh -c \'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1681167189.9324355-43258-17675217124816 `" && echo ansible-tmp-1681167189.9324355-43258-17675217124816="` echo /root/.ansible/tmp/ansible-tmp-1681167189.9324355-43258-17675217124816 `" ) && sleep 0\'']
Using module file /home/user/Documents/Projects/ansible/.venv/lib64/python3.11/site-packages/ansible/modules/async_status.py
<test> PUT /home/user/.ansible/tmp/ansible-local-431996hdrm4ms/tmptolr2c4t TO /root/.ansible/tmp/ansible-tmp-1681167189.9324355-43258-17675217124816/AnsiballZ_async_status.py
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1681167189.9324355-43258-17675217124816/ /root/.ansible/tmp/ansible-tmp-1681167189.9324355-43258-17675217124816/AnsiballZ_async_status.py && sleep 0'"]
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c '/usr/bin/python3.11 /root/.ansible/tmp/ansible-tmp-1681167189.9324355-43258-17675217124816/AnsiballZ_async_status.py && sleep 0'"]
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c 'echo ~ && sleep 0'"]
Using module file /home/user/Documents/Projects/ansible/.venv/lib64/python3.11/site-packages/ansible/modules/async_status.py
<test> PUT /home/user/.ansible/tmp/ansible-local-431996hdrm4ms/tmpgz9bh8lh TO /root/.ansible/tmp/ansible-tmp-1681167189.9324355-43258-17675217124816/AnsiballZ_async_status.py
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1681167189.9324355-43258-17675217124816/ /root/.ansible/tmp/ansible-tmp-1681167189.9324355-43258-17675217124816/AnsiballZ_async_status.py && sleep 0'"]
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c '/usr/bin/python3.11 /root/.ansible/tmp/ansible-tmp-1681167189.9324355-43258-17675217124816/AnsiballZ_async_status.py && sleep 0'"]
<test> EXEC ['/bin/sh', '-c', "/bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1681167189.9324355-43258-17675217124816/ > /dev/null 2>&1 && sleep 0'"]
ASYNC OK on test: jid=170992352143.4734
changed: [test] => {
    "ansible_job_id": "170992352143.4734",
    "changed": true,
    "cmd": [
        "whoami"
    ],
    "delta": "0:00:00.002106",
    "end": "2023-04-10 22:53:08.805420",
    "finished": 1,
    "invocation": {
        "module_args": {
            "_raw_params": "whoami",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true
        }
    },
    "msg": "",
    "rc": 0,
    "results_file": "/root/.ansible_async/170992352143.4734",
    "start": "2023-04-10 22:53:08.803314",
    "started": 1,
    "stderr": "",
    "stderr_lines": [],
    "stdout": "root",
    "stdout_lines": [
        "root"
    ]
}

TASK [debug] *******************************************************************
task path: /home/user/Documents/Projects/ansible/playbooks/test.yml:11
ok: [test] => {
    "command": {
        "ansible_job_id": "170992352143.4734",
        "changed": true,
        "cmd": [
            "whoami"
        ],
        "delta": "0:00:00.002106",
        "end": "2023-04-10 22:53:08.805420",
        "failed": false,
        "finished": 1,
        "msg": "",
        "rc": 0,
        "results_file": "/root/.ansible_async/170992352143.4734",
        "start": "2023-04-10 22:53:08.803314",
        "started": 1,
        "stderr": "",
        "stderr_lines": [],
        "stdout": "root",
        "stdout_lines": [
            "root"
        ]
    }
}

PLAY RECAP *********************************************************************
test                       : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

[felixfontein]

Sorry this took so long (too busy with other things). I've been able to reproduce this and will try to debug it over the next days. In my case already interpreter discovery has some problems (and I found and fixed one bug in case Docker SDK for Python isn't installed)...

[felixfontein]

The problem is shutdown_writing() (in plugins/module_utils/socket_helper.py). It is called after sending all stdin data to tell Docker daemon that we are done sending stdin (without that it would hang as well, since it's piping program code into Python and Python is waiting for eof on stdin). This unfortunately causes the SSLSocket to stop decrypting, and returning raw data. Which in turn doesn't make any sense, thus leading to the observed behavior.

So shutdown_writing() needs to be fixed somehow for SSLSockets.

[felixfontein]

Hmm, the longer I look at this, the more I get the feeling that this is simply not possible at all with Python's SSLSocket. Basically what needs to be done is to send a close_notify alert. The only way to do this with SSLSocket/ssl seems to be shutting the socket down, but that completely turns off encryption for that socket, resulting in the behavior describe above - we get random-looking byte strings when reading.

[felixfontein]

I created #621 to warn about this in the docs and changelog.

I'm currently thinking that it might be possible to work around this by using PyOpenSSL's SSL.Connection, but that will probably require a lot of work, and has potential security pitfalls...

[felixfontein]

References:

• ssl.SSLSocket shutdown doesn't behave like socket.shutdown python/cpython#63080
• https://www.pyopenssl.org/en/latest/api/ssl.html
• https://docs.docker.com/engine/security/protect-access/ (for setting up Docker daemon with TCP TLS)
• https://www.rfc-editor.org/rfc/rfc5246#section-7.2.1

[felixfontein]

Some test code that can be useful for reproducing this without the whole baggage of Ansible modules/plugins:

import os
import selectors
import socket
import sys
import time

from ansible_collections.community.docker.plugins.module_utils._api.api.client import APIClient
from ansible_collections.community.docker.plugins.module_utils._api.tls import TLSConfig

from ansible_collections.community.docker.plugins.plugin_utils.socket_handler import (
    DockerSocketHandler,
)

# from docker import APIClient
# from docker.tls import TLSConfig

client = APIClient(base_url='tcp://127.0.0.1:2376', tls=TLSConfig(client_cert=(os.path.expanduser('~/.docker/cert.pem'), os.path.expanduser('~/.docker/key.pem')), ca_cert=os.path.expanduser('~/.docker/ca.pem'), verify=True))

url = client._url('/containers/{0}/exec', 'test')
res = client._post_json(url, data={
    'Container': 'test',
    'User': '',
    'Privileged': False,
    'Tty': False,
    'AttachStdin': True,
    'AttachStdout': True,
    'AttachStderr': True,
    'Cmd': ['python'],
    'Env': None,
})
cmd = client._result(res, True)

res = client._post_json(
    client._url('/exec/{0}/start', cmd['Id']),
    headers={
        'Connection': 'Upgrade',
        'Upgrade': 'tcp'
    },
    data={
        'Tty': False,
        'Detach': False
    },
    stream=True
)
exec_socket = client._get_raw_response_socket(res)

if True:
    from ansible_collections.community.docker.plugins.module_utils.socket_helper import (
        make_unblocking,
        shutdown_writing,
    )

    make_unblocking(exec_socket)
    exec_socket.send(b'print(1)\n\n\n')
    selector = selectors.DefaultSelector()
    selector.register(exec_socket, selectors.EVENT_READ)
    # exec_socket.shutdown(socket.SHUT_WR)
    eof = False
    while not eof:
        print('select...')
        events = selector.select(1)
        print(f'got {len(events)} events')
        for key, event in events:
            if key.fileobj == exec_socket:
                buf = exec_socket.recv(262144)
                print(buf)
                if not buf:
                    eof = True

    sys.exit()
else:
    class Display:
        def vvvv(self, output, host=None):
            print(host, output)

    display = Display()

    try:
        with DockerSocketHandler(display, exec_socket, container='test') as exec_socket_handler:
            exec_socket_handler.write(b'print(1)')
            stdout, stderr = exec_socket_handler.consume()
            print(stdout, stderr)
    finally:
        exec_socket.close()

This needs a container called test; something like docker run -d --name test python:3-alpine sleep 10h will do.