community.crypto.openssl_privatekey module also needs 'cffi'

[villapx-path]

SUMMARY

The community.crypto.openssl_privatekey module also requires the cffi package, however the documentation only lists cryptography as a dependency.

ISSUE TYPE

• Documentation Report

COMPONENT NAME

community.crypto.openssl_privatekey

ANSIBLE VERSION

ansible [core 2.17.4]
  config file = /home/jvilla/projects/path-production-ansible/ansible.cfg
  configured module search path = ['/home/jvilla/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jvilla/.local/pipx/venvs/ansible-core/lib/python3.10/site-packages/ansible
  ansible collection location = /home/jvilla/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/jvilla/.local/bin/ansible
  python version = 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] (/home/jvilla/.local/pipx/venvs/ansible-core/bin/python)
  jinja version = 3.1.4
  libyaml = True

[villapx-path]

Some more details:

I am calling the community.crypto.openssl_privatekey module, and it fails because it is missing the "cffi backend" module.

Here is the -vvv output of this failing task:

ansible-playbook -vvv output

<localhost> ESTABLISH LOCAL CONNECTION FOR USER: jvilla
<localhost> EXEC /bin/sh -c 'echo ~jvilla && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/jvilla/.ansible/tmp `"&& mkdir "` echo /home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243 `" && echo ansible-tmp-1727964631.7938662-12315-25707778985243="` echo /home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243 `" ) && sleep 0'
Using module file /home/jvilla/.ansible/collections/ansible_collections/community/crypto/plugins/modules/openssl_privatekey.py
<localhost> PUT /home/jvilla/.ansible/tmp/ansible-local-121319ffj6jk5/tmp8rv9m_a5 TO /home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243/AnsiballZ_openssl_privatekey.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243/ /home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243/AnsiballZ_openssl_privatekey.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.11 /home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243/AnsiballZ_openssl_privatekey.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243/AnsiballZ_openssl_privatekey.py", line 107, in <module>
    _ansiballz_main()
  File "/home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243/AnsiballZ_openssl_privatekey.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243/AnsiballZ_openssl_privatekey.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.community.crypto.plugins.modules.openssl_privatekey', init_globals=dict(_module_fqn='ansible_collections.community.crypto.plugins.modules.openssl_privatekey', _modlib_path=modlib_path),
  File "<frozen runpy>", line 226, in run_module
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/ansible_community.crypto.openssl_privatekey_payload_lu_qg4ot/ansible_community.crypto.openssl_privatekey_payload.zip/ansible_collections/community/crypto/plugins/modules/openssl_privatekey.py", line 301, in <module>
  File "/tmp/ansible_community.crypto.openssl_privatekey_payload_lu_qg4ot/ansible_community.crypto.openssl_privatekey_payload.zip/ansible_collections/community/crypto/plugins/modules/openssl_privatekey.py", line 281, in main
  File "/tmp/ansible_community.crypto.openssl_privatekey_payload_lu_qg4ot/ansible_community.crypto.openssl_privatekey_payload.zip/ansible_collections/community/crypto/plugins/module_utils/crypto/module_backends/privatekey.py", line 498, in select_backend
  File "/tmp/ansible_community.crypto.openssl_privatekey_payload_lu_qg4ot/ansible_community.crypto.openssl_privatekey_payload.zip/ansible_collections/community/crypto/plugins/module_utils/crypto/module_backends/privatekey.py", line 301, in __init__
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/backends/__init__.py", line 14, in default_backend
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 6, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 114, in <module>
    from cryptography.hazmat.bindings.openssl import binding
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 14, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
ModuleNotFoundError: No module named '_cffi_backend'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243/AnsiballZ_openssl_privatekey.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243/AnsiballZ_openssl_privatekey.py\", line 99, in _ansiballz_main\n    in
voke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/jvilla/.ansible/tmp/ansible-tmp-1727964631.7938662-12315-25707778985243/AnsiballZ_openssl_privatekey.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.community.crypto.plugins.modules.openssl_privatekey', init_globals=dict(_module_fqn='ansible_collections.community.crypto.plugins
.modules.openssl_privatekey', _modlib_path=modlib_path),\n  File \"<frozen runpy>\", line 226, in run_module\n  File \"<frozen runpy>\", line 98, in _run_module_code\n  File \"<frozen runpy>\", line 88, in _run_code\n  File \"/tmp/ansible_community.crypto.openssl_privatekey_payload_lu_qg4ot/ansible_community.crypto.openssl_privatekey_payload.zip/ansible_collections/community/crypt
o/plugins/modules/openssl_privatekey.py\", line 301, in <module>\n  File \"/tmp/ansible_community.crypto.openssl_privatekey_payload_lu_qg4ot/ansible_community.crypto.openssl_privatekey_payload.zip/ansible_collections/community/crypto/plugins/modules/openssl_privatekey.py\", line 281, in main\n  File \"/tmp/ansible_community.crypto.openssl_privatekey_payload_lu_qg4ot/ansible_commun
ity.crypto.openssl_privatekey_payload.zip/ansible_collections/community/crypto/plugins/module_utils/crypto/module_backends/privatekey.py\", line 498, in select_backend\n  File \"/tmp/ansible_community.crypto.openssl_privatekey_payload_lu_qg4ot/ansible_community.crypto.openssl_privatekey_payload.zip/ansible_collections/community/crypto/plugins/module_utils/crypto/module_backends/pr
ivatekey.py\", line 301, in __init__\n  File \"/usr/lib/python3/dist-packages/cryptography/hazmat/backends/__init__.py\", line 14, in default_backend\n    from cryptography.hazmat.backends.openssl.backend import backend\n  File \"/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/__init__.py\", line 6, in <module>\n    from cryptography.hazmat.backends.openssl.bac
kend import backend\n  File \"/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/backend.py\", line 114, in <module>\n    from cryptography.hazmat.bindings.openssl import binding\n  File \"/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py\", line 14, in <module>\n    from cryptography.hazmat.bindings._openssl import ffi, lib\nModuleNot
FoundError: No module named '_cffi_backend'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

Per that output, it is executing /usr/bin/python3.11 on the host, which at the time of this error had the following packages installed:

pip list, when the error occurred

$ /usr/bin/python3.11 -m pip list
Package             Version
------------------- -------------
argcomplete         1.8.1
attrs               21.2.0
Automat             20.2.0
Babel               2.8.0
bcrypt              3.2.0
blinker             1.4
certifi             2020.6.20
chardet             4.0.0
click               8.0.3
cloud-init          24.3.1
colorama            0.4.4
command-not-found   0.3
configobj           5.0.6
constantly          15.1.0
cryptography        3.4.8
dbus-python         1.2.18
distro              1.7.0
distro-info         1.1+ubuntu0.2
httplib2            0.20.2
hyperlink           21.0.0
idna                3.3
importlib-metadata  4.6.4
incremental         21.3.0
jeepney             0.7.1
Jinja2              3.0.3
jsonpatch           1.32
jsonpointer         2.0
jsonschema          3.2.0
keyring             23.5.0
launchpadlib        1.10.16
lazr.restfulclient  0.14.4
lazr.uri            1.0.6
livereload          2.6.3
Markdown            3.3.6
MarkupSafe          2.0.1
mkdocs              1.1.2
more-itertools      8.10.0
netifaces           0.11.0
oauthlib            3.2.0
packaging           21.3
pip                 22.0.2
pipx                1.0.0
psutil              5.9.0
pyasn1              0.4.8
pyasn1-modules      0.2.1
pycurl              7.44.1
Pygments            2.11.2
PyGObject           3.42.1
PyHamcrest          2.0.2
pyinotify           0.9.6
PyJWT               2.3.0
pyOpenSSL           21.0.0
pyparsing           2.4.7
pyrsistent          0.18.1
pyserial            3.5
python-apt          2.4.0+ubuntu4
pytz                2022.1
PyYAML              5.4.1
requests            2.25.1
SecretStorage       3.3.1
service-identity    18.1.0
setuptools          59.6.0
six                 1.16.0
systemd-python      234
tornado             6.1
Twisted             22.1.0
ubuntu-pro-client   8001
ufw                 0.36.1
unattended-upgrades 0.1
urllib3             1.26.5
userpath            1.8.0
wadllib             1.3.6
wheel               0.37.1
zipp                1.0.0
zope.interface      5.4.0

After installing cffi (output below), the task succeeded:

pip install cffi, and re-run playbook

[10:14] jvilla@PATH-HPWNJX3:~/projects/my-playbook $ python3.11 -m pip install cffi
Defaulting to user installation because normal site-packages is not writeable
Collecting cffi
  Downloading cffi-1.17.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (467 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 467.2/467.2 KB 4.5 MB/s eta 0:00:00
Collecting pycparser
  Downloading pycparser-2.22-py3-none-any.whl (117 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.6/117.6 KB 26.9 MB/s eta 0:00:00
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.17.1 pycparser-2.22
...
...
...
...
[10:14] jvilla@PATH-HPWNJX3:~/projects/my-playbook $ ansible-playbook -vvv generate_vpn_certs.yml
ansible-playbook [core 2.17.4]
  config file = /home/jvilla/projects/my-playbook/ansible.cfg
  configured module search path = ['/home/jvilla/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jvilla/.local/pipx/venvs/ansible-core/lib/python3.10/site-packages/ansible
  ansible collection location = /home/jvilla/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/jvilla/.local/bin/ansible-playbook
  python version = 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] (/home/jvilla/.local/pipx/venvs/ansible-core/bin/python)
  jinja version = 3.1.4
  libyaml = True
...
...
... lots of -vvv output and normal task output ...
...
...
TASK [vpn_config_generate : Generate an OpenSSL private key with the default values (4096 bits, RSA)] ***
task path: /home/jvilla/projects/my-playbook/roles/vpn_config_generate/tasks/main.yml:33
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: jvilla
<localhost> EXEC /bin/sh -c 'echo ~jvilla && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/jvilla/.ansible/tmp `"&& mkdir "` echo /home/jvilla/.ansible/tmp/ansible-tmp-1727964878.9725957-12642-118435769711826 `" && echo ansible-tmp-1727964878.9725957-12642-118435769711826="` echo /home/jvilla/.ansible/tmp/ansible-tmp-1727964878.9725957-12642-118435769711826 `" ) && sleep 0'
Using module file /home/jvilla/.ansible/collections/ansible_collections/community/crypto/plugins/modules/openssl_privatekey.py
<localhost> PUT /home/jvilla/.ansible/tmp/ansible-local-124581t36wg16/tmp1ziz87xo TO /home/jvilla/.ansible/tmp/ansible-tmp-1727964878.9725957-12642-118435769711826/AnsiballZ_openssl_privatekey.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/jvilla/.ansible/tmp/ansible-tmp-1727964878.9725957-12642-118435769711826/ /home/jvilla/.ansible/tmp/ansible-tmp-1727964878.9725957-12642-118435769711826/AnsiballZ_openssl_privatekey.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.11 /home/jvilla/.ansible/tmp/ansible-tmp-1727964878.9725957-12642-118435769711826/AnsiballZ_openssl_privatekey.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/jvilla/.ansible/tmp/ansible-tmp-1727964878.9725957-12642-118435769711826/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => {
    "changed": true,
    "diff": {
        "after": {
            "can_parse_key": true,
            "key_is_consistent": null,
            "public_data": {
                <<<redacted>>>
            },
            "public_key": "-----BEGIN PUBLIC KEY-----\n<<<redacted>>>\n-----END PUBLIC KEY-----\n",
            "public_key_fingerprints": {
                "sha256": <<<redacted>>>
            },
            "type": "RSA"
        },
        "before": {}
    },
    "filename": "./ssl_tmp/client.key",
    "fingerprint": {
        <<<redacted>>>
    },
    "invocation": {
        "module_args": {
            "attributes": null,
            "backup": false,
            "cipher": "auto",
            "curve": null,
            "force": false,
            "format": "auto_ignore",
            "format_mismatch": "regenerate",
            "group": null,
            "mode": "0600",
            "owner": null,
            "passphrase": null,
            "path": "./ssl_tmp/client.key",
            "regenerate": "full_idempotence",
            "return_content": false,
            "select_crypto_backend": "auto",
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "size": 4096,
            "state": "present",
            "type": "RSA",
            "unsafe_writes": false
        }
    },
    "size": 4096,
    "type": "RSA"
}
Read vars_file 'openvpn_vaulted.yml'
Read vars_file 'terraform_vars.yml'

[felixfontein]

The module only needs cryptography. cffi is a transitive dependency needed by cryptography: https://github.com/pyca/cryptography/blob/main/pyproject.toml#L52

[villapx-path]

@felixfontein Thanks for the response!

What do you think then is the "proper" course of action for me? This has already been filed as a bug with the cryptography project 7 years ago, and they said that it is explicitly configured to not list cffi as a dependency when installing cryptography from PyPi.

Is the requirement basically, "ensure you have cryptography installed on the target system, and also cffi if you installed cryptography via PyPi"?

[felixfontein]

The issue you are linking is about PyPy (another Python engine), not PyPI (the Python package index). If you install cryptography via pip (or any other proper resolver) for CPython (or in fact any other Python engine than PyPy), cffi should also be installed since cffi is a dependency of cryptography.

(Your Python from ansible --version doesn't look like PyPy.)

If you have cryptography installed but not cffi, then your environment is broken.

[villapx-path]

Gotcha lol, thanks for correcting my misunderstanding on the PyPy vs. PyPi 😄

then your environment is broken

This is a pretty standard install of the python3.11 apt package on Ubuntu 22.04 in WSL. Although, Ubuntu 22.04's default Python version (i.e. /usr/bin/python3) is Python 3.10, and so perhaps that is what was causing the issue?

---

Some details:

[18:04] jvilla@PATH-HPWNJX3:~ $ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
[17:04] jvilla@PATH-HPWNJX3:~ $ 
[17:04] jvilla@PATH-HPWNJX3:~ $ 
[17:04] jvilla@PATH-HPWNJX3:~ $ 
[17:04] jvilla@PATH-HPWNJX3:~ $ dpkg -l python3.11 python3-cryptography python3-cffi-backend
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                       Version            Architecture Description
+++-==========================-==================-============-=======================================================================
ii  python3-cffi-backend:amd64 1.15.0-1build2     amd64        Foreign Function Interface for Python 3 calling C code - runtime
ii  python3-cryptography       3.4.8-1ubuntu2.2   amd64        Python library exposing cryptographic recipes and primitives (Python 3)
ii  python3.11                 3.11.0~rc1-1~22.04 amd64        Interactive high-level object-oriented language (version 3.11)
[17:04] jvilla@PATH-HPWNJX3:~ $ 
[17:04] jvilla@PATH-HPWNJX3:~ $ 
[17:04] jvilla@PATH-HPWNJX3:~ $ dpkg -L python3-cffi-backend 
/.
/usr
/usr/lib
/usr/lib/python3
/usr/lib/python3/dist-packages
/usr/lib/python3/dist-packages/_cffi_backend.cpython-310-x86_64-linux-gnu.so
/usr/share
/usr/share/doc
/usr/share/doc/python3-cffi-backend
/usr/share/doc/python3-cffi-backend/changelog.Debian.gz
/usr/share/doc/python3-cffi-backend/copyright

Not being quite this level of Python expert, the _cffi_backend.cpython-310-x86_64-linux-gnu.so seems somewhat suspicious to me, in that Python 3.11 maybe isn't able to dynamically load that shared library that's (maybe) only compatible with Python 3.10?

[felixfontein]

Not being quite this level of Python expert, the _cffi_backend.cpython-310-x86_64-linux-gnu.so seems somewhat suspicious to me, in that Python 3.11 maybe isn't able to dynamically load that shared library that's (maybe) only compatible with Python 3.10?

This is likely the cause for your problem. _cffi_backend.cpython-310-x86_64-linux-gnu.so will only be used by Python 3.10, but not by Python 3.11. Since this file is specific to the CPython version, CPU architecture etc., all that information is encoded in the extension so you can have multiple versions present at the same time.

It's kind of strange that Python 3.10 and 3.11 use the same modules directory (/usr/lib/python3/dist-packages in your case). I'm not an expert on Python packaging, but I would expect there to be different directories for every Python version. (I think Fedora is one of the distributions which natively supports having different Python versions installed in parallel, but I don't know how they're doing that, and I don't use it so I cannot simply check ;) )