community.general.dconf does not find psutil from virtualenv

[kgraefe]

Summary

I use a Python virtualenv to install ansible and ansible-lint. Now I want to use the community.general.dconf role which requires psutil. So I did pip install psutil but the role does not find it.

Installing python3-psutils from my distro's package manager resolves the issue, but I'd like to avoid polluting my work machine.

The reason might be the the shebang in plugins/modules/dconf.py being #!/usr/bin/python rather than #!/usr/bin/env python.

Issue Type

Bug Report

Component Name

dconf

Ansible Version

$ ansible --version
ansible [core 2.18.4]
  config file = /home/kvite/ansible/ansible.cfg
  configured module search path = ['/home/kvite/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/kvite/ansible/venv/lib/python3.13/site-packages/ansible
  ansible collection location = /home/kvite/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/kvite/ansible/venv/bin/ansible
  python version = 3.13.2 (main, Feb  4 2025, 00:00:00) [GCC 14.2.1 20250110 (Red Hat 14.2.1-7)] (/home/kvite/ansible/venv/bin/python)
  jinja version = 3.1.6
  libyaml = True

Community.general Version

$ ansible-galaxy collection list community.general

# /home/kvite/ansible/venv/lib/python3.13/site-packages/ansible_collections
Collection        Version
----------------- -------
community.general 10.5.0 

Configuration

$ ansible-config dump --only-changed
CACHE_PLUGIN(/home/kvite/ansible/ansible.cfg) = jsonfile
CACHE_PLUGIN_CONNECTION(/home/kvite/ansible/ansible.cfg) = .facts
CONFIG_FILE() = /home/kvite/ansible/ansible.cfg
DEFAULT_BECOME_METHOD(/home/kvite/ansible/ansible.cfg) = su
DEFAULT_GATHERING(/home/kvite/ansible/ansible.cfg) = smart
DEFAULT_HOST_LIST(/home/kvite/ansible/ansible.cfg) = ['/home/kvite/ansible/hosts']
DEFAULT_VAULT_PASSWORD_FILE(/home/kvite/ansible/ansible.cfg) = /home/kvite/ansible/vault-password
EDITOR(env: EDITOR) = nvim
INTERPRETER_PYTHON(/home/kvite/ansible/ansible.cfg) = auto_silent

GALAXY_SERVERS:

OS / Environment

Fedora 41 Workstation

Steps to Reproduce

- name: Set dark mode
  community.general.dconf:
    key: /org/gnome/desktop/interface/color-scheme
    value: "'prefer-dark'"

Expected Results

the dconf plugin should work

Actual Results

TASK [gnome : Set dark mode] ****************************************************************************************************************
task path: /home/kvite/ansible/roles/gnome/tasks/main.yml:2
<hermos> ESTABLISH LOCAL CONNECTION FOR USER: kvite
<hermos> EXEC /bin/sh -c 'echo ~kvite && sleep 0'
<hermos> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/kvite/.ansible/tmp `"&& mkdir "` echo /home/kvite/.ansible/tmp/ansible-tmp-1744035082.4922552-50110-246453920777471 `" && echo ansible-tmp-1744035082.4922552-50110-246453920777471="` echo /home/kvite/.ansible/tmp/ansible-tmp-1744035082.4922552-50110-246453920777471 `" ) && sleep 0'
Using module file /home/kvite/ansible/venv/lib/python3.13/site-packages/ansible_collections/community/general/plugins/modules/dconf.py
<hermos> PUT /home/kvite/.ansible/tmp/ansible-local-50092679xhrzc/tmpnim_30d9 TO /home/kvite/.ansible/tmp/ansible-tmp-1744035082.4922552-50110-246453920777471/AnsiballZ_dconf.py
<hermos> EXEC /bin/sh -c 'chmod u+x /home/kvite/.ansible/tmp/ansible-tmp-1744035082.4922552-50110-246453920777471/ /home/kvite/.ansible/tmp/ansible-tmp-1744035082.4922552-50110-246453920777471/AnsiballZ_dconf.py && sleep 0'
<hermos> EXEC /bin/sh -c '/home/kvite/ansible/venv/bin/python3.13 /home/kvite/.ansible/tmp/ansible-tmp-1744035082.4922552-50110-246453920777471/AnsiballZ_dconf.py && sleep 0'
<hermos> EXEC /bin/sh -c 'rm -f -r /home/kvite/.ansible/tmp/ansible-tmp-1744035082.4922552-50110-246453920777471/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_community.general.dconf_payload_z7j8irdi/ansible_community.general.dconf_payload.zip/ansible_collections/community/general/plugins/module_utils/deps.py", line 65, in declare
    yield dep
  File "/tmp/ansible_community.general.dconf_payload_z7j8irdi/ansible_community.general.dconf_payload.zip/ansible_collections/community/general/plugins/modules/dconf.py", line 149, in <module>
    import psutil
ModuleNotFoundError: No module named 'psutil'
fatal: [hermos]: FAILED! => 
    changed: false
    invocation:
        module_args:
            key: /org/gnome/desktop/interface/color-scheme
            state: present
            value: '''prefer-dark'''
    msg: Failed to import the required Python library (psutil) on hermos's Python /usr/bin/python3.
        Please read the module documentation and install it in the appropriate location.
        If the required library is installed, but Ansible is using the wrong Python interpreter,
        please consult the documentation on ansible_python_interpreter

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• plugins/modules/dconf.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @azaghal
click here for bot help

[felixfontein]

This is very likely a Ansible usage problem. Modules do not run in the same Python context that ansible-core runs in. You need to explicitly tell Ansible where to run modules, also on localhost. (The error message tells you that Ansible uses /usr/bin/python3, which is very likely not the binary of your virtual environment. You basically have to point Ansible to the python binary of your venv.) If you have trouble figuring out how to get it to work, I'd suggest to ask in the forum.

[kgraefe]

The reason might be the the shebang in plugins/modules/dconf.py being #!/usr/bin/python rather than #!/usr/bin/env python.

This conclusion is probably wrong, as the module is called with the correct interpreter. see this line in the logs above:

EXEC /bin/sh -c '/home/kvite/ansible/venv/bin/python3.13 /home/kvite/.ansible/tmp/ansible-tmp-1744035082.4922552-50110-246453920777471/AnsiballZ_dconf.py && sleep 0'

/home/kvite/ansible/venv/bin/python is a symlink to /usr/bin/python3.

Setting ansible_python_interpreter does not change anything (I tried).

[felixfontein]

The shebang is definitely unrelated. You need to configure ansible-core to run modules in the venv. How exactly that works I don't know (never tried it).

[kgraefe]

so the reason is that I did not have the gi module in my virtualenv. In that case the dconf modules does not error out:

try:
    from gi.repository.GLib import Variant, GError
except ImportError:
    Variant = None
    GError = AttributeError

Instead it tries to detect system interpreters on its own behalf, ignoring the ansible_python_interpreter variable:

    if Variant is None:
        # This interpreter can't see the GLib module. To try to fix that, we'll
        # look in common locations for system-owned interpreters that can see
        # it; if we find one, we'll respawn under it. Otherwise we'll proceed
        # with degraded performance, without the ability to parse GVariants.
        # Later (in a different PR) we'll actually deprecate this degraded
        # performance level and fail with an error if the library can't be
        # found.

        if has_respawned():
            # This shouldn't be possible; short-circuit early if it happens.
            module.fail_json(
                msg="%s must be installed and visible from %s." %
                (glib_module_name, sys.executable))

        interpreters = ['/usr/bin/python3', '/usr/bin/python2',
                        '/usr/bin/python']

        interpreter = probe_interpreters_for_module(
            interpreters, glib_module_name)

        if interpreter:
            # Found the Python bindings; respawn this module under the
            # interpreter where we found them.
            respawn_module(interpreter)
            # This is the end of the line for this process, it will exit here
            # once the respawned module has completed.

The issue is fixed for me by creating the virtualenv with virtualenv venv --system-site-packages so that system modules are present in the virtualenv as well.

Anyway, ignoring ansible_python_interpreter seems like a bug to me which cost me a couple of hours. I don't know how you want to proceed with this. If you don't want to change behaviour I would suggest to at least issue a warning.

[felixfontein]

Ref: ansible/ansible#85037

[russoz]

Personally I think this kind of behaviour should be removed from the module. The vast majority of the modules will try importing and fail if that fails. I don't see why this one should be any different.