Mosquitto not working on 64 bit Raspberry Pi OS

[dannystaple]

I've gotten part way through this process and it seems like mosquitto isn't working.

1. Loaded 64 bit Raspberry Pi OS (full) on a new 32 GB SD card
2. Used pyinfra to run deploy_all, deploy_services, deploy_vision, deploy_speaking
3. Restart, but noticed that inventor hat light wasn't coming on
4. Used journalctl to check inventor hat service

doug@raspi4:~ $ journalctl -u inventor_hat_service -f
Oct 12 05:24:46 raspi4 robotpython[3651]:   File "/usr/lib/python3.13/socket.py", line 849, in create_connection
Oct 12 05:24:46 raspi4 robotpython[3651]:     sock.connect(sa)
Oct 12 05:24:46 raspi4 robotpython[3651]:     ~~~~~~~~~~~~^^^^
Oct 12 05:24:46 raspi4 robotpython[3651]: ConnectionRefusedError: [Errno 111] Connection refused
Oct 12 05:24:46 raspi4 systemd[1]: inventor_hat_service.service: Main process exited, code=exited, status=1/FAILURE
Oct 12 05:24:46 raspi4 systemd[1]: inventor_hat_service.service: Failed with result 'exit-code'.
Oct 12 05:24:46 raspi4 systemd[1]: inventor_hat_service.service: Scheduled restart job, restart counter is at 5.
Oct 12 05:24:46 raspi4 systemd[1]: inventor_hat_service.service: Start request repeated too quickly.
Oct 12 05:24:46 raspi4 systemd[1]: inventor_hat_service.service: Failed with result 'exit-code'.
Oct 12 05:24:46 raspi4 systemd[1]: Failed to start inventor_hat_service.service - inventor_hat_service.

5. Check mosquitto

doug@raspi4:~ $ mosquitto_sub -t "hello/#" -u robot -P robot -v
Error: Connection refused
doug@raspi4:~ $ mosquitto_pub -u robot -P robot -t sensors/encoders/control/reset -n
Error: Connection refused

I wasn't able to find any troubleshooting tips for mosquitto. How should I proceed?

Originally posted by @dblanding in #9

[dannystaple]

SD card setup - https://github.com/user-attachments/files/22870922/sd-card4-setup.txt

[dannystaple]

When you ran pyinfra inventory.py deploy/deploy_all.py, the output showed a problem installing mosquitto:

--> Starting operation: Restart/enable mosquitto 
    [raspi4.local] Job for mosquitto.service failed because the control process exited with error code.
    [raspi4.local] See "systemctl status mosquitto.service" and "journalctl -xeu mosquitto.service" for details.
    [raspi4.local] Error: executed 1 commands

@dblanding can you send me the output of those suggested commands, via ssh on the pi:

• systemctl status mosquitto.service
• journalctl -xeu mosquitto.service

[dblanding]

doug@raspi4:~ $ systemctl status mosquitto.service
× mosquitto.service - Mosquitto MQTT Broker
     Loaded: loaded (/usr/lib/systemd/system/mosquitto.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Sun 2025-10-12 14:58:43 EDT; 13min ago
 Invocation: bcb3c4b9d5284ee4ac14c00d261c4230
       Docs: man:mosquitto.conf(5)
             man:mosquitto(8)
    Process: 1597 ExecStartPre=/bin/mkdir -m 740 -p /var/log/mosquitto (code=exited, status=0/SUCCESS)
    Process: 1599 ExecStartPre=/bin/chown mosquitto:mosquitto /var/log/mosquitto (code=exited, status=0/SUCCESS)
    Process: 1601 ExecStartPre=/bin/mkdir -m 740 -p /run/mosquitto (code=exited, status=0/SUCCESS)
    Process: 1606 ExecStartPre=/bin/chown mosquitto:mosquitto /run/mosquitto (code=exited, status=0/SUCCESS)
    Process: 1610 ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf (code=exited, status=13)
   Main PID: 1610 (code=exited, status=13)

Oct 12 14:58:43 raspi4 systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 5.
Oct 12 14:58:43 raspi4 systemd[1]: mosquitto.service: Start request repeated too quickly.
Oct 12 14:58:43 raspi4 systemd[1]: mosquitto.service: Failed with result 'exit-code'.
Oct 12 14:58:43 raspi4 systemd[1]: Failed to start mosquitto.service - Mosquitto MQTT Broker.

doug@raspi4:~ $ journalctl -xeu mosquitto.service
░░ 
░░ An ExecStart= process belonging to unit mosquitto.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 13.
Oct 12 14:58:43 raspi4 systemd[1]: mosquitto.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit mosquitto.service has entered the 'failed' state with result 'exit-code'.
Oct 12 14:58:43 raspi4 systemd[1]: Failed to start mosquitto.service - Mosquitto MQTT Broker.
░░ Subject: A start job for unit mosquitto.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mosquitto.service has finished with a failure.
░░ 
░░ The job identifier is 2828 and the job result is failed.
Oct 12 14:58:43 raspi4 systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 5.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ Automatic restarting of the unit mosquitto.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Oct 12 14:58:43 raspi4 systemd[1]: mosquitto.service: Start request repeated too quickly.
Oct 12 14:58:43 raspi4 systemd[1]: mosquitto.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit mosquitto.service has entered the 'failed' state with result 'exit-code'.
Oct 12 14:58:43 raspi4 systemd[1]: Failed to start mosquitto.service - Mosquitto MQTT Broker.
░░ Subject: A start job for unit mosquitto.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mosquitto.service has finished with a failure.
░░ 
░░ The job identifier is 3164 and the job result is failed.

[dannystaple]

I've just started an install on a similar fresh card (32Gb) with the 64 bit Raspberry Pi OS, and running deploy all.
I've been able to get the same initial error condition:

--> Starting operation: Restart/enable mosquitto 
    [learnrob3-2.local] Job for mosquitto.service failed because the control process exited with error code.
    [learnrob3-2.local] See "systemctl status mosquitto.service" and "journalctl -xeu mosquitto.service" for details.
    [learnrob3-2.local] Error: executed 1 commands

--> Disconnecting from hosts...
--> pyinfra error: No hosts remaining!

I'll probe the Pi to find out what is going on here...

[dannystaple]

So Mosquitto is not logging to journalctl, but now to /var/log/mosquitto/mosquitto.log which is also only readable by root - a little annoying. It can be read over ssh with sudo cat /var/log/mosquitto/mosquitto.log.
There I see the following errors:

1760297053: Error opening password file "/etc/mosquitto/passwd".
1760297054: mosquitto version 2.0.21 starting
1760297054: Config loaded from /etc/mosquitto/mosquitto.conf.
1760297054: Error: Unable to open pwfile "/etc/mosquitto/passwd".
1760297054: Error opening password file "/etc/mosquitto/passwd".

This should have been setup in the deploy script with

if mosquitto_packages.changed:
    # set mosquitto password
    server.shell(
        f"mosquitto_passwd -c -b /etc/mosquitto/passwd {mqtt_username} {mqtt_password}",
        _sudo=True)

I'm yet to determine what happened. The file exists:

root@learnrob3-2:/home/danny# ls -al /etc/mosquitto/passwd 
-rw------- 1 root root 119 Oct 12 20:24 /etc/mosquitto/passwd

It's owned by root.
However, in "Trixie" (the current Raspberry Pi OS/Debian version), mosquitto now runs as a mosquitto user:

ExecStartPre=/bin/chown mosquitto:mosquitto

Running chown mosquitto:mosquitto /etc/mosquitto/passwd as root allowed me to restart the service.

So for trixie, it now seems that a chown is necessary. Working out the pyinfra for it.

[dannystaple]

I'm trying this one after the shell command in deploy/deploy_mqtt.py:

files.file(
    name="Ensure mosquitto password file exists",
    path="/etc/mosquitto/passwd",
    present=True,
    user="mosquitto",
    group="mosquitto",
    _sudo=True
)

[dannystaple]

Yup - that works. @dblanding can you try that deploy_mqtt change out? You probably want to rerun deploy_all since other dpeloy steps would not have been run after that first failure.

[dblanding]

Not sure exactly where that goes... Inside the if clause or after it?

[dannystaple]

Good question, it is after the if clause.

[dblanding]

Yep, that works. So now my inventor_hat service works too.