mqttwarn does not like py39-paho-mqtt-2.0.0

[dlangille]

I've been unable to get mqttwarn to start today after updating py39-paho-mqtt: 1.6.1 -> 2.0.0

Reverting to 1.6.1 gets it running again.

Seem related to https://stackoverflow.com/questions/77984857/paho-mqtt-unsupported-callback-api-version-error?noredirect=1

[12:20 mqtt01 dvl /usr/local/etc/rc.d] % sudo /usr/local/bin/mqttwarn --config-file=/usr/local/etc/mqttwarn/mqttwarn.ini
2024-03-02 12:20:58,853 INFO     [mqttwarn.commands         ] Starting mqttwarn
2024-03-02 12:20:58,853 INFO     [mqttwarn.commands         ] Log level is DEBUG
2024-03-02 12:20:58,853 DEBUG    [mqttwarn.core             ] Trying to load built-in service "file" from "file"
2024-03-02 12:20:58,854 DEBUG    [mqttwarn.core             ] Trying to load service "file" from file "/usr/local/lib/python3.9/site-packages/mqttwarn/services/file.py"
2024-03-02 12:20:58,855 INFO     [mqttwarn.core             ] Successfully loaded service "file"
2024-03-02 12:20:58,855 DEBUG    [mqttwarn.core             ] Trying to load built-in service "pushover" from "pushover"
2024-03-02 12:20:58,855 DEBUG    [mqttwarn.core             ] Trying to load service "pushover" from file "/usr/local/lib/python3.9/site-packages/mqttwarn/services/pushover.py"
2024-03-02 12:20:58,985 INFO     [mqttwarn.core             ] Successfully loaded service "pushover"
2024-03-02 12:20:58,986 DEBUG    [mqttwarn.core             ] Trying to load built-in service "log" from "log"
2024-03-02 12:20:58,986 DEBUG    [mqttwarn.core             ] Trying to load service "log" from file "/usr/local/lib/python3.9/site-packages/mqttwarn/services/log.py"
2024-03-02 12:20:58,986 INFO     [mqttwarn.core             ] Successfully loaded service "log"
Traceback (most recent call last):
  File "/usr/local/bin/mqttwarn", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/commands.py", line 93, in run
    run_mqttwarn(configfile=options["--config-file"])
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/commands.py", line 155, in run_mqttwarn
    subscribe_forever()
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/core.py", line 688, in subscribe_forever
    mqttc = connect()
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/core.py", line 642, in connect
    mqttc = paho.Client(cf.clientid, clean_session=cf.cleansession, protocol=cf.protocol)
  File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 766, in __init__
    raise ValueError(
[12:20 mqtt01 dvl /usr/local/etc/rc.d] % sudo /usr/local/bin/mqttwarn --config-file=/usr/local/etc/mqttwarn/mqttwarn.ini
2024-03-02 12:20:58,853 INFO     [mqttwarn.commands         ] Starting mqttwarn
2024-03-02 12:20:58,853 INFO     [mqttwarn.commands         ] Log level is DEBUG
2024-03-02 12:20:58,853 DEBUG    [mqttwarn.core             ] Trying to load built-in service "file" from "file"
2024-03-02 12:20:58,854 DEBUG    [mqttwarn.core             ] Trying to load service "file" from file "/usr/local/lib/python3.9/site-packages/mqttwarn/services/file.py"
2024-03-02 12:20:58,855 INFO     [mqttwarn.core             ] Successfully loaded service "file"
2024-03-02 12:20:58,855 DEBUG    [mqttwarn.core             ] Trying to load built-in service "pushover" from "pushover"
2024-03-02 12:20:58,855 DEBUG    [mqttwarn.core             ] Trying to load service "pushover" from file "/usr/local/lib/python3.9/site-packages/mqttwarn/services/pushover.py"
2024-03-02 12:20:58,985 INFO     [mqttwarn.core             ] Successfully loaded service "pushover"
2024-03-02 12:20:58,986 DEBUG    [mqttwarn.core             ] Trying to load built-in service "log" from "log"
2024-03-02 12:20:58,986 DEBUG    [mqttwarn.core             ] Trying to load service "log" from file "/usr/local/lib/python3.9/site-packages/mqttwarn/services/log.py"
2024-03-02 12:20:58,986 INFO     [mqttwarn.core             ] Successfully loaded service "log"
Traceback (most recent call last):
  File "/usr/local/bin/mqttwarn", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/commands.py", line 93, in run
    run_mqttwarn(configfile=options["--config-file"])
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/commands.py", line 155, in run_mqttwarn
    subscribe_forever()
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/core.py", line 688, in subscribe_forever
    mqttc = connect()
  File "/usr/local/lib/python3.9/site-packages/mqttwarn/core.py", line 642, in connect
    mqttc = paho.Client(cf.clientid, clean_session=cf.cleansession, protocol=cf.protocol)
  File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 766, in __init__
    raise ValueError(
ValueError: Unsupported callback API version: version 2.0 added a callback_api_version, see migrations.md for details

[jpmens]

To be expected; paho mqtt has broken most programs with their 2.0; quite sad. I don’t think we’ve yet added support for it.

[adequacy1]

core.py:

- mqttc = paho.Client(cf.clientid, clean_session=cf.cleansession, protocol=cf.protocol)
+ mqttc = paho.Client(paho.CallbackAPIVersion.VERSION1, cf.clientid, clean_session=cf.cleansession, protocol=cf.protocol)

[jpmens]

Yes, Paho broke that very badly, and it affects hundreds of peoples' programs. If you care to produce a PR we'll take it.

[amotl]

omg. Thanks for letting us know, and apologies for the late reply. We will take corresponding steps to remedy the problem in one way or another.

Paho™ MQTT Python v2.0.0, released on Feb 10 2024, includes breaking changes. See migration instructions for more details on how to upgrade. The release also introduces new documentation available online.

-- https://github.com/eclipse/paho.mqtt.python/releases/tag/v2.0.0

[amotl]

At least the dependency version pinning in project metadata is right. So, when package managers are taking them into account, nothing should break from one day to the other accidentally?

mqttwarn/setup.py

Line 21 in e096289

"paho-mqtt<2",

NB: I am just trying to assess the seriousness and impact of the issue. @dlangille: Did you upgrade py39-paho-mqtt to the next major version deliberately, or did it happen automatically? Is it possible for BSD to have both versions at the same time, or are you currently destined to exclusively run the previous version of paho mqtt, because of this situation, while you actually would like to upgrade?

[amotl]

@dlangille: Relaying the error you reported....

ValueError: Unsupported callback API version: version 2.0 added a callback_api_version, see migrations.md for details

... it probably relates to the topmost item from the v2.0.0 release notes:

Caution

BREAKING: Added callback_api_version. This breaks ALL users of paho-mqtt's Client class.

See migrations.rst for details on how to upgrade.
tl; dr; add CallbackAPIVersion.VERSION1 to first argument of Client().

Fixing this sounds easy.

[amotl]

Thoughts

• There might be issues with 2.0.0 yet, for example AttributeError: ‘MQTTClient’ object has no attribute ‘_sock’, see below.
• Not sure if we should better wait upgrading until a 2.0.3 version will be released? Maybe wait for 2.0.1 at least?
• Sure enough, a patch can be prepared already, that can be used at everyones disposal.

References

• paho-mqtt-2.0.0 client issue eclipse-paho/paho.mqtt.python#814
• Undefined CallbackAPIVersion  eclipse-paho/paho.mqtt.python#822
• Since version 2.0 calling client.reinitialise() raise RuntimeError: Unsupported callback API version eclipse-paho/paho.mqtt.python#824

Downstream

• MQTT client initialization failed nicolargo/glances#2707
• Pin paho-mqtt version < 2.0.0 caronc/apprise#1065
• I just installed with misquitto mqtt on home assistant and get the following error when I try to start blueti-mqtt warhammerkid/bluetti_mqtt#107
• Breaking changes in paho-mqtt version 2.0.0 greenbone/notus-scanner#580
• paho-mqtt pypi package breaking changes greenbone/ospd-openvas#968

[amotl]

I've just submitted a patch to remedy the breaking change on the most prominent spot, based on learnings from mqtt-tools/pytest-mqtt#18.

• Remedy breaking change with 2.x: Use CallbackAPIVersion.VERSION1 as a default eclipse-paho/paho.mqtt.python#831

However, I don't think that patch will be accepted, as the authors seem to pretty strong about this detail?

• paho-mqtt-2.0.0 client issue eclipse-paho/paho.mqtt.python#814 (comment) ff.

[amotl]

Hi. It looks like my patch might get accepted, helping users in basic use case scenarios to upgrade to Paho MQTT client version 2 without a breaking change. Naturally, it will not work in more advanced cases, as outlined by @PierreF on the same discussion. Thanks!

• Remedy breaking change with 2.x: Use CallbackAPIVersion.VERSION1 as a default eclipse-paho/paho.mqtt.python#831

[amotl]

It looks like paho-mqtt v2 is causing serious havoc throughout the community. That is natural with breaking changes, and apparently, much of what is behind is about the proper transitioning to support MQTTv5, which makes things different under the hood.

However, I did not dig deeper, and, in this spirit, I am not able to evaluate the situation better than submitting that humble patch, and hoping for the best outcomes in the long run. That such an event feels annoying, is perfectly reasonable. However, sometimes such changes must happen, and all we can take away from that is that all users of packages should apply version pinning properly, in order not to run into such situations. In this case, paho-mqtt<2.

• paho-mqtt-2.0.0 client issue eclipse-paho/paho.mqtt.python#814

[amotl]

For mqttwarn, I think it is sensible to wait for any further changes happen on paho-mqtt. Because mqttwarn is correctly applying the version pinning, I guess there is not much else to do at this point in time? Please let us know if you think differently.

mqttwarn/setup.py

Line 21 in 9d61fc5

"paho-mqtt<2",