Setting EdgeX MessageBus to use MQTT Pub/Sub

[neerajasjawali]

Hi all,
I am working on setting the MessageBus configuration for EdgeX (jakarta) to use MQTT Pub/Sub.
I have modified the configuration.toml file of core-data, app-service and device-service. Moqsuitto broker is running, While running the device-service for reading camera data, I am getting following errors on various services terminal.

Error on core-data service:
level=ERROR ts=2023-02-16T04:55:59.428261237Z app=core-data source=subscriber.go:72 msg="fail to persist the event, Event Id exists"

Error on metadata service:
level=INFO ts=2023-02-16T06:23:47.431125067Z app=core-metadata source=message.go:58 msg="Service started in: 31.079106ms" level=ERROR ts=2023-02-16T05:33:47.746644181Z app=device-virtual source=utils.go:71 msg="Failed to publish event to MessageBus: Timeout occured while performing a 'Publish' operation: Unable to publish message"

Error on device-service:
level=ERROR ts=2023-02-16T05:33:47.746644181Z app=device-virtual source=utils.go:71 msg="Failed to publish event to MessageBus: Timeout occured while performing a 'Publish' operation: Unable to publish message"

Error on mosquitto broker:

New connection from 10.168.1.190 on port 1883.
1676525380: New client connected from 10.168.1.190 as core-data (p2, c1, k10).
1676525590: Client core-data has exceeded timeout, disconnecting.
1676525605: New connection from 10.168.1.190 on port 1883.
1676525605: New client connected from 10.168.1.190 as core-data (p2, c1, k10).
1676525629: Client mqtt-app-service has exceeded timeout, disconnecting.

Attached the screenshots of configuration.toml files

It would be helpful, if you could give me suggestions to resolve the issue.
I would also like to know the modifications to be done in mqtt broker configuration.

Thanks in advance

[herrdragon]

Hi, have you looked at this https://docs.edgexfoundry.org/2.1/microservices/application/Triggers/#external-mqtt-trigger
Try using :
[Trigger]
Type="external-mqtt"
[Trigger.ExternalMqtt]
Url = "tcp://edgex-mqtt-broker:1883" # fully qualified URL to connect to the MQTT broker
SubscribeTopics="SomeTopics"...

The way you have is more for redis pub-sub

[neerajasjawali]

Hi @herrdragon
The configuration is likely similar to redis pub/sub. I looked at this https://docs.edgexfoundry.org/2.1/microservices/application/Triggers/. Within which they have given an example for using mqtt, I referred the same and made the changes. As you can see I have written the type as mqtt and modified some changes as below

[Writable.Pipeline.Functions.MQTTExport]                            
      [Writable.Pipeline.Functions.MQTTExport.Parameters]                  
      BrokerAddress = "tcp://edgex-mqtt-broker:1883" # Hive Public Test Broker
      Topic = "edgex-export"                            
      SecretPath = "mqtt"                             
      ClientId = "MQTT-Export"                      
      QOS="0"                                                           
      AutoReconnect="true"                                                 
      KeepAlive = "10m" # Empty indicates to use default value                
      ConnectTimeout = "5m" # Empty indicates to use default value
      Retain="true"                                                        
      SkipVerify = "false"                          
      AuthMode = "none"                                                 
      PersistOnError = "false"  

should I not use mqtt-export and instead use external-mqtt?
If Yes, then why?
I would also like to know if we are using mosquitto broker for pub sub, do I need to modify the mosquitto configuration file and how?
Thanks in advance

[herrdragon]

I can’t tell you in details why, all I know if that I, like you, am using mosquito as a broker and since it is not part of edgex, I use the external-mqtt configuration for the [Trigger] part and then [Trigger.ExternalMqtt] for the broker config and is working fine with no issues. Maybe someone from edgex here can explain why is not working the way you have. Hope this helps you.

[neerajasjawali]

Hi @herrdragon,

Thank you for your suggestion.

[lenny-goodell]

FYI, external-mqtt is only for App Service to receive "trigger" data from an external MQTT broker instead of the internal MessageBus.
It is not applicable to switching the internal MessageBus to use MQTT.

[neerajasjawali]

Hi @lenny-intel
Thank you for explaining the difference.

[lenny-goodell]

@neerajasjawali , are you running in Docker?
If so, the easiest way to switch to using MQTT for the internal MessageBus is to use compose builder to generate a compose file properly configured for this.
https://github.com/edgexfoundry/edgex-compose/blob/jakarta/compose-builder/README.md#gen
The base command is make gen no-secty mqtt-bus <device-service>|<app-service>
See README linked above for all options.

[neerajasjawali]

@lenny-intel, I am running services without Docker

[lenny-goodell]

I am running services without Docker

Why?
There is a reason we use Docker....

[bnevis-i]

You need to do some hostname debugging. When vault is started, what does the "listener" configuration block say to use as the address (the docker version uses edgex-vault:8200, the snap version uses localhost:8200). Also what is your host ip address (ip addr list). What does localhost and edgex-vault resolve to (nslookup localhost, nslookup edgex-vault, nslookup $(hostname))

Listening on hostname or 0.0.0.0 means that it is reachable from a remote machine on the network.

If vault isn't listening on all interfaces (0.0.0.0), then the clients need to be connecting to vault on the same interface that it is listening on (either localhost or edgex-vault). If you're not running docker then I wouldn't recommend using edgex-vault, because any 'ol system on your network can claim that name and suddenly all your vault traffic is going to who-knows-where.

[neerajasjawali]

I am running services without Docker

Why? There is a reason we use Docker....

I am assigned to test EdgeX security without Docker and other teammate is looking into running with Docker.

[neerajasjawali]

You need to do some hostname debugging. When vault is started, what does the "listener" configuration block say to use as the address (the docker version uses edgex-vault:8200, the snap version uses localhost:8200). Also what is your host ip address (ip addr list). What does localhost and edgex-vault resolve to (nslookup localhost, nslookup edgex-vault, nslookup $(hostname))

Listening on hostname or 0.0.0.0 means that it is reachable from a remote machine on the network.

If vault isn't listening on all interfaces (0.0.0.0), then the clients need to be connecting to vault on the same interface that it is listening on (either localhost or edgex-vault). If you're not running docker then I wouldn't recommend using edgex-vault, because any 'ol system on your network can claim that name and suddenly all your vault traffic is going to who-knows-where.

Vault listener block:

listener "tcp" { 
             address = "edgex-vault:8200" 
             tls_disable = "1" 
             cluster_address = "edgex-vault:8201" 
         } 
         backend "file" {
             path = "/home/root/secret_jakarta/edgex_secretstore/vault/file"
         } 
         default_lease_ttl = "168h" 
         max_lease_ttl = "720h"

Within the ip address list we have defined edgex-vault,edgex-core-consul,edgex-redis and so on with the system ip address (10.168.1.0)