[app-mqtt-export] Service stops exporting data after some time without errors

[gnanavel-balamurugan]

[app-mqtt-export] Service stops exporting data after some time without errors

Description

The edgex-app-mqtt-export service intermittently stops exporting data after running for some time. There are no abnormal behaviors observed, such as high CPU or memory usage, and no error logs indicating the cause. The service appears to go into a dormant state or deadlock. The only way to restore functionality is by restarting the container.

Steps to Reproduce

1. Start the edgex-app-mqtt-export service.
2. Allow it to run for an extended period under normal conditions.
3. Observe that after some time, the service stops exporting data without any errors or high resource usage.
4. Restarting the container restores functionality.

Expected Behavior

The service should continuously export data without requiring manual intervention or a container restart.

Observations

• No error logs are generated when the issue occurs.
• CPU and memory usage remain stable.
• Restarting the container resolves the issue temporarily.

Environment

• EdgeX Version: 3.1.0 NAPA
• Deployment Method: Docker Compose,
• OS & Architecture: Ubuntu 22.04, x86_64
• MQTT Broker: EMQX

CustomConfiguartion : configuration.yaml:

Writable:
LogLevel: "DEBUG"
StoreAndForward:
Enabled: true
RetryInterval: "1m"
MaxRetryCount: 4320

Pipeline:
PerTopicPipelines:
M0001:
Id: "M0001"
Topics: "events/#"
ExecutionOrder: "Transform, MQTTExport"

Functions:
  FilterByProfileName:
    Parameters:
      ProfileNames: ""
      FilterOut: "false"
  FilterByDeviceName1:
    Parameters:
      DeviceNames: "M0001"
      FilterOut: "false"
  FilterBySourceName:
    Parameters:
      SourceNames: ""
      FilterOut: "false"
  FilterByResourceName:
    Parameters:
      ResourceNames: ""
      FilterOut: "false"
  Transform:
    Parameters:
      Type: "json"
  Compress:
    Parameters:
      Algorithm: "gzip"
  Encrypt:
    Parameters:
      Algorithm: "aes256"
      SecretName: "aes"
      SecretValueKey: "key"
  Batch:
    Parameters:
      Mode: "bytimecount" # can be "bycount", "bytime" or "bytimecount"
      BatchThreshold: "30"
      TimeInterval: "60s"
      MergeOnSend: "false"
  AddTags:
    Parameters:
      Tags: ""
  MQTTExport:
    Parameters:
      BrokerAddress: "mqtts://$(vb):8884"
      Topic: "{SOURCENAME}"
      SecretName: "mqttsecret"
      ClientId: ""
      QOS: "1"
      AutoReconnect: "true"
      KeepAlive: "0s" # Empty indicates to use default value
      ConnectTimeout: "" # Empty indicates to use default value
      Retain: "false"
      SkipVerify: "true"
      AuthMode: "None"
      PersistOnError: "true"
      WillEnabled: "false"
      WillPayload: ""
      WillQos: "0"
      WillRetained: "false"
      WillTopic: ""

InsecureSecrets:
mqttsecret:
SecretName: "mqttsecret"
SecretData:
username: ""
password: ""
cacert: "cacert"
clientcert: "clientcert"
clientkey: "clientkey"
aes:
SecretName: "aes"
SecretData:
key: "217A24432646294A404E635266556A586E3272357538782F413F442A472D4B6150645367566B59703373367639792442264529482B4D6251655468576D5A7134"

Service:
Host: "localhost"
Port: 59703
StartupMsg: "app-mqtt-export has Started"

MessageBus:
Optional:
ClientId: "app-mqtt-export"

Using default Trigger config from common config

Additional Context

• Are there any particular conditions under which this issue occurs more frequently?
• Have you tried increasing log verbosity to debug the issue?

Logs

[cloudxxx8]

Could you provide the full docker-compose file and sample device config? What's the auto event frequency? How long should we wait it to happen?

[gnanavel-balamurugan]

Thank you @cloudxxx8 for looking into this issue.

Full Docker Compose File

You can find the full docker-compose.yml file here:
docker-compose.yaml.txt

Sample Device Profile

Here is the sample device profile:
modbus_profile.json

Sample Device File

The sample device file can be found here:
modbus_device.json

Auto Event Frequency & Observations

• The auto event frequency is 5s and it subject to change.
• We are using multiple devices from different device services simultaneously to poll and export data.
• There is no specific time after which the issue occurs; however, it happens unpredictably.

Let me know if any additional details are required!

[gnanavel-balamurugan]

Thank you @cloudxxx8 for looking into this issue.

Full Docker Compose File

You can find the full docker-compose.yml file here:
docker-compose.yaml.txt

Sample Device Profile

Here is the sample device profile:
modbus_profile.json

Sample Device File

The sample device file can be found here:
modbus_device.json

Auto Event Frequency & Observations

• The auto event frequency is 5s and it subject to change.
• We are using multiple devices from different device services simultaneously to poll and export data.
• There is no specific time after which the issue occurs; however, it happens unpredictably.

Let me know if any additional details are required!

[lindseysimple]

Hi @gnanavel-balamurugan I've tried with the app service configuration file you provided but was unlucky to reproduce your issue, here are some questions:

1. Could you please enable the log level to DEBUG and provide the full app-mqtt-export service log including the error in your attached log screenshot:

MessageBus Trigger: Failed to process message on pipeline(s): 1 error occurred, publish was broken by the timeout

2. I noticed that in the modbus_profile.json you provided, the device command name is Export/Topic which includes a slash, just to confirm that if you have enabled the EnableNameFieldEscape service configuration to true from common-config?

[gnanavel-balamurugan]

Hi @gnanavel-balamurugan I've tried with the app service configuration file you provided but was unlucky to reproduce your issue, here are some questions:

1. Could you please enable the log level to DEBUG and provide the full app-mqtt-export service log including the error in your attached log screenshot:

MessageBus Trigger: Failed to process message on pipeline(s): 1 error occurred, publish was broken by the timeout

2. I noticed that in the modbus_profile.json you provided, the device command name is Export/Topic which includes a slash, just to confirm that if you have enabled the EnableNameFieldEscape service configuration to true from common-config?

Hi @lindseysimple

Thanks for checking this issue.

1. I haven't enabled EnableNameFieldEscape as true. However, this hasn't caused any issues so far in any of the device services.
2. Even after enabling DEBUG logs, no additional logs were recorded. The service seems to go into a deadlock condition instead.

Let me know if you need any further details to debug this issue.

Thanks!

[lindseysimple]

Hi @gnanavel-balamurugan Could you please provide the full app-mqtt-export service debug log for me to dig further? Thanks.

[gnanavel-balamurugan]

Hi @gnanavel-balamurugan Could you please provide the full app-mqtt-export service debug log for me to dig further? Thanks.

hi [@lindseysimple ] these are logs that printed at the time of deadlock condition

[lindseysimple]

Hi @gnanavel-balamurugan, sorry for the late response. From the error message from app service, it looks like the AddEventRequest payload received on the message bus was incorrect. Could you please check the device-modbus log and see if any errors occurred at the same timestamps?