Configuration with MQTT sink throws "packet_id exhausted error" after few hours of running

[jaxer]

Hey full error below. It is repeated many many times.
I am running two agents both on separate Ubuntu machines (one on AWS EC2 and another in Docker container on windows host). Both exhausted packet_id's at about same time (overnight).

Part of agent.log

2022-12-06T11:53:11.440091Z (0x00007efeebb01640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2022-12-06T11:53:11.440273Z (0x00007efeebb01640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2022-12-06T11:53:11.440300Z (0x00007efeebb01640) [error] : Stream body but no request: Retry last failed request
2022-12-06T11:53:11.440325Z (0x00007efeebb01640) [error] : Error getting response: system Bad file descriptor
2022-12-06T11:53:11.440338Z (0x00007efeebb01640) [error] : read: Retry last failed request

agent.cfg:

AllowPut = true
SchemaVersion = 1.5
Pretty = true
Port = 5222

Adapters {
   F900 {
      SourceDevice = F900
      Url = http://mtconnect.stratasys.com:5000
   }
   f370 {
      SourceDevice = f370
      Url = http://10.14.77.189:5000
   }
   H350 {
      SourceDevice = H350
      Url = http://10.14.77.165:5000
   }
}

Sinks {
    MqttService {
        MqttHost = localhost
        MqttPort = 8883
        MqttTls = true
        MqttCaCert = certs/ca.pem
        MqttCert = certs/cert.pem
        MqttPrivateKey = certs/key.pem
        MqttClientId = Alex_MTConnectAgent_Greengrass

        DeviceTopic = Device/
        AssetTopic = Asset/
        ObservationTopic = Observation/
    }
}

Full logs: logs.zip
Full Devices.xml: Devices.xml

[jaxer]

By quickly googling and looking through the code this mqtt_cpp issue came up: redboltz/mqtt_cpp#540

It seems fixed long ago though. Although it is not clear to me if "acquire_unique_packet_id" should be used without explicit release.

Commenters seem to have conflicting opinion on that.

In cppagent i see it being used, i.e. in here:

cppagent_dev/src/mqtt/mqtt_client_impl.hpp

Line 249 in 61a8537

m_clientId = derived().getClient()->acquire_unique_packet_id();

[wsobel]

I’ll look i into it. In the samples they don’t release the packet ids after use. I’ll check the code. I’m also not sure why we call it clientId and not packetId and why it’s an instance variable instead of a local. A lot was taken from samples. The limit is uint16 so 65536. Not sure about lifecycle and max rates. I remember one of the SpB saying they overloaded the broker and needed to package multiple measurements together. This may be the same issue. Do you have any insights? I’ll look into the mqtt code this afternoon and see what the unique packet id method does. BestW(Sent from mobile)On Dec 6, 2022, at 04:35, Aleksandr Smirnov ***@***.***> wrote: By quickly googling and looking through the code this mqtt_cpp issue came up: redboltz/mqtt_cpp#540 It seems fixed long ago though. Although it is not clear to me if "acquire_unique_packet_id" should be used without explicit release. Commenters seem to have conflicting opinion on that. If cppagent i see it being used, i.e. in here: https://github.com/mtconnect/cppagent_dev/blob/61a853793a9509146a5c8628312eefa89879b6c0/src/mqtt/mqtt_client_impl.hpp#L249 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jaxer]

Got another occurrence of that. What's interesting is that its actually happening pretty quickly (after about 40 minutes of running). And first error is not about packet_id but something else that gets followed by exhaust of packet_ids. See part of log below, thats about the time packet_id's errors started:

2022-12-07T18:19:57.058450Z (0x00007f7007fff640) [info] : MQTT ConnAck: MQTT Connected
2022-12-07T18:23:49.052597Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:23:49.052655Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:23:49.052750Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:44:56.078750Z (0x00007f7007fff640) [error] : Error getting response: beast.http body limit exceeded
2022-12-07T18:44:56.078925Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:45:06.086513Z (0x00007f7007fff640) [error] : Received protocol error: OUT_OF_RANGE 'from' must be greater than or equal to 36957090.
2022-12-07T18:45:06.086700Z (0x00007f7007fff640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2022-12-07T18:45:06.086815Z (0x00007f7007fff640) [error] : Error occurred processing data: The data stream needs to restart
2022-12-07T18:45:16.094591Z (0x00007f7007fff640) [error] : Received protocol error: UNSUPPORTED The following path is invalid: /H350/assets
2022-12-07T18:45:16.094705Z (0x00007f7007fff640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2022-12-07T18:45:16.094834Z (0x00007f7007fff640) [error] : Error occurred processing data: The data stream needs to restart
2022-12-07T18:45:16.100658Z (0x00007f7007fff640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:activeAlarms
2022-12-07T18:45:16.100730Z (0x00007f7007fff640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_chg
2022-12-07T18:45:16.100742Z (0x00007f7007fff640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_rem
2022-12-07T18:45:16.102495Z (0x00007f7007fff640) [error] : Error getting request header: beast.http end of stream
2022-12-07T18:45:16.102631Z (0x00007f7007fff640) [error] : header: Retry last failed request
2022-12-07T18:45:27.983920Z (0x00007f7007fff640) [error] : Error getting response: beast.http body limit exceeded
2022-12-07T18:45:27.984007Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:45:38.466486Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:45:38.466529Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:45:38.467687Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:50:47.482037Z (0x00007f7007fff640) [error] : Error getting response: beast.http body limit exceeded
2022-12-07T18:50:47.482258Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:50:57.982034Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:50:57.982074Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:50:57.983228Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:52:07.467340Z (0x00007f7007fff640) [error] : Error getting response: beast.http body limit exceeded
2022-12-07T18:52:07.467464Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:52:17.890404Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:52:17.890451Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:52:17.891746Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:52:24.106991Z (0x00007f7007fff640) [error] : Error getting response: beast.http body limit exceeded
2022-12-07T18:52:24.107442Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:52:34.354029Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:52:34.354072Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:52:34.355301Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:52:37.018117Z (0x00007f7007fff640) [error] : Error getting response: beast.http body limit exceeded
2022-12-07T18:52:37.018363Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:52:47.558852Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:52:47.558895Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:52:47.561044Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:52:53.273330Z (0x00007f7007fff640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2022-12-07T18:52:53.273459Z (0x00007f7007fff640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2022-12-07T18:52:53.276823Z (0x00007f7007fff640) [error] : Error getting response: system Bad file descriptor
2022-12-07T18:52:53.276846Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:53:03.520595Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:53:03.520638Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:53:03.521804Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:53:05.218692Z (0x00007f7007fff640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2022-12-07T18:53:05.218796Z (0x00007f7007fff640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2022-12-07T18:53:05.218848Z (0x00007f7007fff640) [error] : Error getting response: system Bad file descriptor
2022-12-07T18:53:05.218861Z (0x00007f7007fff640) [error] : read: Retry last failed request

Makes me wonder if actual reason is in Error getting response: beast.http body limit exceeded which triggers resubmit of same data and saturates some queue?

[jaxer]

Another observation is that when i am starting agent then CPU is around 20%, memory on 37%. And it keeps around those values during normal operation (for 40~ minutes). After that CPU spikes to 300-400% (and stays there until restart) but memory stays the same. Its about time packet_id errors starting to come into agent.log.

Hard to say which comes first.

Restarting agent process resets everything back to normal.

UPDATE: Now i am seeing those packet_id errors with CPU very low

[jaxer]

Unfortunately #242 didn't appear to have solved the issue. I've rebuilt agent and running under similar conditions and getting same results.
Here is a part of log between connection until first failure (after that it just repeats same 4 lines over and over):

2023-01-06T12:39:45.619365Z (0x00007f1da2fd0640) [warning] : Device: lffs9487: Cannot data item for id and no name:dev_asset_rem
2023-01-06T12:39:45.619422Z (0x00007f1da2fd0640) [warning] : Device: lffs9487: Cannot data item for id and no name:lightTowerState
2023-01-06T12:39:45.619922Z (0x00007f1da2fd0640) [warning] : Device: lffs9487: Cannot data item for id submissionTime or name:SubmissionTime
2023-01-06T12:39:45.621971Z (0x00007f1da2fd0640) [error] : Error getting request header: beast.http end of stream
2023-01-06T12:39:45.622076Z (0x00007f1da2fd0640) [error] : header: Retry last failed request
2023-01-06T12:39:45.786349Z (0x00007f1da2fd0640) [info] : MqttClientImpl::connect: connected
**2023-01-06T12:39:46.361625Z (0x00007f1da2fd0640) [info] : MQTT ConnAck: MQTT Connected**
2023-01-06T12:55:44.156401Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http partial message
2023-01-06T12:55:44.156524Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:04:50.443161Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:04:50.443381Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:05:00.450302Z (0x00007f1da2fd0640) [error] : Received protocol error: OUT_OF_RANGE 'from' must be greater than or equal to 28665110.
2023-01-06T13:05:00.450490Z (0x00007f1da2fd0640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2023-01-06T13:05:00.450562Z (0x00007f1da2fd0640) [error] : Error occurred processing data: The data stream needs to restart
2023-01-06T13:05:10.458148Z (0x00007f1da2fd0640) [error] : Received protocol error: UNSUPPORTED The following path is invalid: /H350/assets
2023-01-06T13:05:10.458293Z (0x00007f1da2fd0640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2023-01-06T13:05:10.458452Z (0x00007f1da2fd0640) [error] : Error occurred processing data: The data stream needs to restart
2023-01-06T13:05:10.463744Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:activeAlarms
2023-01-06T13:05:10.463931Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_chg
2023-01-06T13:05:10.463965Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_rem
2023-01-06T13:05:10.466672Z (0x00007f1da2fd0640) [error] : Error getting request header: beast.http end of stream
2023-01-06T13:05:10.466796Z (0x00007f1da2fd0640) [error] : header: Retry last failed request
2023-01-06T13:25:23.828245Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:25:23.828464Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:30:15.731961Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:30:15.732126Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:30:25.739372Z (0x00007f1da2fd0640) [error] : Received protocol error: OUT_OF_RANGE 'from' must be greater than or equal to 28693009.
2023-01-06T13:30:25.739459Z (0x00007f1da2fd0640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2023-01-06T13:30:25.739573Z (0x00007f1da2fd0640) [error] : Error occurred processing data: The data stream needs to restart
2023-01-06T13:30:35.747288Z (0x00007f1da2fd0640) [error] : Received protocol error: UNSUPPORTED The following path is invalid: /H350/assets
2023-01-06T13:30:35.747375Z (0x00007f1da2fd0640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2023-01-06T13:30:35.747486Z (0x00007f1da2fd0640) [error] : Error occurred processing data: The data stream needs to restart
2023-01-06T13:30:35.754383Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:activeAlarms
2023-01-06T13:30:35.754477Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_chg
2023-01-06T13:30:35.754490Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_rem
2023-01-06T13:30:35.757392Z (0x00007f1da2fd0640) [error] : Error getting request header: beast.http end of stream
2023-01-06T13:30:35.757471Z (0x00007f1da2fd0640) [error] : header: Retry last failed request
2023-01-06T13:43:40.363907Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:43:40.364084Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:50:47.868802Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:50:47.868950Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:51:06.317723Z (0x00007f1da2fd0640) [warning] : Device: lffs9487: Cannot data item for id and no name:lightTowerState
2023-01-06T13:51:09.329672Z (0x00007f1da2fd0640) [warning] : Device: lffs9487: Cannot data item for id submissionTime or name:SubmissionTime
2023-01-06T13:53:39.749963Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:53:39.750133Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:54:51.062896Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:54:51.063012Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:55:19.529316Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:55:19.529426Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:55:33.425201Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:55:33.425436Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:55:41.031630Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:55:41.031797Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:55:47.220647Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:55:47.220796Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:55:51.036617Z (0x00007f1da2fd0640) [error] : Received protocol error: OUT_OF_RANGE 'from' must be greater than or equal to 28720907.
2023-01-06T13:55:51.036749Z (0x00007f1da2fd0640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2023-01-06T13:55:51.036835Z (0x00007f1da2fd0640) [error] : Error occurred processing data: The data stream needs to restart
2023-01-06T13:55:59.766996Z (0x00007f1da2fd0640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2023-01-06T13:55:59.767379Z (0x00007f1da2fd0640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2023-01-06T13:55:59.767414Z (0x00007f1da2fd0640) [error] : Stream body but no request: Retry last failed request
2023-01-06T13:55:59.767456Z (0x00007f1da2fd0640) [error] : Error getting response: system Bad file descriptor
2023-01-06T13:55:59.767498Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:56:01.041568Z (0x00007f1da2fd0640) [error] : Received protocol error: UNSUPPORTED The following path is invalid: /H350/assets
2023-01-06T13:56:01.041961Z (0x00007f1da2fd0640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2023-01-06T13:56:01.042099Z (0x00007f1da2fd0640) [error] : Error occurred processing data: The data stream needs to restart
2023-01-06T13:56:01.044670Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:activeAlarms
2023-01-06T13:56:01.044831Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_chg
2023-01-06T13:56:01.044878Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_rem
2023-01-06T13:56:01.049061Z (0x00007f1da2fd0640) [error] : Error getting request header: beast.http end of stream
2023-01-06T13:56:01.049282Z (0x00007f1da2fd0640) [error] : header: Retry last failed request
2023-01-06T13:56:11.692088Z (0x00007f1da2fd0640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2023-01-06T13:56:11.692467Z (0x00007f1da2fd0640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2023-01-06T13:56:11.692548Z (0x00007f1da2fd0640) [error] : Stream body but no request: Retry last failed request
2023-01-06T13:56:11.693061Z (0x00007f1da2fd0640) [error] : Error getting response: system Bad file descriptor
2023-01-06T13:56:11.693120Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:56:23.666464Z (0x00007f1da2fd0640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2023-01-06T13:56:23.667046Z (0x00007f1da2fd0640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2023-01-06T13:56:23.667084Z (0x00007f1da2fd0640) [error] : Stream body but no request: Retry last failed request
2023-01-06T13:56:23.667135Z (0x00007f1da2fd0640) [error] : Error getting response: system Bad file descriptor
2023-01-06T13:56:23.667171Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:56:23.669000Z (0x00007f1da2fd0640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2023-01-06T13:56:23.669212Z (0x00007f1da2fd0640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2023-01-06T13:56:23.669305Z (0x00007f1da2fd0640) [error] : Error getting response: system Bad file descriptor
2023-01-06T13:56:23.669324Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:56:35.560654Z (0x00007f1da2fd0640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2023-01-06T13:56:35.560964Z (0x00007f1da2fd0640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2023-01-06T13:56:35.561012Z (0x00007f1da2fd0640) [error] : Stream body but no request: Retry last failed request
2023-01-06T13:56:35.564180Z (0x00007f1da2fd0640) [error] : Error getting response: system Bad file descriptor
2023-01-06T13:56:35.564295Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:56:47.509094Z (0x00007f1da2fd0640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2023-01-06T13:56:47.509411Z (0x00007f1da2fd0640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2023-01-06T13:56:47.509453Z (0x00007f1da2fd0640) [error] : Stream body but no request: Retry last failed request
2023-01-06T13:56:47.509868Z (0x00007f1da2fd0640) [error] : Error getting response: system Bad file descriptor
2023-01-06T13:56:47.509918Z (0x00007f1da2fd0640) [error] : read: Retry last failed request

[MRIIOT]

@jaxer what broker are you running this against? Can you share its configuration?
Also, are you sending with QoS 0, 1, 2?

[jaxer]

@MRIIOT I was trying AWS IoT Core and MQTT broker inside AWS Greengrass (based on Moquette)

I was not changing any configuration of those so i guess QoS should be 0

[wsobel]

I remember running into this issue before. The problem had to do with the data rates to the broker and overloading the broker before it could free up another another free packet id. If we reduce the QoS we may lose data. The guys from SpB mentioned this as a problem they had, which is why the started combining multiple observations into a singe message.

There are a few solutions. We could do something similar to what SpB did, but then we have issues with retention and we need a way to get the current state (SpB does this put publishing a "REBIRTH" from the subscriber)

If this is what is happening and greengrass can't offload fast enough, then we may need to do something.

Another option is to publish a current to one topic periodically and then deltas to another topic. The streams will sync every so often and an application can figure it out by checking sequence.

What do you think?

[MRIIOT]

Can you try it against a local Mosquitto instance, log everything, and share the broker logs?