not all data captures being decoded

[stepir]

Describe the bug
Hey, I'm running latest version trough HA addon 1.11

I'm using gateway to track TPMS data (cheap Chinese BLE sensors). What I'm noticing is that the data is update very often but only decoded few times.

In example here the same sensor has 3 historical data:

While the decoded topic only one:

What can I look at to gather additional info on why not all the data is decoded? What I'm seeing is that the sensors latest data is not always decoded and values reported in mqtt/ha is hours behind the latest captured.

Thanks

[DigiH]

Hi @stepir

It looks to me that everything is working fine and as expected.

In example here the same sensor has 3 historical data:

The messages received under the home/TheengsGateway/BTtoMQTT/82EACA31D2E3 topic are the constantly received messages from your TPMS expectedly with a changing value for count for which of the four pressure sensors the message is coming from, including its pressure, temperature, battery level and alarm state - these are the decoded messages, and all data is decoded.

While the decoded topic only one:

These are not decoded messages, but the auto-discovery messages for HA, only ever a single one for each property of your TPMS, indicated by their ending subtopic with the MAC address and -property of the device. These allow the automatic discovery in HA and for you to add the device easily.

I hope this clarifies what you are seeing in MQTT Explorer.

[stepir]

Hi @DigiH

thanks a lot for helping understand better and sorry for some wrong wording used.

The auto-discovery messages have older values than the decoded messages. If you look my screenshots 3 newer values have been decoded but the auto-discovery (second screenshot) one is not updated.

The behaviour I'm seeing is that if I use auto-discovered entities in HA the values reported are hours behind, while the TPMS is updating its values every x minutes and so is the decoded messages in home/TheengsGateway/BTtoMQTT

[DigiH]

The auto-discovery messages have older values than the decoded messages. If you look my screenshots 3 newer values have been decoded but the auto-discovery (second screenshot) one is not updated.

Yes, that is correct, the auto-discovery messages for each property are being created only once, the very first time a new, not yet discovered devices is being received by Theengs Gateway. Only this one single message per property is required for HA as a discovery message. They do not contain any other up to date sensor information at all, only the discovery details. Depending on when you discovered a device through Theengs Gateway, or any other MQTT discovery enabled gateway for that matter, they can be weeks, months or even years old - like some of mine for example.
You should not really be concerned with these auto-discovery messages unless something is not working correctly with the discovery in HA.

I assume you also have the MQTT integration installed in HA and have it hooked up to your MQTT Broker, correct?

If so, you should see one entry under the MQTT integration for your TMPS. Clicking on it you should see the five properties for the device - these device and properties entries are created by the discovery messages, whereas the every changing actual values you see in this preview, or when adding this to any of your dashboards now - these are created by the constant incoming messages with up to date values.

Maybe you can post a screenshot of this when you have set it up like that.

[stepir]

Yes it's like what you describe except for the update frequencies of the values in the autodiscovery case

see here my mqtt integration

here the pressure value is reported to be updated 2hours ago:

but If I look at the same data from the raw decoded message:

then I have a more recent value of 1 hour ago:

When driving and the TPMS values are updated more frequently, the values in the MQTT integration are still hours behind - while the one under home/TheengsGateway/BTtoMQTT/ much more recent.

[DigiH]

Could you look into MQTT INFO just under the Device Info of one of the TPMS entries, where you selected the pressure property history above?

It shows the discovery Payload for each property, but also the 10 most recently received message(s) to be expanded.

Could you post a screenshot with both these expanded for -pres?

I am a bit confused by you also having Ruuvi Tags showing in your MQTT discovered devices, but they are not actually showing as retained discovery messages in MQTT Explorer!?! So is everything hooked up to the same MQTT broker?

And could it just be that the properties haven't actually changed their value for the last 2 hours, like the pressure in your screenshot above? So even if messages do come in regularly, the actual value(s) not changing will show what you are seeing in the pressure screenshot with 2 hours.

What do you get as a popup when you hover over the 2 hours ago?

[stepir]

Could you look into MQTT INFO just under the Device Info of one of the TPMS entries, where you selected the pressure property history above?

It shows the discovery Payload for each property, but also the 10 most recently received message(s) to be expanded.

Could you post a screenshot with both these expanded for -pres?

I need to await some updates now - unfortunately they are both synced with the same last value (and entering a conf.call in 5 mins)

I am a bit confused by you also having Ruuvi Tags showing in your MQTT discovered devices, but they are not actually showing as retained discovery messages in MQTT Explorer!?! So is everything hooked up to the same MQTT broker?

Yes everything showing up on the same broker:

but there is multiple things not properly working I believe. The first is that even if 'discovery' is disabled I get homeassistant/sensor populated anyway. Is this correct? or i misunderstood the documentation?

The other one in some cases I have a message only in the discovery topic but not under home/TheengsGateway/BTtoMQTT/

And could it just be that the properties haven't actually changed their value for the last 2 hours, like the pressure in your screenshot above? So even if messages do come in regularly, the actual value(s) not changing will show what you are seeing in the pressure screenshot with 2 hours.

What do you get as a popup when you hover over the 2 hours ago?

No, now that the vehicle is parked is more difficult to reproduce but when moving the sensors sends updated every few seconds. I can see it with their own IOS app and under home/TheengsGateway/BTtoMQTT/ I have many newer values of pressure (e.g.) while the autodiscvered MQTT sensor is still showing values of hours ago (different value)

[DigiH]

I have never actually looked into this much until you posted this issues now ;) but to me it looks like some HA idiosyncrasy/bug with the property history time stamp only showing the last time the value was updated, not received.so far so good, if that is its designed functionality. BUT hovering over it and seeing the different Last changed: and Last updated:, at least for me here, they are the same time stamps, where I would at least expect Last updated: to be from the last message received, as can be confirmed when looking at the 10 most recently received message(s) under MQTT INFO.

I can also see this with my Inkbird TH1 sensors, where the temperature, humidity and battery level are showing their regularly changing value updated timestamp

BUT the binary_sensor for the external probe being connected or not - which hasn't changed, shows

which doesn't really make sense for Last updated: 2 hours ago either, as the same property came in 1 second ago with the same MQTT message the battery status was updated.

This is with HA Core 2024.3.3, and definitely an issue with HA.

Out of curiosity, what does the tempc history look for you, I assume that at least the temperature would change and therefor be updated more frequently.

And did you have a look at the 10 most recently received message(s) under MQTT INFO for your TPMSs?

[DigiH]

but there is multiple things not properly working I believe. The first is that even if 'discovery' is disabled I get homeassistant/sensor populated anyway. Is this correct? or i misunderstood the documentation?

Populated anew, or you keep the retianed discovery messages which were populated when discovery was on?

The latter is the expected and actually suggested behaviour - You turn on discovery until all your own devices are being discovered and the discovery messages are created under the homeassistant topic. Once you have your own devices discovered you turn off discovery. This is so that you won't have any unwanted devices discovered by accident which might just pass your house or from your neighbours. Whenever you get new compatible devices you turn off discovery again for a short time until they are all discovered and the discovery messages created, to then turn it off again.

The other one in some cases I have a message only in the discovery topic but not under home/TheengsGateway/BTtoMQTT/

Where this could have been a case like that, someone passing your house with a Theengs Gateway compatible Bluetooth device. When discovery is on it will create a discovery entry, but by the time you might look at BTtoMQTT the person has already passed your house and is no longer in Bluetooth range.

Do you have an example of such an unknown discovery message?

[stepir]

but there is multiple things not properly working I believe. The first is that even if 'discovery' is disabled I get homeassistant/sensor populated anyway. Is this correct? or i misunderstood the documentation?

Populated anew, or you keep the retianed discovery messages which were populated when discovery was on?

The latter is the expected and actually suggested behaviour - You turn on discovery until all your own devices are being discovered and the discovery messages are created under the homeassistant topic. Once you have your own devices discovered you turn off discovery. This is so that you won't have any unwanted devices discovered by accident which might just pass your house or from your neighbours. Whenever you get new compatible devices you turn off discovery again for a short time until they are all discovered and the discovery messages created, to then turn it off again.

The other one in some cases I have a message only in the discovery topic but not under home/TheengsGateway/BTtoMQTT/

Where this could have been a case like that, someone passing your house with a Theengs Gateway compatible Bluetooth device. When discovery is on it will create a discovery entry, but by the time you might look at BTtoMQTT the person has already passed your house and is no longer in Bluetooth range.

Do you have an example of such an unknown discovery message?

ok clear very clear. I'll look for this occurrence but let's focus here on the issue of values not updated

[stepir]

I have never actually looked into this much until you posted this issues now ;) but to me it looks like some HA idiosyncrasy/bug with the property history time stamp only showing the last time the value was updated, not received.so far so good, if that is its designed functionality. BUT hovering over it and seeing the different Last changed: and Last updated:, at least for me here, they are the same time stamps, where I would at least expect Last updated: to be from the last message received, as can be confirmed when looking at the 10 most recently received message(s) under MQTT INFO.

I can also see this with my Inkbird TH1 sensors, where the temperature, humidity and battery level are showing their regularly changing value updated timestamp

**BUT** the binary_sensor for the external probe being connected or not - which hasn't changed, shows which doesn't really make sense for **Last updated: 2 hours ago** either, as the same property came in 1 second ago with the same MQTT message the battery status was updated.

This is with HA Core 2024.3.3, and definitely an issue with HA.

Out of curiosity, what does the tempc history look for you, I assume that at least the temperature would change and therefor be updated more frequently.

temp and pressure are updated at the same time - then the value itself might be unchanged

And did you have a look at the 10 most recently received message(s) under MQTT INFO for your TPMSs?

here is the raw data

and here the discovered one (screenshots taken at the same time)

can you see that the first one is updated at 16:17PM

while the second only at 16:13PM?

[stepir]

And did you have a look at the 10 most recently received message(s) under MQTT INFO for your TPMSs?

this one you mean?

[DigiH]

while the second only at 16:13PM?

This again is a discovery message, it is completely irrelevant for the actual received values of when it was created. There is no value at looking at the discovery message time stamps ;)

• These are only discovery messages created once to tell HA where and how to look for the regularly updated messages from the MQTT broker.
• They do not have anything to do with any regularly updated received values themselves, and their time stamp is completely irrelevant.
• It's actually best if you collapse the whole homeassistant topic, and only expand it if there are discovery issues in HA with any of your devices.

10 most recently received message(s) under MQTT INFO

The screenshot above is not for what I meant. Go further left on the same screen as the screenshot, to the Device info section, which has a blue MQTT INFO button/link at the bottom.

Click on MQTT INFO and then expand the 10 most recently received message(s) for any or all the properties. There you will see when the last actual MQTT messages for these properties have been received. Something which does not show up properly under the hover Last updated: info in the history of the actual properties.

[stepir]

ok this one

so you are saying that even if a newer message is received the value is not updated because there was not actually a real change in the value itself?

What if I test it while driving and the values of the sensors are not updated if not in hours or many minutes?

[DigiH]

so you are saying that even if a newer message is received the value is not updated because there was not actually a real change in the value itself?

That is correct, if the most 10 most recently received message(s) match with the last messages received under home/TheengsGateway/BTtoMQTT/82EACA31D2E3 then all is fine, and I see it as a HA bug that in the hover Last updated: this is not being shown correctly, but that it is showing the same value as Last changed:, whereas for the raw data history Last updated: actually seems to working as expected.

How does this all look for the tempc property?

What if I test it while driving and the values of the sensors are not updated if not in hours or many minutes?

I would assume that you'd be leaving the Bluetooth reception range when driving away, so then the last received value would stay stuck as long as the car is away, only updating again once the car and its TPMS sensors are back in Bluetooth range.

[DigiH]

What if I test it while driving and the values of the sensors are not updated if not in hours or many minutes?

And for on the road checking of your TPMS status with your mobile phone, you might want to look at our Theengs Mobile App, which also displays the pressure when away from home ;)