Decoding sensor data

[krconv]

Repurposing this issue for the discussion that it started. Was originally a question about connecting, and turned into a discussion about deciding the sensor data.

---

In the blog post, you mentioned that one of your USB-Serial adaptors wasn't working when you tried to connect to the ESP32. I'm wondering if I'm hitting the same issue; I only have one serial adaptor to test with at the moment though.

I'm using the FT232RL FTDI Mini USB to TTL Serial Converter, and getting nothing while trying to connect:

> esptool.py --port /dev/cu.usbserial-A50285BI read_flash 0 0x400000 flash_contents.bin
esptool.py v2.8
Serial port /dev/cu.usbserial-A50285BI
Connecting........_____....._____....._____....._____....._____....._____....._____

A fatal error occurred: Failed to connect to Espressif device: Timed out waiting for packet header

Process I'm taking to enter flashing mode:

1. Power with 3.3v
2. Pull IO0 pin to GND
3. Pull EN pin to GND, and release
4. Release IO0 pin

But that's not working; my only thought is that maybe there is something else connected with TX/RX that is interfering? And a different serial converter could sort through the noise?

Any guidance would be appreciated!

---

Edit by @flaviut for visibility:

There is now a ESPHome component. You can follow development at

• https://github.com/krconv/esphome/tree/add-emporia-vue-maelstrom96
• https://github.com/Maelstrom96/esphome/tree/add-emporia-vue2-staging
• Add new component - Emporia Vue esphome/esphome#2871
• Adding documentation for new component - emporia_vue esphome/esphome-docs#1702

[krconv]

Actually, I had TX and RX backwards; TX on the debug header goes to RX on the ESP32. My bad (you mentioned it in the blog post and I missed it).
Just successfully downloaded the flash and got debug logs

[flaviut]

Ha! This stuff happens, glad you got it figured out! Please feel free to ask anything else, and let me know if you end up building something on top of all this!

[krconv]

My main goal is to build an esphome integration. I've never had to fully reverse a chip before, but it feel like a fun challenge that I might be able to help with, and I like the small form-factor and low price of the Vue. I'd appreciate any pointers if you have any; currently waiting for a USB oscilloscope to help with identifying what each of the pins are doing.

[flaviut]

Sure! First off, oscilloscopes are very helpful for identifying the function of things, but when it comes to actually reading digital data you want a logic analyzer.

Anyway, please do poke at it, it's a lot of fun. If you'd like though, I briefly described the physical communication on the board here: https://flaviutamas.com/2021/reversing-emporia-vue-2#on-board-communications

There's another chip on the board that communicates with the ESP32 over I2C. The raw I2C message gets dumped over MQTT as hex here: https://github.com/flaviut/emporia-vue2-reversing/blob/master/example_messages.txt#L2-L19. Note that right below that hex data are the output results: the I2C data is combined with the calibration data stored in NVS to produce the final output: The calibration data in NVS is nulled and not used at all.

[flaviut]

By the way, I've given you write access to this repo! Please feel free to do whatever you wish with that access, but let's keep Emporia-copyrighted stuff off of here to avoid causing them problems :)

[Maelstrom96]

Hey - I'm actually working on the exact same goal as @krconv. I'm mostly trying to make it so I can add a separate ESP32 running ESPHome that would read the messages on the i2c bus and publish that data to MQTT or through the native ESPHome API, without the need to flashing the onboard module. The reason being that I like having the data in their UI, but I also want to be able to do second triggers on the energy consumption. At first I was thinking about making a MQTT bridge to their AWS IOT mqtt endpoint, but I wasn't able to reverse engineer the certificates/auth method.

Because of this, I've been working on reverse engineering the i2c message data and I think I mostly have it figured out at this point. Here is my data spreadsheet I used to figure out / validate the parsing logic: https://docs.google.com/spreadsheets/d/1Yu98tDRhFJqqMntNak6X297SKyzQtf0_Cvv_g7qKewk/edit?usp=sharing.

The only data missing from the message is the calibration - which is only on the esp32 - and the voltage frequency.

Edit: Forgot to say, thanks @flaviut for the post and the data in the repo, it was immensely useful. Also, I updated my Vue2 firmware to the last version (Vue2-1612298550, Tue Feb 2 20:42:33 UTC 2021) and it's still outputting logs and the TEST_MODE is still there and it looks like it's still working.

[flaviut]

WOW! This is incredible! I'm curious about how you went about doing this, I did spend a while looking at things in a hex editor, but didn't really get anywhere.

[flaviut]

It took me a bit to understand the spreadsheet, but now that I do, I've written up a C declaration for that data:

// all data is little-endian
struct __attribute__((__packed__)) ch_power {
    int32_t ph1, ph2, ph3;
}

struct __attribute__((__packed__)) vue2_i2c_data {
    uint32_t unknown1;
    ch_power power[19];
    int16_t ph_voltage[3];
    int16_t frequency;
    int16_t ph2_offset_deg;
    int16_t ph3_offset_deg;
    int16_t current[19];
    uint16_t unknown2;
}

One thing that I suspect is going on here (from my use of the Vue 2 locally) is a non-1:1 mapping of channel numbers to data indexes. I think, but am not sure, that some of the indexes here do not match up with the numbers printed on the box.

[Maelstrom96]

I'll start off by saying that I'm mostly out of my league when it comes down to reverse engineering things, but I wanted to give it a shot anyway.

Input P[V] values

The first thing I did was to manually check the debug/v2 message that you had in your post to see if I could find any pattern. An obvious one was the amount of FFFF value bytes separated by 2 bytes of data. I also noticed that all the P[V] values were small-ish negative ones. Doing a find and count for FFFF in Notepad++ returned 57 match. Dividing that value with the 3 P[V] per input results in 19 - the same amount of input we have in the logs. That block was then clearly the data for all the input power values.

I copied the first 12 bytes (which should correspond to the first 3 P[V] of I01) in this amazing hex tool and started to check for other patterns in the data. I also copied another 12 bytes from another message with positive values, and the values seemed to correlate when parsed as INT32 (DCBA).

I wasn't sure what to do then, since the decimal values weren't the same as the expected values. I decided to plot values from multiple inputs on a graph to see if there was a linear equation hidden somewhere.

Here's what it gave me :

As you can see, it does look like it's linear, but calculating the actual expression gave me a pretty significant offset (which can also be seen on the graph since the line is not going through 0). A small offset is expected since the values are most likely rounded, but the result is too much. Also, by looking at the graph, we can see that there are 2 general groups. I was able to see that the slope value was different between I01-I03 and I04+.

After thinking about it, and checking the points (expected value and RAW decimal) on the graph, I noticed that not all the points were exactly on the trend line, so I started to wonder if the calibration numbers could be at play, since it would explain the deviation. I decided to multiply the values by their respective calibration and plot those values.

Here's the graph after that :

As you can see, the deviation became negligible, and the offset was almost 0 (still some but very small, > 0.03). This told me I should now be able to calculate the slope and then use that to get the P[V] value. The slope 5.5% for the first 3 inputs, and 22% for the rest.

Voltage ADC

After that, I was wondering what other values I could find, and decided to enter the next couple of bytes into the HEX tool, and got three values that I could spot easily in the debug output:

pos	Raw	INT16
0	00 0F	15
2	00 0C	12
4	00 0B	11

V1:   0.3,  0.4 Hz, *15*, 0.0225336
V2:   0.3, 0 degrees, *12*, 0.0220000
V3:   0.2, 0 degrees, *11*, 0.0220000

I wasn't sure what those values where, but I decided to see if multiplying this with the calibration data would actually give the voltage, and it actually did. So I figure this is a voltage ADC value.

Input Current?

At this point, the amount of bytes of data was pretty low, not enough for 19 x int32, I also noticed that the voltage ADC was an int16, so I started to see if I could notice any pattern with 19 x int16 in a row. It was pretty easy to spot since all the values were pretty much the same, see for yourself :

0050 0050 0050 FD4F FD4F FD4F FD4F FD4F 0350 0350 0350 FD4F FD4F FD4F FD4F FD4F 0350 0350 0350

I01:  372.4, P[V1]:   -80.0, P[V2]:   -21.9, P[V3]:    -4.9
I02:  372.4, P[V1]:   -80.0, P[V2]:   -21.9, P[V3]:    -4.9
I03:  372.4, P[V1]:   -80.0, P[V2]:   -21.9, P[V3]:    -4.9
I04:   93.1, P[V1]:   -19.4, P[V2]:    -5.2, P[V3]:    -0.9
I05:   93.1, P[V1]:   -20.3, P[V2]:    -5.6, P[V3]:    -1.7
I06:   93.1, P[V1]:   -20.5, P[V2]:    -5.8, P[V3]:    -1.3
I07:   93.1, P[V1]:   -19.2, P[V2]:    -4.4, P[V3]:    -2.4
I08:   93.1, P[V1]:   -20.0, P[V2]:    -5.7, P[V3]:    -0.7
I09:   93.1, P[V1]:   -20.9, P[V2]:    -4.8, P[V3]:    -1.0
I10:   93.1, P[V1]:   -19.4, P[V2]:    -6.2, P[V3]:    -1.0
I11:   93.1, P[V1]:   -20.1, P[V2]:    -6.1, P[V3]:    -0.8
I12:   93.1, P[V1]:   -19.4, P[V2]:    -5.2, P[V3]:    -0.9
I13:   93.1, P[V1]:   -20.3, P[V2]:    -5.6, P[V3]:    -1.7
I14:   93.1, P[V1]:   -20.5, P[V2]:    -5.8, P[V3]:    -1.3
I15:   93.1, P[V1]:   -19.2, P[V2]:    -4.4, P[V3]:    -2.4
I16:   93.1, P[V1]:   -20.0, P[V2]:    -5.7, P[V3]:    -0.7
I17:   93.1, P[V1]:   -20.9, P[V2]:    -4.8, P[V3]:    -1.0
I18:   93.1, P[V1]:   -19.4, P[V2]:    -6.2, P[V3]:    -1.0
I19:   93.1, P[V1]:   -20.1, P[V2]:    -6.1, P[V3]:    -0.8

I wasn't sure why there were 2 different values FD4F & 0350 for what looked like 93.1, but I was still pretty sure that they were the right bytes. I tried to multiply one of the values with it's input offset annnnnnnnnnnd:

0x0050 = 20480

20480 x 5.5 = 112,640

Well, that doesn't look anything close to 372.4...
How about a division?

20480 / 5.5 = 3,723.63636363636[...]

That was mostly pure luck to find that, but it looks like it is indeed how it's calculated. Just need to shift the period and we're golden!

Voltage degrees

I know it's pretty useless and not needed, but at this point I feel like superman, so I wanted to see If I could crack the code.

The amount of bytes left is pretty limited : 03EC 52EA [...] A601 8E00 0000 [...] 0000

There's only a single block with 3 int16, so it's only logical that A601 8E00 0000 are the right ones, but converting them didn't really give useful numbers :

V1: 120.2, 61.6 Hz, 5241 ADC, 0.0229308 Calibration
V2: 121.3, 121 degrees, 5574 ADC, 0.0217630 Calibration
V3:   8.1, 0 degrees, 369 ADC, 0.0220000 Calibration

pos	Raw	INT16
0	01 A6	422
2	00 8E	142
4	00 00	0

The 0 degree made sense, but how about the 142? How does that become 121 degrees? I wondered if the values were a ratio of 360°. Calculating it actually gives the right value.

142       x
---  =  ----
422     360

x = 360 x 142 / 422
x = 121.13[...]

I haven't figured out if the voltage frequency is in the data, but I don't think it is, since the first 4 bytes seems to be some kind of counter, and the last 2 bytes are always 0x0000. It could actually be the 422 value, but I would need to have more data with variations to confirm it.

[Maelstrom96]

One thing that I suspect is going on here (from my use of the Vue 2 locally) is a non-1:1 mapping of channel numbers to data indexes. I think, but am not sure, that some of the indexes here do not match up with the numbers printed on the box.

If you have a message that you think doesn't match the mapping, let me know. I don't believe that it will deviate from this structure but we never know.

[Maelstrom96]

@flaviut I wanted to know - do you happen to know the i2c frequency? I'm trying to connect to it but I'm not having much luck at 50kHz or 100kHz.

[flaviut]

@Maelstrom96 I believe it's 100kHz. I uploaded a dump from sigrok pulseview to https://github.com/flaviut/emporia-vue2-reversing/blob/master/i2c%20dump.bin, and unless the time base is off in that, it shows 100kHz.

That hopefully the screenshot will also explain how it expects to be asked for data.

[krconv]

This is awesome! I just had fun decoding the message too, and it's cool to see that I landed in the same place. I used Python to help me walk through the message, activating each of the inputs and finding the bytes that changed. In case your curious, this is my helper script. I was just about do try to figure out the calibration as well, and ended up with a linear relationship (just eye-balled it, didn't verify it was exactly linear) between points on I04[V2], similar to you.

From my testing, I'm pretty sure the uint16 with offset 0x00EE is the frequency; it was sandwiched between the AC voltages and degrees, and it was consistently changing with the frequency output in MQTT. The conversion for my device is 0.0014268004584473604 * <sensor_value> + 0.4 = frequency; tried it on the example messages @flaviut and it was a little off, but I think that is due to a different calibration.

For calibration data; I think we could do the same thing that the existing CT clamp sensor in ESPhome does, where whoever is flashing the device would need to manually calibrate each of the sensors. Maybe we could reverse engineer the calibration data from nvs, and make that easier? I think it's worth skipping that for now.

@Maelstrom96 Would you want to work together on the integration? I'm not familiar with building an ESPHome integration, are you? I am familiar with C++ and think I could make faster progress on the integration though if you want to keep looking into the connections on the board because I'm not as familiar with the signal processing side of things.