Filtering out extreme values

[mark-10-5]

I seem to be getting some noise on my sensor and I get very extreme values when this happens. These readings are usually over 100,000 W up to over 1 million some times.

I have found a lambda filler that has a max difference value but am not smart enough to combine the current lambda filter with this one and I can't find in the esphome docs if I can have two lambda filters and it process in order or not.

This is the filter I have found
filters:
- lambda: |-
float MAX_DIFFERENCE = 50000.0; // adjust this!
static float last_value = NAN;
if (isnan(last_value) || std::abs(x - last_value) < MAX_DIFFERENCE)
return last_value = x;
else
return {};

And I need to combine it with the current filter of

• lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);

[lemppari]

You could try this (I haven't tested this...)

        - lambda: |-
            float new_value = x * ((60.0 / ${pulse_rate}) * 1000.0);
            float MAX_DIFFERENCE = 50000.0.0;  // adjust this!
            static float last_value = NAN;
            if (isnan(last_value) || std::abs(new_value - last_value) < MAX_DIFFERENCE)
              last_value = new_value;
            return last_value;

The problem with this is that if the first value is a noise value then the following sane values will be ignored. I think I found the source of your filter example and there is an updated version in that page which should handle that situation a little better. Here is a modified code for that

        - lambda: |-
            float new_value = x * ((60.0 / ${pulse_rate}) * 1000.0);
            float MAX_DIFFERENCE = 50000.0;  // adjust this!
            static float last_value_t = NAN;
            static int count_missed_t = 0;
            if (count_missed_t == 3) last_value_t = new_value;
            if (isnan(last_value_t) || std::abs(new_value - last_value_t) < MAX_DIFFERENCE) {
              count_missed_t = 0;
              last_value_t = new_value;
            } else {
              count_missed_t += 1;
            }
            return last_value_t;

This will handle the first noise value problem. It will revert to new "range of values" after getting three out of range values in a row. Let us know if either of these work.

EDITED: First version compared the values before applying the pulse calculation. Should be fixed now.

[mark-10-5]

I found out by asking on the ESPHome discord that filter values are processed in order of entry in YAML and you can have as many as you like. So i ended up testing it like this last night.

  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 50000.0;  // adjust this!
      static float last_value = NAN;
      if (isnan(last_value) || std::abs(x - last_value) < MAX_DIFFERENCE)
        return last_value = x;
      else
        return {};

I ran the pulse rate calc first as the values i was working with in the graphs in home assistant are after the calculation so I wanted to filter it after the calculation. This worked pretty well, i didn't get any 1 million W readings but i did get 1 96000W reading. So i think your suggestion of the new code is a good one so i am going to test this.

  - lambda: return x * ((60.0 / ${pulse_rate}) * 1000.0);
  - lambda: |-
      float MAX_DIFFERENCE = 30000.0;  // adjust this!
      static float last_value_t = NAN;
      static int count_missed_t = 0;
      if (count_missed_t == 3) last_value_t = x;
      if (isnan(last_value_t) || std::abs(x - last_value_t) < MAX_DIFFERENCE) {
        count_missed_t = 0;
        return last_value_t = x;
      } else {
        count_missed_t += 1;
        return last_value_t;
      }

I am going with 30000W now as my panel is 125Amp max which is 24000W max draw so anything over 30000W should be well safe as being false.

Thanks for the suggestions and i will report back if this works.

[lemppari]

Sorry, I made the comparing to previous values before doing the pulse rate calculation. I modified my code also. Please report your progres. :)

[onkelbeh]

Had the same problem, with rare very high spikes (>8kW), solved it with a filter:

sensor:
  - platform: pulse_meter
    name: '${name} - Power consumption'
    unit_of_measurement: 'W'
    state_class: measurement
    device_class: power
    icon: mdi:flash-outline
    accuracy_decimals: 0
    internal_filter: 50ms

I did not use an ESP32, have a NodeMCU12S.
Comparison with the values shown directly on the meter matches.

[mark-10-5]

I still get the odd value with the max difference code. I will add in the internal_filter also.

It has been much better though. I am still trying to pin point where the noise is coming from but not having much luck.

[Brink3]

I also get these spikes, tried your filters but still gets those spkies time to time.
Screenshot from the original code:

.

[lemppari]

That is very strange. Do you get only single spikes of crazy values or are there multiple crazy values in row?

If the code from my last example works, you should get four crazy values in a row. Here is a sample what should happen. First row are the new-values calculated in the first line of the code (s prefix is sane value and c is a crazy value). Second row is the output of the filter. Third row is the value of the count_missed_t variable after each step.

s1, s2, c3, s4, s5, c6, c7, c8, c9, s10, s11, s12, s13, s14
s1, s2, s2, s4, s5, s5, s5, s5, c9,  c9,  c9,  c9, s13, s14
0,   0,  1,  0,  0,  1,  2,  3,  0,   1,   2,   3,   0,   0

As we can see there needs to be four crazy values in a row to produce crazy values. And if that happens, even after sane values start coming right after those, the output will be four crazy values in a row before sanes values are the "new normal" again.

[Brink3]

There is just one single crazy spike. I have attached a log from the esphome logging.
Line 4 in this log shows a spike.

[19:21:06][D][light:035]: 'Red' Setting:
[19:21:06][D][light:046]:   State: ON
[19:21:06][D][sensor:121]: 'House - Total energy': Sending state 8.43400 kWh with 3 decimals of accuracy
[19:21:06][D][sensor:121]: 'House - Power consumption': Sending state 45569.61719 W with 0 decimals of accuracy
[19:21:06][D][light:035]: 'Red' Setting:
[19:21:06][D][sensor:121]: 'House - Total energy': Sending state 8.43500 kWh with 3 decimals of accuracy
[19:21:06][D][light:035]: 'Red' Setting:
[19:21:06][D][light:046]:   State: OFF
[19:21:06][D][light:035]: 'Red' Setting:
[19:21:09][D][sensor:121]: 'House - Power consumption': Sending state 1107.69226 W with 0 decimals of accuracy
[19:21:09][D][light:035]: 'Red' Setting:
[19:21:09][D][light:046]:   State: ON
[19:21:09][D][sensor:121]: 'House - Total energy': Sending state 8.43600 kWh with 3 decimals of accuracy
[19:21:10][D][light:035]: 'Red' Setting:
[19:21:10][D][light:046]:   State: OFF
[19:21:12][D][sensor:121]: 'House - Power consumption': Sending state 1116.97168 W with 0 decimals of accuracy
[19:21:12][D][light:035]: 'Red' Setting:
[19:21:12][D][light:046]:   State: ON
[19:21:12][D][sensor:121]: 'House - Total energy': Sending state 8.43700 kWh with 3 decimals of accuracy
[19:21:13][D][light:035]: 'Red' Setting:
[19:21:13][D][light:046]:   State: OFF

[lemppari]

Makes no sense to me. You could try to add some logging or take this up with the ESPHome people.

[Brink3]

Makes no sense to me. You could try to add some logging or take this up with the ESPHome people.

Dude, I'm so friggin' stupid. I never changed value of MAX_DIFFERENCE. Changed it now to 40000.0, no spikes so far.

[mark-10-5]

Did you leave it at 50000.0? I think it works better if you calculate the max wattage of your power board and set it to that.

I am finding the internal_filter is working better then the filter code.

[Brink3]

Did you leave it at 50000.0? I think it works better if you calculate the max wattage of your power board and set it to that.

I am finding the internal_filter is working better then the filter code.

Yes, I had it at 50000.0 ;)

Tried the internal_filter @ 50ms and 100ms, but I got same spikes.

[mark-10-5]

Interesting your spikes are always around 40,000, the internal_filter wouldn't catch that as it is not extreme enough. My spikes were much higher, around 100,000+ up to 1,000,000 often and a much wider value range, which the internal filter would catch as it is so extreme so the pulse would be very quick so be filtered out.

40,000 although a high value is a possible value so you will need the max difference calc to filter that one. Glad it seems to be working so far.

[JamesSwift]

I'm running on a D1 mini (esp8266) and also getting crazy values at random times.

I wanted to add some thoughts on the above ideas. The internal_filter is handy, but it's essentially a low pass filter. Any crazy value BELOW the internal_filter will still get through. So you still might see a 24,000w crazy value next to a 100w known good value.

The advantage of the lambda filter though, is that you are looking for the DIFFERENCE between the last known good reading and the current reading. However, you still seem to be configuring it based on the maximum possible load of your board, Surely MAX_DIFFERENCE shouldn't be the maximum power draw of your house, but somewhere near the biggest possible load added by a single appliance. For instance, in my house the electric shower is about 40a (around 10kw). The chances of someone turning the shower and another appliance on in the exact same instant is basically zero. So I think a saner value for MAX_DIFFERENCE would be 11000.0.