[BUG] Device disconnects for hours/days randomly, and then it connects and works again for a while

[heisenberg2980]

Bug report

Describe the bug

Device disconnects for hours/days randomly, and then it connects and works again for a while. This is the error displayed in the web interface while the issue is happening:

And this is the log from ESPHome:

Additional information

• used Hardware:
  • EspHoMaTriXv2 version: 2024.12.1

To Reproduce

Install ESPHome into the EspHoMaTriXv2 and let it run for some days

Expected behavior

Device shouldn´t disconnect.

Configuration

substitutions:
  devicename: ulanzi
  friendly_name: LED Matrix
  board: esp32dev
  # Pin definition from https://github.com/aptonline/PixelIt_Ulanzi 
  battery_pin: GPIO34 
  ldr_pin: GPIO35 
  matrix_pin: GPIO32 
  left_button_pin: GPIO26 
  mid_button_pin: GPIO27 
  right_button_pin: GPIO14 
  scl_pin: GPIO22 
  sda_pin: GPIO21 

switch:
  - platform: template
    name: "Auto-Adjust Brightness"
    id: switch_autobrightness
    icon: mdi:brightness-auto
    restore_mode: RESTORE_DEFAULT_ON
    lambda: |-
      if (id(aab_enable)) {
        return true;
      } else {
        return false;
      }
    turn_on_action:
      lambda: |-
        id(aab_enable) = true;
    turn_off_action:
      lambda: |-
        id(aab_enable) = false;

globals:
  # aab = auto-adjustable brightness
  - id: aab_enable
    type: "bool"
    restore_value: true
    initial_value: "true"
  - id: aab_add
    type: int
    initial_value: '10'
  - id: aab_max
    type: int
    initial_value: '220'
  - id: aab_min
    type: int
    initial_value: '20'

external_components:
  - source:
      type: git
      url: https://github.com/lubeda/EspHoMaTriXv2
      ref: 2024.12.1
    refresh: 60s 
    components: [ ehmtxv2 ]    

esphome:
  comment: "EHMTXv2 from LuBeDa"
  name: $devicename 
  project:
    name: "Ulanzi.EHMTXv2"
    version: "2.0.0"
  on_boot:
    then:
      - ds1307.read_time:

esp32:
  board: esp32dev

font:
  # Font included in this folder
  # - file: mateine.ttf
  #   size: 16
  #   id: default_font
  #   glyphs:  |
  #     !?"%()+*=,-_.:°0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnÖÄÜöäüopqrstuvwxyz@<>ߧ€/
  # Matrix Clock Fonts
  # Grab these from https://github.com/trip5/Matrix-Fonts
  # See https://github.com/lubeda/EspHoMaTriXv2#font
  # Store in a "fonts" subfolder of your ESPHome config folder or change paths below
  - file: fonts/MatrixLight8.bdf
    id: default_font
    glyphs:  |
      ! "#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz°
  - file: fonts/MatrixChunky6.bdf
    id: special_font
    glyphs:  |
      ! "#$%&'()*+,-./0123456789:APMTapmt
  
binary_sensor:
  - platform: status
    name: "$devicename Status"
  - platform: gpio
    pin:
      number: $left_button_pin
      inverted: true
    name: "Left button"
  - platform: gpio
    pin: 
      inverted: true
      number: $mid_button_pin
      mode: INPUT_PULLUP
    name: "Middle button"
  - platform: gpio
    pin: 
      number: $right_button_pin
      inverted: true
    name: "Right button"

logger:
  level: WARN

output:
  - platform: gpio
    pin:
      number: GPIO15
      ignore_strapping_warning: true
    id: buzzer_pin

api:

sensor:
  - platform: sht3xd
    temperature:
      name: "$devicename Temperature"
    humidity:
      name: "$devicename Relative Humidity"
    update_interval: 60s
  - platform: adc
    pin: $battery_pin
    name: "$devicename Battery"
    id: battery_voltage
    update_interval: 10s
    device_class: battery
    accuracy_decimals: 0
    attenuation: auto
    filters:
      - sliding_window_moving_average:
          window_size: 15
          send_every: 15
          send_first_at: 1
      - multiply: 1.6
      - lambda: |-
          auto r = ((x - 3) / 0.69 * 100.00);
          if (r >= 100) return 100;
          if (r > 0) return r;
          if (r <= 0) return 1;
          return 0;
    unit_of_measurement: '%'
  - platform: adc
    id: light_sensor
    name: "$devicename Illuminance"
    pin: $ldr_pin
    update_interval: 10s
    attenuation: auto
    unit_of_measurement: lx
    device_class: illuminance
    accuracy_decimals: 0
    filters:
      - lambda: |-
          return (x / 10000.0) * 2000000.0 - 15 ;
    on_value:
      then:
        - lambda: |-
            if ( id(aab_enable) ) {
              int n = x / 4 + id(aab_add); // new_value
              if (n > id(aab_max)) n = id(aab_max);
              if (n < id(aab_min)) n = id(aab_min);
              int c = id(rgb8x32)->get_brightness(); // current value
              c = c>0?c:1 ; 
              int d = (n - c) * 100 / c; // diff in %
              if ( abs(d) > 2 ) id(rgb8x32)->set_brightness(n);
            }
      
ota:
  password: !secret ota_password
  platform: esphome

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

web_server:

i2c:
  sda: $sda_pin
  scl: $scl_pin
  scan: true
  id: i2cbus

light:
  - platform: neopixelbus
    id: ehmtx_light
    type: GRB
    internal: true
    variant: WS2812
    pin: $matrix_pin
    num_leds: 256
    color_correct: [30%, 30%, 30%]
    gamma_correct: 2.0
    name: "$devicename Light"
    restore_mode: ALWAYS_OFF
    
time:
  - platform: homeassistant
    on_time_sync:
      then:
        ds1307.write_time:
  - platform: ds1307
    update_interval: never
    id: ehmtx_time

display:
  - platform: addressable_light
    id: ehmtx_display
    addressable_light_id: ehmtx_light
    width: 32
    height: 8
    pixel_mapper: |-
      if (y % 2 == 0) {
        return (y * 32) + x;
      }
      return (y * 32) + (31 - x);
    rotation: 0°
    update_interval: 16ms
    auto_clear_enabled: true
    lambda: |-
      id(rgb8x32)->tick();
      id(rgb8x32)->draw();
      
ehmtxv2:
  id: rgb8x32
  icons2html: true
  matrix_component: ehmtx_display
  time_component: ehmtx_time
  time_format: "%H:%M"
  date_format: "%d.%m."
  show_seconds: false
  # Uncomment below if using the mateine font
  # default_font_id: default_font
  # special_font_id: default_font
  # Comment out below if using mateine font
  default_font_id: default_font
  default_font_yoffset: 8
  special_font_id: special_font
  special_font_yoffset: 6
  # until here
  boot_logo: "[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,63519,63519,63519,63519,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,63519,0,0,0,0,2016,0,0,0,0,0,0,0,0,0,0,31,0,0,0,0,0,0,0,0,0,63488,0,63488,0,0,0,63519,0,0,0,0,2016,2016,0,0,0,65514,0,65514,0,0,0,31,0,0,0,64512,0,0,64512,0,63488,63488,0,63488,63488,0,0,63519,63519,63519,0,0,2016,0,2016,0,65514,0,65514,0,65514,0,31,31,31,0,0,0,64512,64512,0,0,63488,63488,63488,63488,63488,0,0,63519,0,0,0,0,2016,0,2016,0,65514,0,65514,0,65514,0,0,31,0,0,0,0,64512,64512,0,0,0,63488,63488,63488,0,0,0,63519,63519,63519,63519,0,2016,0,2016,0,65514,0,65514,0,65514,0,0,0,31,31,0,64512,0,0,64512,0,0,0,63488,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]"
  icons: 
    - id: error
      lameid: 40530
    - id: home_assistant
      lameid: 47693
    - id: temperature
      lameid: 2056
    - id: lightbulb
      lameid: 1762
    - id: music
      lameid: 45625
    - id: phone
      lameid: 1232
    - id: car
      lameid: 2819
    - id: sleep8x32
      url: https://user-images.githubusercontent.com/16407309/224850723-634c9b2d-55d9-44f2-9f93-765c0485b090.gif 
    - id: cat
      lameid: 12200
  on_next_screen:
    - homeassistant.event:
        event: esphome.new_screen
        data_template:
          iconname: !lambda "return icon.c_str();"
          text: !lambda "return text.c_str();"

image:
   - file: 1pixel.gif
     id: breaking20237

animation:
  - file: 1pixel.gif
    id: breaking202371

[lubeda]

   if (!this->is_connected()) {
      if (now - this->last_connected_ > this->reboot_timeout_) {
        ESP_LOGE(TAG, "No client connected to API. Rebooting...");
        App.reboot();
      }
      this->status_set_warning();
    }

The message from the log come from a "watchdog", there should be only one entry in the log and the device should reboot.
I don't think a change in EspHoMatriX could solve this problem. Is a hardware defect possible, that causes the error and blocks blocks the reboot?

I don't know.

[heisenberg2980]

@lubeda why do you think the device is not rebooting? There is a 15 minutes delay between messages, so I believe the device is actually rebooting, but it doesn´t connect to ESPHome after the reboot so it keeps rebooting every 15 minutes. But the logs from ESPHome shows that it's actually connected to the device, so I´m not sure what is going on.

[lubeda]

Ussually after a reboot, there are a lot more log-entries.
can you change the loglevel to debug?

logger:
  level: DEBUG

and try again. Does the history of home assistant show the device as unavailable before it reboots?

[lubeda]

This could also help the reason of the connection problem

api:
  # ...
  on_client_disconnected:
    - logger.log: "API client disconnected!"

[heisenberg2980]

Now the device has started to work again on its own and it´s completely back to normal (as usual).

While the device was not working it appeared as disconnected in HA:

I have added the on_client_disconnected to the API section to see if I get any extra info next time the issue happens

[heisenberg2980]

After adding the on_client_disconnected I restarted the device and the issue happened again, but I don´t see any extra lines in the logs:

[heisenberg2980]

I just saw a new warning, not sure what it means or if it´s relevant:

[andrewjswan]

I just saw a new warning, not sure what it means or if it´s relevant:

Add to esp32 section ignore_efuse_custom_mac like:

esp32:
  framework:
    advanced:
        ignore_efuse_custom_mac: true

[heisenberg2980]

@andrewjswan will that fix the main error of the device losing its connection for hours/days, or will it just remove the warning I mentioned in my last comment from the logs?

[andrewjswan]

@heisenberg2980 Should solve both problems.

[heisenberg2980]

@andrewjswan unfortunatelly after adding that code the original issue is still there, and the device is still not showing anything on the screen

These are the logs after installing the new firmware with the ignore_efuse_custom_mac added:

And this is the log on the UI:

[trip5]

I'm going to pipe in. I've been watching this go on and I'm wondering if the issue isn't more basic... sometimes we skip over simple things when the problem looks complicated...

0. You're running Home Assistant 24-hours and this is the only device that goes offline?
1. Have you checked your router's interface and the device isn't randomly going offline?
2. Have you tried deleting the device from Home Assistant? (reboot the clock to get it reconnected - sometimes stray entities cause so you may need to search and delete all entities that look like they're associated with the clock before rebooting the clock)
3. Have you tried using a physical connection to update the ESPHome firmware of the clock? (sometimes it's necessary because this includes a format that clears the flash memory completely.)
4. Are you sure the power supply is adequate?

[heisenberg2980]

@trip5 thanks for piping in. I have ruled out some of those, let me share my thought process and please shout if any of my assumptions could be incorrect:

• The device doesn´t go offline, as I can access its user interface the whole time. The device just stops displaying data on the screen and keeps rebooting, which makes me think the problem is not related to the network and there is some issue with the firmware.
• I have ruled out any issues related with the power supply because this device has a battery that can power the device for hours, and the battery is always 100%.
• Not sure why you think deleting the device from Home Assistant would help. I have over 20 esphome devices connected to HA and all of them work perfectly fine except this one.

My feeling is that I have hit a bug that is present in every device but for some reason other people are not hitting it. It may be something to do with the data I´m displaying on the screen, or how I turn the screen of the device on only when there is presence detected in the room, but I believe the issue is somewhere in the firmware. I read somewhere that the hardware of this device is not very stable, so that could be the problem as well, but in that case I would expect more people to see the same behaviour.

[trip5]

Deleting the device (and checking for any stray entities) has helped me before. Home Assistant has a tendency to duplicate entities sometimes when installing ESPHome firmware that has sometimes changed ever so slightly and thanks to the weirdness surrounding the MAC address issue, which is how HA keeps track of unique devices... I noticed a few entities that had a _1 or a _2 after applying the new fix... as if it were adding an entirely new device, but without asking, it just created new entities that didn't quite match was in my automations. I actually just cleared out everything ehmtxv2-related, rebooted all clocks, re-added the devices... not a lot of work, so why not do it to rule it out?

And flashing the physical connection can be absolutely necessary sometimes. OTA is a tricky thing and with the Ulanzi clocks it's dead-simple, so why not do it to rule it out?

And yup, you've probably hit a bug that other people aren't experiencing, or perhaps their update process is different than yours, or perhaps their Home Assistant is operating differently... so many variables involved, so you should probably try everything that has a chance to fix the error. BTW, that 15 minute reboot time is absolutely related to Home Assistant: https://community.home-assistant.io/t/why-are-my-esps-api-rebooting-every-15-minutes-even-if-they-are-accessible-from-hass/476016/5 which is why I think something to do with HA and the ESPHome API.

Some more reading if you want to to continue diagnosing API events: https://esphome.io/components/api.html