LED Output glitches on C3

[DedeHai]

What happened?

I would like to understand and possibly try to fix the glitching issues on the C3.
This bug is well known, so nothing new. There are two flavours of it I think: one is related to wifi-sleep, random glitching every few seconds if wifi-sleep is enabled. The second one may be related to file access and / or wifi resource conflicting with LED output.
Does anyone know more about this or whether it can be fixed with the currently used core libraries?

To Reproduce Bug

• enable wifi sleep on C3, run any FX
• use an ESPnow Remote or do frequent UI refreshes

Expected Behavior

Install Method

Self-Compiled

What version of WLED?

0.15 (but maybe also in 0.14, did not check)

Which microcontroller/board are you seeing the problem on?

ESP32-C3

Relevant log/trace output

No response

Anything else?

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[softhack007]

Hi @DedeHai yes this glitching is really a hard-to-solve problem. I've made lots of attempts, even tried a ton of snake oil, but I still have a setup with esp32 and 2 outputs (512 and 256 leds) where the second output regularly flickers especially when reloading the UI.

Below is what I've learned so far.

Some observations:

• the root cause seems to be the RMT driver in NeoPixelBus - the RMT unit has a very small buffer, which needs to be re-filled very often (by an interrupt routine). We are talking about several thousands of interrupts fired per second per RMT output. If the buffer is not re-filled in time, the RMT unit sends wrong data (random colors) or a pause of more than 50microseconds (not millis!) is generated which means "reset" (i.e. end of frame) and next data packets are misunderstood by the LEDs -> random flickering.

• The above observation was also confirmed when capturing the led data (coming from ESP board) with a Logic Analyser : flickering is correlated with
  a) random leds data sent by the esp (sometimes invalid data), and/or
  b) a reset (long pause >300 micros) that appears too early in the data stream

• a (theoretical) way to get rid of flickering is using FastLED drivers instead of NeoPixelBus. We made an experiment in the MoonModules fork PR 134 and did not see any flickering with FastLED drivers. However framerates were a bit slower, and mixing NPB and FastLED outputs caused crashes.

• boards with PSRAM flicker less often, thanks a patch from @willmmiles that caches presets.json in PSRAM.

• It sometimes helps to reduce the Target FPS in led settings -> if an effect runs with 32 fps, ty to reduce target FPS to 30

The snake oil section:

• ESP.getFreeHeap() has some strange side-effects that seem to increase flicker - uncommenting the calls in the WLED::handleConnection(), serializeInfo()and sendDataWs() functions reduces regular flickering (occurring each ~10 seconds). Alternatively, before calling ESP.getFreeHeap(), wait until ! strip.isUpdating().
• likewise, a build with -D WLED_DISABLE_WEBSOCKETS -D WLED_DISABLE_MQTT -D WLED_DISABLE_ADALIGHT is less flickery for me
• ESP32-S2 was especially "flicker-prone" in my tests, while classic esp32 and esp32-S3 had fewer problems
• For some users, it got better when not using I2S#1 for the very first output. However for others it got worse
• don't use the "Android" effect (WLED < 0.15.0)
• Delaying any file operations (read/write/close) until ! strip.isUpdating() also helps, however it makes the UI less responsive.

These ideas did not work:

• There is a hint in the NeoPixel Docs to use a high-priority task for calling show(). I've tried to implement it, however it did not help at all.
• Same "did not help" about reducing the task priorities of async_tcp (web server) and async_udp.

[willmmiles]

WLED forces the web server to run on the same core as loop() as a way to avoid concurrency issues. I've definitely seen short term freezeups even on baseline ESP32s as a result of wifi traffic (HomeAssistant's frequent periodic fetch of presets.json is a big source of grief). There's even special logic to cache presets.json in PSRAM if available to try to avoid the filesystem delays.

[DedeHai]

Thank you for the explanation and sharing your experience.
I played around a little with ESPNow which very reliably causes a glitch on the C3. I pinned it down to file access: when not accessing any files, there was no glitching. I then fixed that particular case with #4386
Web UI access could be solved in a similar manner if the main culprit for glitching is also file access and not wifi callbacks.

The proper fix would be to promote RMT interrupts to highest priority over any others, but I don't know if that is even possible and it probably requires changes to the IDF config, which of course we cannot change using the Arduino framework.

@Makuna do you know why enabling wifi-sleep on a C3 causes glitches in LED data sendout and is there a fix or workaround that could be used in WLED?

[KrX3D]

Thanks for that hint: enable wifi sleep on C3, run any FX

so i checkd the checkbox to disable wifi sleep on all my devices and not mostly all glitches are away.
on nearly 6 devices it happened all the times every couple of seconds, now i only see on one device (esp32 wroom, flashed as esp32dev) like every couple of minutes on just a couple of leds randomly glitch.

on my esp32 c3/s2 minis work now nicely without them

[Makuna]

Note that RMT API I am using is being depreciated. The Core3 branch of my library uses the new RMT APIs which may solve this issue in the long term.

But...

In "NeoEsp32RmtMethod.h", you will find the following code...

#if ESP_IDF_VERSION >= ESP_IDF_VERSION_VAL(4, 3, 0)
#define NEOPIXELBUS_RMT_INT_FLAGS (ESP_INTR_FLAG_LOWMED)
#else
#define NEOPIXELBUS_RMT_INT_FLAGS (ESP_INTR_FLAG_IRAM | ESP_INTR_FLAG_LEVEL1)
#endif

The flag values are defined in "esp_intr_alloc.h" as

#define ESP_INTR_FLAG_LEVEL1        (1<<1)  ///< Accept a Level 1 interrupt vector (lowest priority)
#define ESP_INTR_FLAG_LEVEL2        (1<<2)  ///< Accept a Level 2 interrupt vector
#define ESP_INTR_FLAG_LEVEL3        (1<<3)  ///< Accept a Level 3 interrupt vector
#define ESP_INTR_FLAG_LEVEL4        (1<<4)  ///< Accept a Level 4 interrupt vector
#define ESP_INTR_FLAG_LEVEL5        (1<<5)  ///< Accept a Level 5 interrupt vector
#define ESP_INTR_FLAG_LEVEL6        (1<<6)  ///< Accept a Level 6 interrupt vector
#define ESP_INTR_FLAG_NMI           (1<<7)  ///< Accept a Level 7 interrupt vector (highest priority)
#define ESP_INTR_FLAG_SHARED        (1<<8)  ///< Interrupt can be shared between ISRs
#define ESP_INTR_FLAG_EDGE          (1<<9)  ///< Edge-triggered interrupt
#define ESP_INTR_FLAG_IRAM          (1<<10) ///< ISR can be called if cache is disabled
#define ESP_INTR_FLAG_INTRDISABLED  (1<<11) ///< Return with this interrupt disabled

#define ESP_INTR_FLAG_LOWMED    (ESP_INTR_FLAG_LEVEL1|ESP_INTR_FLAG_LEVEL2|ESP_INTR_FLAG_LEVEL3) ///< Low and medium prio interrupts. These can be handled in C.

I would try adding the ESP_INTR_FLAG_IRAM back first and see the results

#define NEOPIXELBUS_RMT_INT_FLAGS (ESP_INTR_FLAG_IRAM | ESP_INTR_FLAG_LOWMED)

If that doesn't solve it, then change to only accept level 3 or level 2.

#define NEOPIXELBUS_RMT_INT_FLAGS (ESP_INTR_FLAG_IRAM | ESP_INTR_FLAG_LEVEL3)

[Makuna]

Also note, the C3 (all the C chips?) is a single core. This is problematic with any routines that have long ISRs or long "high priority" level. And due to RMT have a very small, dedicated DMA memory bank, you will be limited to size of the strip if you don't use a "translate and fill" ISR. This is the ISR priority that is mentioned in my last post; and it can't be higher that LEVEL 3. I haven't confirmed the C3 has the same limitations on RMT DMA memory as previous version, but the API should have buffered NPB from the problem, but they deprecated the API and I believe thus didn't update it.

AND, you can't really put any ISRs at a higher level that WiFi or it will cause problems with connections and data transfers. I tried this with the bitbang method and they just aren't compatible and its why I do not recommend the use of bitbang as it can't be set at a high enough priority to be consistent.

Note, the I2S Parallel doesn't have this limitation.

I have always suggested for most platforms (NPB is multiplatform, not just ESP) that when you have such routines, that you pause the NPB output completely (don't call show) while they are active. Not always a workable solution.

[DedeHai]

thank you for the in-depth explanation, appreciate it.
I am aware of hardware limitiations of the C3 with a single core and read a fair bit of the espressif documentation, but never touchted RMT or DMA so its good to get some first-hand info about the limitations.

[Jason2866]

Seeing the many changes and bug fixes in IDF 5.x. I doubt that the old not maintained IDF (version which is used in the Arduino cores for WLED) will ever work statisfying for the C3.
As far as i can say, Arduino core 3.1.0 is a big step forward, since it uses IDF APIs directly and Arduino does no more code "trickery". The new RMT API works reliable. But before WLED can move on a really stable and performant AsyncHTTP is needed. The current used one is not good enough. As mentioned the DMA buffer is smaller for the C3. To get a decent performance code needs to be optimized in WLED

[DedeHai]

just FYI: some of the glitches are related to file-access and not wifi, the ESPNow glitching mentioned in the opening question for example is fixed with #4386