Panic when driving larger matrixes (one long strip) on 32MB 8MB ESP32-S3-DevKitC1

[GameTec-live]

Bug report

32MB FLASH 8MB PRAM ESP32-S3-DevKitC1(ESP32-S3-DevKitC-1-N32R8V)

Problem

Steps

1. Modify env:demo to compile properly for the chip (SPIFFS Failing on a 32MB 8MB ESP32-S3-DevKitC1 #579)

[env:demo]
extends         = dev_esp32-s3
build_flags     = -DDEMO=1
                  ${dev_esp32-s3.build_flags}
                  ${psram_flags.build_flags}
board_build.partitions = config/partitions_custom_8M.csv
board_upload.flash_size = 32MB
board_build.flash_mode = qio
board_build.arduino.memory_type = opi_opi

2. In global.h define a matrix width and height larger than 36 (so 37+ results in the panic)
3. See core 1 panic and the system reboot

Example

Notes
In this case a "Matrix" is just a bunch of daisychained LED strips going back and forth.

My globals.h
globals.h.txt

Same exact config file works fine on a generic, less powerful, esp32.
And reportedly running 1500 leds on one controller isnt optimal, but it works fine and i get a decent frame rate on my underpowered esp32. It shouldnt crash anyways ;)

Monitor Log:
https://hastebin.skyra.pw/wikevijele.yaml

[rbergen]

@GameTec-live The hastebin link at the end of your description leads to an empty page with a blinking cursor in my browser.

[GameTec-live]

@GameTec-live The hastebin link at the end of your description leads to an empty page with a blinking cursor in my browser.

oh, weird... ill reupload later... (when I'm home)

[GameTec-live]

crashlog.log
Sorry for the late reply, but here you go...

[robertlipe]

Please build debug, as instructed, and be sure you're running the trace decode filter. I think most o the flags are already present because I get them by default. https://docs.platformio.org/en/latest/core/userguide/device/cmd_monitor.html#filters platformio/platform-espressif32#105 (comment) I suspect you're just building opt and not debug.

…

On Thu, Jan 4, 2024 at 1:39 PM GameTec-live ***@***.***> wrote: crashlog.log <https://github.com/PlummersSoftwareLLC/NightDriverStrip/files/13834734/crashlog.log> Sorry for the late reply, but here you go... — Reply to this email directly, view it on GitHub <#580 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD36TAZ5QBUHW5EBIVTTYM4APBAVCNFSM6AAAAABBFMSILWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXGY3DAMJRGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[GameTec-live]

changed build_type under base from release to debug and set monitor_filters = esp32_exception_decoder under [env:demo]

heres the new log:
debug-log.log

[robertlipe]

That's indicating a problem in a library we use, somewher around: https://github.com/FastLED/FastLED/blob/09c5fb8f74c43191974c09e1f31edda8281eab7e/src/platforms/esp/32/clockless_rmt_esp32.cpp#L503 You're going to have to get a debugger (or other stone-banging techniques) in there to see if mCur is walking past the end of mPixelData[] (questioning mSize corruption?) or some other zany behaviour. I don't recognize it and don't see smoking guns on this topic in the fastled buganizer. https://github.com/FastLED/FastLED/issues?q=+ESP32RMTController%3A%3AfillNext+

…

On Thu, Jan 4, 2024 at 5:00 PM GameTec-live ***@***.***> wrote: changed build_type under base from release to debug and set monitor_filters = esp32_exception_decoder under [env:demo] heres the new debug-log.log <https://github.com/PlummersSoftwareLLC/NightDriverStrip/files/13836222/debug-log.log> log: — Reply to this email directly, view it on GitHub <#580 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD3YYAWV3WFOXS2FPSF3YM4YAPAVCNFSM6AAAAABBFMSILWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXHA3TQMRXGU> . You are receiving this because you commented.Message ID: ***@***.***>

[GameTec-live]

probably the stupidiest thing youve heard in a while, but.... i do have a pico debug probe (SWD) and cant figure out where im supposed to hook it up...

[robertlipe]

Dude, you can't imagine the stupid stuff I hear. That's not even on the list. :-) There is an embarrassment of debugging options on S3. Picking one may actually be the hardest part. Espressif has this locked down. Even if you don't follow all the steps, reading this chapter is worthwhile: https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/api-guides/jtag-debugging/# You could hook up wires for TDO, TDI, TMS, and TCK like a caveman and be proud of your pico purchase. If you work with lots of micros and have a good workflow around that, go for it. https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/api-guides/jtag-debugging/configure-other-jtag.html If you don't, *don't do that.* "The quickest and most convenient way to start with JTAG debugging is through a USB cable connected to the D+/D- USB pins of ESP32-S3. No need for an external JTAG adapter and extra wiring/cable to connect JTAG to ESP32-S3." So configure it for the built-in JTAG interface. This may require some driver fiddling depending on yoru OS. The one downfall of this approach over a "real" JTAG pod, IMO, is that it resets the connection when the ESP32 resets, such as when you load new code. This has been fixed in newer ESP parts like the H2 and C6. Now here's where I'll get hand-wavy because you surely don't want to do it the way I do it. (See also: working with dozens of micros.) I try to use as little of platformio and visual studio as I can because I want to use the SAME debuggers on as many of those micros as I can. I understand, however, that Platformio has some pointy-clicky stuff that automates some of the above like building the openocd config. (So why did I tell you to read that? Because IMO, a developer should know these things about our tools.) This is very Windows-centric, but it gives you the needed debug_FOO and build_type stuff and a nickel tour of GDB. https://community.platformio.org/t/how-to-use-jtag-built-in-debugger-of-the-esp32-s3-in-platformio/36042/3 This talks about the platformio doc being wrong/misleading. https://community.platformio.org/t/esp32-s3-jtag-debugging-over-usb/28182 Looks like the official doc is fixed: https://docs.platformio.org/en/latest/boards/espressif32/esp32-s3-devkitc-1.html#debugging The concepts you have to embrace are that OpenOCD is the middle-man. It uses libusb (or Windows driver stuff) to actually open the debug interface on the board. It creates a couple of network sockets. IIRC, port 3333 is the one that GDB does the 'target extended remote:localhost:3333" to connect to (look all this up...I may be typing crazy talk, but I have the big picture right) and there's an additional port that you can telnet to that is, I think, 4444. This lets you talk to the board directly, but you can also confuse GDB if you, say, change a memory location that happens to hold a variable and you change it without GDB knowing about it. Be careful. Once you "get it", I think you really will find that the hardest part is wading through the redundant documentation of expanations fo rmanually setting up openocd (for both the SoC and the board) which may be different for CLI use and Platformio use and the differences in the 'real' jtag interface and the chip-resident jtag interface and all that. This is why it's worth reading the whole thing - jumping around and copy-pasting bits will surely get you in trouble. There are a lot of moving parts, but the ability to single step, display variables and stack traces, and set breakpoints is worth its weight in gold and worth the finicky setup. Good night, good luck and... May the source be with you. RJL

…

On Fri, Jan 5, 2024 at 2:20 AM GameTec-live ***@***.***> wrote: probably the stupidiest thing youve heard in a while, but.... i do have a pico debug probe (SWD) and cant figure out where im supposed to hook it up... — Reply to this email directly, view it on GitHub <#580 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD35GVN36RNONZI6BR4LYM6ZWRAVCNFSM6AAAAABBFMSILWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZYGI4TCMZZG4> . You are receiving this because you commented.Message ID: ***@***.***>

[GameTec-live]

Okay, thanks for the help, i managed to get a debugger running (it was a driver issue...)... I don't know what the value of mCur is supposed to be, etc... I'll poke around a bit more and report back...

[GameTec-live]

cant really see anything unusal? (i mean; i also dont know what those values are supposed to be lol)
mPixelData seems to be large or even "infinite" as i can request a lot more than 1500 array entries with the debugger... mCur also goes past 1500...

[rbergen]

@GameTec-live Glad to see you got the debugger working, and I appreciate the earlier upload of the debug log. As @robertlipe already indicated, it does show that the actual problem (which is a form of illegal memory access) takes place several levels below "our" code. In fact, the backtrace doesn't even include references to any code that's part of NightDriverStrip.

Concerning your last comment, we're obviously not looking over your shoulder, so we can't see what you are looking at. Also, even if we could then figuring out what the cause of the invalid behavior is would effectively require us to debug the dependency libraries involved. Which isn't entirely impossible, but very difficult if we can't ourselves debug trace through the code just before the problem occurs.

Without having the hardware and software setup that triggers these crashes available, I therefore think we won't be able to solve this. You could (still) consider raising a bug report in the dependent libraries (Espressif ESP-IDF and/or FastLED) and see if they are able to provide pointers to what's actually behind this.

I'll leave this issue open in case someone else runs into the same problem, and may be able to provide additional information that can help get to the root of this.

[GameTec-live]

Yeah, kinda hard to replicate a issue without hardware... Ill open a issue over at FastLED... Thanks for the help though...

[prschguy1]

hey GameTec-live
While i am not one of the bigger brains on this repository, perhaps I can provide some help to the problems you are experiencing.
here. My understanding is that demo is intended for strip effects only. I was a bit surprised that you tried putting spectrum into a demo build. have never tried that, but assumed it wouldn't work as newer spectrum builds use dma. .Instead of using demo as your build, might suggest using spectrum-elecrow, I have that working properly on the chip you specify. it looks good with pdm mic, remote, display, and all the effects. when I run it with strip effects, I run out of memory, but runs well on spectrum. might give it a try. Memory usage looks pretty good here. just my 2 cents.

20240108_191857.mp4