Load access fault (NULL deref) in at_process_task during rapid TCP link cycling on ESP32-C6

[talnirnx]

Environment

Item	Value
Module	ESP32-C6
ESP-AT Version	v4.1.1.0
ESP-IDF Branch	release/v5.4
ESP-IDF Commit	8ad0d3d8f2faab752635bee36070313c47c07a13
AT Core Lib	libesp32c6_at_core.a (commit ba4dd0e)
CONFIG_MBEDTLS_DYNAMIC_BUFFER	y
CONFIG_AT_PROCESS_TASK_STACK_SIZE	6144
CONFIG_AT_SOCKET_MAX_CONN_NUM	5
CONFIG_LWIP_MAX_SOCKETS	16
CONFIG_LWIP_TCP_MAXRTX	6
CONFIG_LWIP_TCP_MSL	default (60000)
CONFIG_PM_ENABLE	y
CONFIG_FREERTOS_UNICORE	y

Bug Description

We observe a reproducible Guru Meditation Error: Core 0 panic'ed (Load access fault) crash in at_process_task on ESP32-C6 running ESP-AT v4.1.1.0. The crash is a NULL pointer dereference inside ROM memcpy, triggered during TLS data transmission (AT+CIPSEND) when TCP connections are being rapidly opened and closed.

The crash has been reproduced 6 times across 4 different firmware builds, always with the identical register signature. This indicates a structural bug in the AT core library, not a one-off corruption.

Crash Signature (Identical Across All 6 Crashes)

Register	Value	Meaning
MEPC	0x40030c36 (constant)	ROM memcpy — same faulting instruction every time
A0	0x00000000 (constant)	NULL destination pointer passed to memcpy
A2	0x00000008 (constant)	memcpy length = 8 bytes
MCAUSE	0x00000005	Load access fault
MTVAL	0x00000000	Faulting address = NULL
S4	0x4081f9e8 (constant)	Static AT core buffer (s_pbuffer)

The stack memory in all crashes contains the string at_process_task (task name), confirming the crash always occurs in this FreeRTOS task.

Register Dumps Across Crashes (3 of 6 shown)

Crash	MEPC	RA	A0	A2	S2=S3	ELF Hash
1	0x40030c36	0x42076afc	0x00000000	0x00000008	0x1000	714f0626e
2	0x40030c36	0x42076b06	0x00000000	0x00000008	0x0578	39dd33b33
6	0x40030c36	0x42076b20	0x00000000	0x00000008	0x052b	23890334c

RA varies within ~36 bytes (0x42076afc–0x42076b20) across builds — same function, different link-time offsets. S2=S3 holds the AT+CIPSEND payload size (varies per call).

Representative Crash Dump

Pre-crash activity (rapid TCP socket churn):

I (4450147) at-net: link:0 is on netif:1
I (4537632) at-net: deleted
I (4541082) at-net: link:0 is on netif:1
I (4586346) at-net: deleted
I (4588697) at-net: link:0 is on netif:1
W (4598122) at-net: fd:50 errno:0 (Success)
W (4598123) at-net: send failed, s:5515 r:0
I (4600540) at-net: deleted
I (4604671) at-net: link:0 is on netif:1
I (4655359) at-net: link:2 is on netif:1

Crash:

Guru Meditation Error: Core  0 panic'ed (Load access fault). Exception was unhandled.

Core  0 register dump:
MEPC    : 0x40030c36  RA      : 0x42076b20  SP      : 0x4083f190  GP      : 0x4081b024
TP      : 0x4083f300  T0      : 0x40030dca  T1      : 0x00000000  T2      : 0xffffffff
S0/FP   : 0x40844390  S1      : 0x40844718  A0      : 0x00000000  A1      : 0x40844718
A2      : 0x00000008  A3      : 0x00000099  A4      : 0x00000000  A5      : 0x00000003
A6      : 0x00000000  A7      : 0x00000000  S2      : 0x0000052b  S3      : 0x0000052b
S4      : 0x4081f9e8  S5      : 0x00000000  S6      : 0x00000000  S7      : 0x00000000
S8      : 0x00000000  S9      : 0x00000000  S10     : 0x00000000  S11     : 0x00000000
T3      : 0x80000000  T4      : 0xff000fff  T5      : 0x193dc918  T6      : 0xf0000000
MSTATUS : 0x00001881  MTVEC   : 0x40800001  MCAUSE  : 0x00000005  MTVAL   : 0x00000000
MHARTID : 0x00000000

Stack memory:
4083f190: 0x0000052b 0x408209e8 0x40844390 0x42077bd4 0x408209e8 0x40844390 0x00000000 0x4207b43e
4083f1b0: 0x00000010 0x4203a9a8 0x4083f1f4 0x40844390 0x0000152b 0x00001000 0x00001000 0x42071bcc
4083f1d0: 0x00000010 0x00000000 0x16ccd4dc 0x00000001 0x4081f9e8 0x0000152b 0x40840a3c 0x42071792
4083f1f0: 0x4081f9e8 0x00000032 0x00000fff 0x4201b2ce 0x4083f230 0x00000010 0x00000000 0x408314f8
4083f210: 0x00000008 0x00000000 0x16ccd4dc 0x00000001 0x4081f9e8 0x0000152b 0x00000000 0x4201f77c
4083f230: 0x00000003 0x00000000 0x00000000 0x00000000 0x65520a0d 0x35207663 0x20393134 0x65747962
4083f250: 0x000a0d73 0x00000000 0x4083d9c8 0x420e9550 0x00000004 0x00000000 0x00000007 0xffffffff

ELF file SHA256: 23890334c
================= CORE DUMP START =================

Root Cause Analysis

Based on register analysis and stack traces, the reconstructed crash call chain is:

at_process_task
  → at_sending_data
    → at_write_data_to_link
      → esp_tls_conn_write
        → __wrap_mbedtls_ssl_write
          → mbedtls_ssl_write → ssl_write_real
            → memcpy(ssl->out_msg, buf, 8)  ← CRASH: out_msg == NULL

The race condition: A TCP connection is torn down (link "deleted") while at_process_task is still processing a queued AT+CIPSEND for that link. The TLS context (esp_tls_t) is freed or partially cleaned up by a network event callback on another task, but at_process_task still holds a stale reference. When it reaches ssl_write_real, ssl->out_msg is NULL (the dynamic TX buffer was freed or never allocated), and memcpy(NULL, buf, 8) crashes in ROM.

Evidence supporting this analysis:

• A0=0x00000000 (NULL) is the memcpy destination — ssl->out_msg
• A2=0x00000008 is the memcpy size (8 bytes = mbedTLS counter/sequence number copy, not the full payload)
• S2=S3 holds the actual payload size (varies: 0x578, 0x6c5, 0x1000, etc.)
• MEPC=0x40030c36 is inside ROM memcpy (constant across all crashes and builds)
• Pre-crash logs show rapid link creation/deletion cycling (6+ cycles in 60 seconds)
• errno:0 (Success) warnings on send failures indicate stale file descriptors

Trigger Conditions

1. Multiple TLS connections (AT+CIPSTART=0,"SSL",... and AT+CIPSTART=2,"SSL",...) opened concurrently
2. Network instability causes connections to drop (errno:113 ECONNABORTED, or server-side close)
3. Rapid reconnection cycling: link open → send → connection lost → link deleted → link re-opened (6+ cycles in ~60 seconds)
4. AT+CIPSEND is queued while a link is being torn down → at_process_task dequeues and attempts the send on a freed TLS context

Uptime Before Crash

Crash #	Uptime	Notes
1	36.8 min
2	3.5 min
3	1.7 min
4	42 sec	Shortest — highest link churn rate
5	2.3 hours
6	1.3 hours

The highly variable uptime (42 seconds to 2.3 hours) is consistent with a race condition whose trigger depends on specific network event timing.

Suggested Fix

In at_write_data_to_link (inside libesp32c6_at_core.a), before calling esp_tls_conn_write:

1. Validate the link is still active — check that the link's socket fd is not -1 and the TLS handle is not NULL
2. Hold the socket mutex during the full send operation including the TLS write, not just the link lookup
3. Drain or invalidate pending AT+CIPSEND commands for a link when it is deleted, preventing stale sends from reaching a freed TLS context
4. Consider a link generation counter — stamp each AT+CIPSEND with the link's generation; discard if the generation doesn't match at send time

As a lighter-weight defense-in-depth fix, a NULL check for ssl->out_msg before the memcpy in ssl_write_real (or in the __wrap_mbedtls_ssl_write wrapper) would prevent the crash, though the underlying race would still exist.

Workarounds We're Applying

• Reducing CONFIG_LWIP_TCP_MSL to 10000 (from 60000) to speed up TIME_WAIT cleanup and reduce FD pool exhaustion
• Reducing CONFIG_LWIP_TCP_FIN_WAIT_TIMEOUT to 5000 (from 20000)
• Increasing CONFIG_AT_PROCESS_TASK_STACK_SIZE to 8192 (from 6144)
• Adding application-layer backoff before reconnecting after connection failures

These narrow the race window but do not eliminate the root cause.

[ustccw]

@talnirnx
Hi, thank you very much for the detailed information and thorough analysis. This is a very clear and well-structured issue report, which is extremely helpful for us. We will take a deeper look into this issue next week.

Just to share some initial thoughts: in the current implementation, both the send operation and connection close are protected by internal locking mechanisms to avoid race conditions. Additionally, the connection state is checked before sending data.

Given this, the issue seems to be caused by a deeper underlying problem, and we will try to reproduce it on our side for further investigation.

Thanks again for your valuable report — we'll keep you updated on any progress.

[Luxin812]

@talnirnx
I am trying to reproduce the issue you reported.
I have run a 2-hour stress test with the C6 release v4.1.1.0 AT version but have not been able to reproduce the problem so far.
I would like to share my reproduction steps with you and would appreciate it if you could help check if there are any differences from your test steps. Thank you.
Reproduction Steps
Use two C6 modules flashed with release V4.1.1.0 AT version, acting as SSL server (AT1) and SSL client (AT2) respectively:
AT1: AT+CWMODE=3
AT2: AT+CWMODE=1
AT[1,2]: AT+CWJAP="ap_ssid","ap_password"
AT[1,2]: AT+CIPMUX=1
AT1: AT+CIPSERVER=1,8443,"SSL", wait for SSL server to be created successfully
AT2: AT+CIPSTART=1,"SSL",<AT1_sta_ip>,8443, wait for SSL connection to be established successfully via linkid 1
AT2: AT+CIPSTART=2,"SSL",<AT1_sta_ip>,8443, wait for SSL connection to be established successfully via linkid 2
AT2: AT+CIPSTART=0,"SSL",<AT1_sta_ip>,8443, wait for SSL connection to be established successfully via linkid 0
AT2: AT+CIPSEND=0,5515, wait for the > response
AT1: AT+CIPCLOSE=2, close the third connection between AT1 and AT2
AT2: Send 5515 bytes of data to AT1
Repeat steps 8–11 for 1000 cycles