Corrupting WebSocket stream

[inF1704]

Platform

ESP32

IDE / Tooling

pioarduino

What happened?

Description

Under TCP send-buffer pressure, webSocketSendFrame() can leave a torn frame on the
wire, permanently corrupting the WebSocket stream for that connection. The client then
reports Received start of new message but previous message is unfinished (Chrome),
closes with 1006, and — since most frontends auto-reconnect — the resulting reconnect
burst re-triggers the condition, producing a reconnect storm.

Environment

• ESPAsyncWebServer v3.11.2 (latest at time of writing; bug present on main)
• AsyncTCP v3.4.10
• ESP32-S3 (Pioarduino - 55.03.35)
• Reproduced with 2 concurrent WS clients receiving several large (4–8 KB) text
  messages plus periodic small binary broadcasts right after connect

Root cause

webSocketSendFrame() (src/AsyncWebSocket.cpp, ~line 51) writes the frame header and
payload as two separate client->add() calls. The failure paths return 0 even when
part of the frame has already been committed to the TCP stream:

if (client->add((const char *)buf, headLen) != headLen) {   // header committed on success
  free(buf);
  return 0;
}
free(buf);

if (len) {
  ...
  if (client->add((const char *)data, len) != len) {        // ← header IS already in the
    return 0;                                               //   TCP stream, but we return 0
  }
}
if (!client->send()) {                                      // ← header+payload are queued
  return 0;                                                 //   in lwIP, but we return 0
}

client->add() can accept fewer bytes than client->space() promised (lwIP segment/
pbuf exhaustion, ERR_MEM), so this is not just theoretical — it fires reliably under
burst load.

The caller AsyncWebSocketMessage::send() (~line 229) then rolls back its offsets:

size_t sent = webSocketSendFrame(client, final, opCode, _mask, dPtr, toSend);
if (toSend && sent != toSend) {
  _sent -= (toSend - sent);
  _ack -= (toSend - sent);
}

On the next _runQueue() pass it re-sends from the rolled-back offset with a new frame
header (opCode = (toSend && _sent == toSend) ? _opcode : WS_CONTINUATION, ~line 227).

The receiver now sees: header(N bytes announced) + partial payload + another frame
header injected mid-payload + payload restarting from an earlier offset. Decoded
symptoms on the client:

• text messages whose content repeats a segment from ~2 KB earlier (JSON parse errors)
• WebSocket connection failed: Received start of new message but previous message is unfinished
• close code 1006, followed by reconnect loops

The server side logs the mirror image on the next RX:
ERROR: 1002 - Previous data frame unfinished.

The !client->send() path is a second, independent corruption source: at that point the
frame is queued in lwIP and will be transmitted by the next poll/ack cycle anyway —
returning 0 causes the same rollback and a duplicate re-send under a fresh header.

Reproduction sketch

1. Serve several multi-KB WS text messages immediately on connect (layout/state dumps)
   plus a periodic broadcast, to 2+ clients simultaneously.
2. Keep lwIP under pressure (default TCP_SND_QUEUELEN, several connections).
3. Reload the clients repeatedly. Within a few connects, Chrome reports
   Received start of new message but previous message is unfinished and the payloads
   show duplicated segments.

Suggested fix

Frames must be all-or-nothing. We run the following local patch successfully:

• If the header add() fails/partials: nothing usable is on the wire → return 0
  (current behaviour is acceptable here only if add() is all-or-nothing for ≤8 bytes;
  aborting is the safe option).
• If the payload add() fails/partials after the header was committed: the stream
  is unrecoverable → client->abort(). An RST is cleanly recoverable for the peer
  (auto-reconnect), a torn frame stream is not.
• A failing client->send() is not an error: the data is queued and will flush via
  the poll/ack path. Return len instead of 0 to prevent the rollback/duplicate.

An alternative (nicer, more invasive) fix would be tracking "payload bytes still owed to
the current on-wire frame header" in AsyncWebSocketMessage and continuing header-less —
but abort-on-torn is minimal and turns silent corruption into a clean, rare reconnect.

Happy to submit a PR with the minimal variant if you agree with the approach.

Stack Trace

No stack trace needed

Minimal Reproductible Example (MRE)

not needed here

I confirm that:

• I have read the documentation.
• I have searched for similar discussions.
• I have searched for similar issues.
• I have looked at the examples.
• I have upgraded to the lasted version of ESPAsyncWebServer (and AsyncTCP for ESP32).

[mathieucarbou]

HI @inF1704
Thank you for this detailed bug report.
Yes, please send a PR if you have a fix in mind. We will have a look and review.
The WebSocket code is quite complex, especially how it is historically implemented in this project, so it is possible that some corner issues are still there.
I understand also that a MRE is also not easy to develop for this use case. I will try your test procedure.