When a peer sends a frame whose Content-Length exceeds max_payload_bytes (hardcoded 64MB in src/ipc/transport.cpp), read_message stops the stream and returns the same nullopt as a real EOF. The receiver can't tell "this one message is too large" from "the connection died" — every pending request fails with the generic "transport closed".
Hit this in production with clice: a worker's index result for a big TU can exceed 64MB while being delivered normally. The master saw "transport closed", assumed the worker died, killed it and requeued — forever. Results up to 63.8MB delivered fine; anything bigger failed 100%.
Repro: frame a >64MB payload over a StreamTransport pair and read it.
We run a patch locally (drain the oversized frame, fail that request with a distinct error, make the cap configurable — the closed #200). Happy to rework it in whatever direction you land on.
When a peer sends a frame whose
Content-Lengthexceedsmax_payload_bytes(hardcoded 64MB insrc/ipc/transport.cpp),read_messagestops the stream and returns the samenulloptas a real EOF. The receiver can't tell "this one message is too large" from "the connection died" — every pending request fails with the generic "transport closed".Hit this in production with clice: a worker's index result for a big TU can exceed 64MB while being delivered normally. The master saw "transport closed", assumed the worker died, killed it and requeued — forever. Results up to 63.8MB delivered fine; anything bigger failed 100%.
Repro: frame a >64MB payload over a
StreamTransportpair and read it.We run a patch locally (drain the oversized frame, fail that request with a distinct error, make the cap configurable — the closed #200). Happy to rework it in whatever direction you land on.