Ensure that we don't send massive error message over gRPC - #1108
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
🟡 Oversized non-fatal error messages can still exceed the gRPC limit and fail the job
The non-fatal error message and details are sent to the controller untruncated (error: message/details at crates/arroyo-worker/src/lib.rs:605-608), unlike the fatal path which was capped, so a large error can still blow past the 4MB gRPC message limit.
Impact: A single oversized non-fatal error (e.g. from bad input data) makes the RPC fail, which cancels the worker and takes the whole job down — the exact scenario this change was meant to prevent.
Why the non-fatal path is also at risk
The fatal path at crates/arroyo-worker/src/lib.rs:579-586 now wraps both error and details in maybe_truncate(..., MAX_TASK_ERROR_FIELD_BYTES). The NonfatalErrorReq built at crates/arroyo-worker/src/lib.rs:597-610 assigns error: message and details directly with no size guard. The details value originates from deserialization/bad-data handling (crates/arroyo-operator/src/context.rs:389), which can embed raw offending record contents and thus be arbitrarily large. When such a message exceeds the 4MB limit, send_control_rpc! returns an error and cancel_token.cancel() is invoked (crates/arroyo-worker/src/lib.rs:637-643), terminating the job.
(Refers to lines 605-608)
Was this helpful? React with 👍 or 👎 to provide feedback.
The gRPC servers are configured with a max size of 4MB, this patch adds more resiliency for potentially large error message that would bust the gRPC message size limit.
f551334 to
854efc9
Compare
| "Truncated oversized String from {} bytes to {} bytes: {}", | ||
| original_bytes, | ||
| value.len(), | ||
| value |
There was a problem hiding this comment.
If I read this right, these logs will be ~64kB of unknown data. Is that something we want to include in the log?
There was a problem hiding this comment.
I decided to log them as they can be useful to understand the root cause. In theory (🤞), we shouldn't really have this truncation happen often, if at all.
3 participants
The gRPC servers are configured with a max size of 4MB, this patch adds more resiliency for potentially large error message that would bust the gRPC message size limit.