Skip to content
54 changes: 22 additions & 32 deletions docs/troubleshooting/blob-size-limit-error.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
id: blob-size-limit-error
title: Troubleshoot the blob size limit error
sidebar_label: Blob size limit error
description: The BlobSizeLimitError occurs when a Workflow's payload exceeds the 2 MB request limit or the 4 MB Event History transaction limit set by Temporal. Reduce blob size via compression or batching.
description: Temporal enforces size limits on data passed between Workers and the Temporal Service. Learn about the 2 MB payload limit and 4 MB gRPC message limit, their error messages, and how to resolve them.
toc_max_heading_level: 4
keywords:
- error
Expand All @@ -12,48 +12,38 @@ tags:
- Failures
---

The `BlobSizeLimitError` is an error that occurs when the size of a blob (payloads including Workflow context and each Workflow and Activity argument and return value) exceeds the set limit in Temporal.
Temporal enforces size limits on the data that passes between the Temporal Client, Workers, and the Temporal Service. There are two distinct limits: a 2 MB limit on individual Payloads enforced by the Temporal Service, and a 4 MB limit on entire gRPC messages enforced by the transport layer. Each produces different error messages and behaviors.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Temporal enforces size limits on the data that passes between the Temporal Client, Workers, and the Temporal Service. There are two distinct limits: a 2 MB limit on individual Payloads enforced by the Temporal Service, and a 4 MB limit on entire gRPC messages enforced by the transport layer. Each produces different error messages and behaviors.
Temporal enforces size limits on the data that passes between the Temporal Client, Workers, and the Temporal Service. There are two distinct limits: a 2 MB limit on individual Payloads enforced by the Temporal Service, and a 4 MB limit on entire gRPC messages enforced by the transport layer. Each produces different error messages and behaviors, and requires different solutions.


- The max payload for a single request is 2 MB.
- The max size limit for any given [Event History](/workflow-execution/event#event-history) transaction is 4 MB.
## Payload size limit

## Why does this error occur?
The Temporal Service enforces a 2 MB size limit on individual Payloads. A Payload represents the serialized binary data for each input and output of Workflows and Activities. When a single Payload exceeds this limit, the Temporal Service rejects the request and terminates the Workflow.

This error occurs when the size of the blob exceeds the maximum size allowed by Temporal.
The error message depends on which operation carried the oversized Payload. Examples include:
- `BadScheduleActivityAttributes: ScheduleActivityTaskCommandAttributes.Input exceeds size limit`
- `WORKFLOW_TASK_FAILED_CAUSE_BAD_UPDATE_WORKFLOW_EXECUTION_MESSAGE`

This limit helps ensure that the Temporal Service prevents excessive resource usage and potential performance issues when handling large payloads.
### How to resolve

## How do I resolve this error?
1. Use compression with a [custom Payload Codec](/payload-codec) for large payloads. This addresses the immediate issue, but if payload sizes continue to grow, the problem can arise again.

To resolve this error, reduce the size of the blob so that it is within the 4 MB limit.

There are multiple strategies you can use to avoid this error:

1. Use compression with a [custom payload codec](/payload-codec) for large payloads.

- This addresses the immediate issue of the blob size limit; however, if blob sizes continue to grow this problem can arise again.

2. Break larger batches of commands into smaller batch sizes:

- Workflow-level batching:
1. Modify the Workflow to process Activities or Child Workflows into smaller batches.
2. Iterate through each batch, waiting for completion before moving to the next.
- Workflow Task-level batching:
1. Execute Activities in smaller batches within a single Workflow Task.
2. Introduce brief pauses or sleeps (for example, 1ms) between batches.

3. Consider offloading large payloads to an object store to reduce the risk of exceeding blob size limits:
2. Offload large payloads to an external object store:
1. Pass references to the stored payloads within the Workflow instead of the actual data.
2. Retrieve the payloads from the object store when needed during execution.

## Workflow termination due to oversized response
## gRPC message size limit

When a Workflow Task response exceeds the 4 MB gRPC message size limit, Temporal automatically terminates the Workflow Execution. This is a non-recoverable error. The Workflow can't progress if it generates a response that's too large, so retrying won't help.
The gRPC transport layer enforces a 4 MB limit on the entire message between a Worker and the Temporal Service. This limit applies to the full message, not individual Payloads. A Workflow can exceed this limit even when every individual Payload is under 2 MB, if the Workflow schedules enough commands in a single Workflow Task that their combined size exceeds 4 MB.

This typically happens when a Workflow schedules too many Activities, Child Workflows, or other commands in a single Workflow Task. The total size of all commands generated by the Workflow Task must fit within the 4 MB limit.
When a message exceeds the 4 MB gRPC limit, the Worker can't deliver the response to the Temporal Service. The Workflow Task fails with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. Because replay produces the same oversized response, the Workflow gets stuck in a retry loop that isn't visible in the Event History.

If your Workflow was terminated for this reason, you'll see a `WorkflowExecutionTerminated` event in the Event History with the cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`.
For Activity results that exceed the gRPC limit, the symptom is different. The Activity completes successfully on the Worker, but the Worker can't deliver the result to the Temporal Service. The server never receives the completion, so the Activity eventually times out with a `StartToCloseTimeout`. The actual `ResourceExhausted` gRPC error only appears in Worker logs.

To prevent this, use the batching strategies described above to split work across multiple Workflow Tasks instead of scheduling everything at once.
### How to resolve

See the [gRPC Message Too Large error reference](/references/errors#grpc-message-too-large) for more details.
Break larger batches of commands into smaller batch sizes:
- Workflow-level batching:
1. Modify the Workflow to process Activities or Child Workflows in smaller batches.
2. Iterate through each batch, waiting for completion before moving to the next.
- Workflow Task-level batching:
1. Execute Activities in smaller batches within a single Workflow Task.
2. Introduce brief pauses or sleeps between batches.
Loading