[Help] Random "peer closed connection without sending complete message body" error on ECS self-hosted Dify v1.11.1

[miyarappo]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I am experiencing random "peer closed connection without sending complete message body (incomplete chunked read)" errors when executing chat workflows via API on self-hosted Dify v1.11.1.

Environment:

• Dify version: 1.11.1
• Deployment: Self-hosted on AWS ECS using the official AWS CDK template (https://github.com/aws-samples/dify-self-hosted-on-aws)
• Usage: Chat workflow API execution

Error message:

Run failed: Failed to transform agent message: req_id: 70b0c0e5af PluginInvokeError: {"args":{},"error_type":"Exception","message":"read llm model failed: request failed: req_id: db91d7f41f PluginInvokeError: {\"args\":{},\"error_type\":\"RemoteProtocolError\",\"message\":\"peer closed connection without sending complete message body (incomplete chunked read)\"}"}

Characteristics:

• The error occurs randomly (not consistently reproducible)
• It happens regardless of token count or processing load
• It occurs specifically during chat workflow API execution

I found similar issues (#9912 and #12802) reported for older versions (v0.9.x), but those were closed as "outdated". I'm wondering if this issue persists in v1.11.1 or if there are any known workarounds for ECS deployments.

2. Additional context or comments

Questions:

1. Is this a known issue in v1.11.1, and is there a fix planned for upcoming releases?
2. Are there any recommended timeout or connection settings for AWS ECS deployments to mitigate this issue?
3. Has anyone successfully resolved this by adjusting environment variables (e.g., API_REQUEST_TIMEOUT)?

Related Issues:

• [azure_openai] Error: peer closed connection without sending complete message body (incomplete chunked read) #9912 - [azure_openai] Error: peer closed connection without sending complete message body
• Error: Peer Closed Connection Without Sending Complete Message Body in Claude 3.5 (Sonnet 20240620) #12802 - Error: Peer Closed Connection Without Sending Complete Message Body in Claude 3.5

Any guidance would be greatly appreciated. Thank you!