DynamoDB returns operation error `use of closed network`

[satorunooshie]

Describe the bug

SDK often returns the error below after updated SDK version.

acutal error:

operation error DynamoDB: Query, $tableName: https response error StatusCode: 200, RequestID: 123456, deserialization failed, failed to decode response body, read tcp $src->$dst: use of closed network connection

Expected Behavior

No errors

Current Behavior

operation error DynamoDB: Query, $tableName: https response error StatusCode: 200, RequestID: 123456, deserialization failed, failed to decode response body, read tcp $src->$dst: use of closed network connection

Reproduction Steps

	ddb := dynamodb.NewFromConfig(defaultCfg)
	result, err := ddb.Query(
		ctx,
		&dynamodb.QueryInput{
			TableName:                 aws.String(tableName),
			ExpressionAttributeNames:  expr.Names(),
			ExpressionAttributeValues: expr.Values(),
			KeyConditionExpression:    expr.KeyCondition(),
		},
	)

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

	github.com/aws/aws-sdk-go-v2 v1.16.8
	github.com/aws/aws-sdk-go-v2/config v1.15.15
	github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue v1.9.8
	github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression v1.4.14
	github.com/aws/aws-sdk-go-v2/service/dynamodb v1.15.10
	github.com/aws/smithy-go v1.12.0

Compiler and Version used

1.19

Operating System and version

Linux

[RanVaknin]

Hi @satorunooshie ,

I saw that you opened 2 issues but Im going to answer them both here:

The reason you are seeing a status 200 followed by a serialization error is that the DynamoDB server is able to send back a valid response, and before the client is finished reading the body of the successful response, the connection errors out. This can happen for many many reasons and is difficult to pinpoint especially when the pattern is inconsistent like in your case.

Regarding your other ticket and PR:
We cannot accept your PR to override the Retryer behavior because not every API operation may be idempotent. The service API models don't have information to determine if an operation is idempotent, thus the SDKs cannot assume if it's safe to retry.

Proposed solution:
You can implement your own Retryer on the client level, or on operation level and configure it to retry when the connection fails.

Let me know if you have any other questions and I can answer them here 🙂.

Thanks you!
~Ran

[github-actions]

This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[SaikyoSaru]

We are seeing the same issue as well, can we please resurrect this issue?

[psmarcin]

I was able to fix this issue but wasn't able to reproduce this error 100% times so I can't create PR.

Before deploying my fix we had ~6 errors per minute, after deploy there is no errors.

Here is my fork of smithy-go: https://github.com/psmarcin/smithy-go there is only one small change - remove conditional body closing.

You can give it a try by simple adding to go.mod file:

replace github.com/aws/smithy-go => github.com/psmarcin/smithy-go v0.0.0-20220923084642-9e31178f1156

[cwd-nial]

@RanVaknin or any other contributor, can you review the solution psmarcin is suggesting? We are having the same issue that is mentioned above:

operation error DynamoDB: Query, https response error StatusCode: 200, RequestID: 1263a6b7-ebec-4a51-9594-b647545da4fa, deserialization failed, failed to decode response body, read tcp 172.17.0.12:36206->172.17.0.15:8000: use of closed network connection\n