Skip to content

Commit 9de5d66

Browse files
committed
backend/s3: Fail fast multipart upload
When a part upload request fails error is returned and gCtx is cancelled. This does not prevent from other parts being tried. They immediately fail due to a canceled context, but are retried by rclone anyway... Example AWS debug output ``` ----------------------------------------------------- 2020/02/11 14:12:17 DEBUG: Retrying Request s3/UploadPart, attempt 4 2020/02/11 14:12:17 DEBUG: Request s3/UploadPart Details: ---[ REQUEST POST-SIGN ]----------------------------- PUT /backuptest-rclone/huge/file.db?partNumber=11&uploadId=190939b4-3c43-4b98-ac11-92303e3f11b0 HTTP/1.1 Host: 192.168.100.99:9000 User-Agent: aws-sdk-go/1.23.8 (go1.13.1; linux; amd64) Content-Length: 5242880 Authorization: AWS4-HMAC-SHA256 Credential=miniouser/20200211/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-md5;expect;host;x-amz-content-sha256;x-amz-date, Signature=3fc03a01f651cec09b05290459e9ceb26db9a8aa00c4e1b16e8cf5617eb81da8 Content-Md5: XzY+DlipXwbL6bvGYsXftg== Expect: 100-Continue X-Amz-Content-Sha256: c036cbb7553a909f8b8877d4461924307f27ecb66cff928eeeafd569c3887e29 X-Amz-Date: 20200211T131217Z Accept-Encoding: gzip ----------------------------------------------------- http://192.168.100.99:9000/backuptest-rclone/huge/file.db?partNumber=11&uploadId=190939b4-3c43-4b98-ac11-92303e3f11b0 2020/02/11 14:12:17 DEBUG: Response s3/UploadPart Details: ---[ RESPONSE ]-------------------------------------- HTTP/1.1 500 InternalServerError Content-Length: 0 ----------------------------------------------------- UploadPartWithContext() error InternalError: We encountered an internal error. Please try again status code: 500, request id: , host id: 2020/02/11 14:12:18 DEBUG ERROR: Request s3/UploadPart: ---[ REQUEST DUMP ERROR ]----------------------------- context canceled ------------------------------------------------------ UploadPartWithContext() error RequestCanceled: request context canceled caused by: context canceled 2020/02/11 14:12:20 DEBUG ERROR: Request s3/UploadPart: ---[ REQUEST DUMP ERROR ]----------------------------- context canceled ------------------------------------------------------ UploadPartWithContext() error RequestCanceled: request context canceled caused by: context canceled 2020/02/11 14:12:22 DEBUG ERROR: Request s3/UploadPart: ---[ REQUEST DUMP ERROR ]----------------------------- context canceled ------------------------------------------------------ UploadPartWithContext() error RequestCanceled: request context canceled caused by: context canceled ``` This adds a fail fast behaviour in case the context was cancelled.
1 parent c12371e commit 9de5d66

File tree

1 file changed

+6
-0
lines changed

1 file changed

+6
-0
lines changed

backend/s3/s3.go

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2117,6 +2117,12 @@ func (o *Object) uploadMultipart(ctx context.Context, req *s3.PutObjectInput, si
21172117
buf = make([]byte, partSize)
21182118
}
21192119

2120+
// Fail fast, in case an errgroup managed function returns an error
2121+
// gCtx is cancelled. There is no point in uploading all the other parts.
2122+
if gCtx.Err() != nil {
2123+
break
2124+
}
2125+
21202126
// Read the chunk
21212127
var n int
21222128
n, err = readers.ReadFill(in, buf) // this can never return 0, nil

0 commit comments

Comments
 (0)