Add support for UploadPartCopy on multipart part uploads - #20
Add support for UploadPartCopy on multipart part uploads#20Pastalikek65 wants to merge 3 commits into
Conversation
UploadPartCopy sends the copy source in an X-Amz-Copy-Source header and an empty body, but putMultipartUploadPart treated it like a regular part upload: it stored the empty body and answered with a bare ETag, so S3 clients hit a deserialization error and multipart server-side copies always failed. putMultipartUploadPart now branches on X-Amz-Copy-Source and copies the source object - or the range from X-Amz-Copy-Source-Range - into the part slot through the same streaming/in-memory path as a regular part, and answers with a CopyPartResult (200, or 206 when a range was copied). Ranged copies resolve the range against the source size from HeadObject and read the bytes via GetObject, so backends don't need to honor range requests themselves.
The in-memory s3mem backend never exercised the part-copy path for backends that implement MultipartBackend: the copy request bypasses AddPart entirely and lands in multipart.UploadPart via the streaming wrapper. Covers that dispatch with a round-trip test so a regression in the streaming copy path is caught without spinning up rclone.
|
Checked this against the four points from rclone/rclone#7454: the streaming path and the missing Two things reproduce against 1. The copy branch sits below the Content-Length read, so a body-less UploadPartCopy answers 411. The new guard is at This is not reachable from a Go client: net/http always sends 2. A ranged copy answers 206. With S3 answers 200 with a |
Two review findings against f5d477d: - UploadPartCopy was parsed as a body-bearing request: the Content-Length guard ran before the copy branch, so a request without a Content-Length header (valid, since the part is copied server-side and carries no body) failed with 411. The copy branch now runs first and reads X-Amz-Copy-Source from the request headers directly. - A ranged UploadPartCopy answered 206 Partial Content. S3 answers 200 with a CopyPartResult body in both cases; the range is an input parameter, not a partial response. Adds tests for both cases.
|
Thanks for the thorough review — both findings reproduce against f5d477d exactly as you described, and both are fixed in 328d50b. 1. Body-less UploadPartCopy now works The copy branch was sitting below the Content-Length read, so a request without a 2. Ranged copy answers 200 The Added |
Multipart server-side copies (rclone/rclone#7454) fail against gofakes3:
UploadPartCopysends the copy source in anX-Amz-Copy-Sourceheader and an empty body, butputMultipartUploadParthandled it like a regular part upload - stored the empty body and answered with a bareETag, which S3 clients can't deserialize as aCopyPartResult.This adds the copy-source branch to
putMultipartUploadPart(mirroring the single-objectPUTpath), copying the source object or the range fromX-Amz-Copy-Source-Rangeinto the part slot through the same streaming/in-memory path as a regular part, and answering withCopyPartResult(200, or 206 for ranged).What I tested:
TestUploadPartCopyandTestUploadPartCopyRange(gofakes3 unit tests, in-memory backend): both fail on master withdeserialization failed, received empty response payloadand pass with the change.go test ./...+go veton the repo.cmd/serve/s3regression test drivingUploadPartCopywith the aws-sdk (the exact request rclone's s3 backend makes for chunked copies) fails against gofakes3 v0.0.7 and passes against this branch. I'll send the rclone side (dep bump + test) once this is merged.What I did not test: conditional copy headers (
x-amz-copy-source-if-matchetc.) - the single-object copy path doesn't support them either, so I left them out for parity.