Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat S3 Transfer Manager v2 GetObject/DownloadObject #2996

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
c4bf70d
Feat s3 transfer manager v2 PutObject (#2733)
wty-Bryant Nov 7, 2024
5370c7c
move transfer manager v2 integration tests to within module (#2987)
lucix-aws Jan 28, 2025
31cf6b3
draft apis for downloader
Nov 14, 2024
cdf6d16
commit changes for now
Nov 15, 2024
5470be3
draft commit
Nov 19, 2024
122488a
draft commit
Nov 26, 2024
1e9a6a1
draft commit
Nov 27, 2024
9d93b24
draft commit
Dec 3, 2024
505167d
draft commit
Dec 4, 2024
3a2e7f8
draft code for downloader
Dec 18, 2024
ee21a65
draft code of test
Dec 23, 2024
15ac689
draft code for getobject test
wty-Bryant Dec 24, 2024
757b9ad
add draft downloader unit test
wty-Bryant Jan 6, 2025
6786b29
update getobject test
wty-Bryant Jan 7, 2025
2830a79
update getobject tests
wty-Bryant Jan 10, 2025
3ec5c9f
update getobject test
wty-Bryant Jan 13, 2025
19ed520
update getobject test
wty-Bryant Jan 16, 2025
75f4938
draft code
wty-Bryant Jan 27, 2025
fd921cb
draft code
wty-Bryant Jan 27, 2025
2eb0471
draft code
wty-Bryant Jan 28, 2025
9d49074
fix chunk writer issue
wty-Bryant Jan 29, 2025
cd4e653
add integ test for getobject
wty-Bryant Jan 30, 2025
37dbf1b
add default checksum behavior to putobject
wty-Bryant Jan 30, 2025
d0cad4c
add default checksum behavior to getobject
wty-Bryant Jan 31, 2025
1f29cd3
remove unrelated go mod change
wty-Bryant Jan 31, 2025
4399690
add comment for exported func and type
wty-Bryant Jan 31, 2025
1cfc1b5
add comment for exported type
wty-Bryant Jan 31, 2025
76311f3
change comment and remove extra error unwrap func
wty-Bryant Feb 3, 2025
93559ed
incomplete draft code
wty-Bryant Feb 7, 2025
dd3bcc7
split downloader and getter
wty-Bryant Feb 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions feature/s3/transfermanager/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,5 @@ type S3APIClient interface {
CreateMultipartUpload(context.Context, *s3.CreateMultipartUploadInput, ...func(*s3.Options)) (*s3.CreateMultipartUploadOutput, error)
CompleteMultipartUpload(context.Context, *s3.CompleteMultipartUploadInput, ...func(*s3.Options)) (*s3.CompleteMultipartUploadOutput, error)
AbortMultipartUpload(context.Context, *s3.AbortMultipartUploadInput, ...func(*s3.Options)) (*s3.AbortMultipartUploadOutput, error)
GetObject(context.Context, *s3.GetObjectInput, ...func(*s3.Options)) (*s3.GetObjectOutput, error)
}
4 changes: 4 additions & 0 deletions feature/s3/transfermanager/api_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ const defaultMultipartUploadThreshold = 1024 * 1024 * 16
// using PutObject().
const defaultTransferConcurrency = 5

const defaultPartBodyMaxRetries = 3
wty-Bryant marked this conversation as resolved.
Show resolved Hide resolved

// Client provides the API client to make operations call for Amazon Simple
// Storage Service's Transfer Manager
// It is safe to call Client methods concurrently across goroutines.
Expand All @@ -39,6 +41,8 @@ func New(s3Client S3APIClient, opts Options, optFns ...func(*Options)) *Client {
resolvePartSizeBytes(&opts)
resolveChecksumAlgorithm(&opts)
resolveMultipartUploadThreshold(&opts)
resolveMultipartDownloadType(&opts)
resolvePartBodyMaxRetries(&opts)

return &Client{
options: opts,
Expand Down
880 changes: 880 additions & 0 deletions feature/s3/transfermanager/api_op_DownloadObject.go

Large diffs are not rendered by default.

24 changes: 24 additions & 0 deletions feature/s3/transfermanager/api_op_DownloadObject_integ_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
//go:build integration
// +build integration

package transfermanager

import (
"bytes"
"strings"
"testing"
)

func TestInteg_DownloadObject(t *testing.T) {
cases := map[string]getObjectTestData{
"seekable body": {Body: strings.NewReader("hello world"), ExpectBody: []byte("hello world")},
"empty string body": {Body: strings.NewReader(""), ExpectBody: []byte("")},
"multipart download body": {Body: bytes.NewReader(largeObjectBuf), ExpectBody: largeObjectBuf},
}

for name, c := range cases {
t.Run(name, func(t *testing.T) {
testDownloadObject(t, setupMetadata.Buckets.Source.Name, c)
})
}
}
Loading