You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+28-30Lines changed: 28 additions & 30 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,8 @@
1
-
# AWS C S3
1
+
##AWS C S3
2
2
3
3
The AWS-C-S3 library is an asynchronous AWS S3 client focused on maximizing throughput and network utilization.
4
4
5
-
## Key features:
5
+
###Key features:
6
6
7
7
***Automatic Request Splitting**: Improves throughput by automatically splitting the request into part-sized chunks and performing parallel uploads/downloads of these chunks over multiple connections. There's a cap on the throughput of single S3 connection, the only way to go faster is multiple parallel connections.
8
8
***Automatic Retries**: Increases resilience by retrying individual failed chunks of a file transfer, eliminating the need to restart transfers from scratch after an intermittent error.
@@ -11,16 +11,16 @@ The AWS-C-S3 library is an asynchronous AWS S3 client focused on maximizing thro
11
11
***Thread Pools and Async I/O**: Avoids bottlenecks associated with single-thread processing.
12
12
***Parallel Reads**: When uploading a large file from disk, reads from multiple parts of the file in parallel. This is faster than reading the file sequentially from beginning to end.
13
13
14
-
## Documentation
14
+
###Documentation
15
15
16
16
*[GetObject](docs/GetObject.md): A visual representation of the GetObject request flow.
17
17
*[Memory Aware Requests Execution](docs/memory_aware_request_execution.md): An in-depth guide on optimizing memory usage during request executions.
The S3 client uses a buffer pool to manage memory for concurrent transfers.
26
26
@@ -39,17 +39,15 @@ Example Usage:
39
39
> };
40
40
> ```
41
41
42
-
**Priority**: The configuration value takes precedence over the environment variable. If `memory_limit_in_bytes` is set to a non-zero value in the config, the environment variable is ignored.
42
+
> [!NOTE]
43
+
> * **Priority**: The configuration value takes precedence over the environment variable. If `memory_limit_in_bytes` is set to a non-zero value in the config, the environment variable is ignored.
44
+
> * **Default Behavior**: If neither is set (config is 0 and environment variable is not set), the client sets a default memory limit based on the target throughput.
45
+
> * The limit applies per client. If multiple clients created, limit will apply to each separately.
46
+
> * The environment variable value must be a valid positive integer representing gigabytes (GiB).
47
+
> * The value is converted from GiB to bytes internally (1 GiB = 1024³ bytes).
48
+
> * Invalid values or overflow conditions will cause client creation to fail with `AWS_ERROR_INVALID_ARGUMENT`.
43
49
44
-
**Default Behavior**: If neither is set (config is 0 and environment variable is not set), the client sets a default memory limit based on the target throughput.
45
-
46
-
**Notes**:
47
-
* The limit applies per client. If multiple clients created, limit will apply to each separately.
48
-
* The environment variable value must be a valid positive integer representing gigabytes (GiB).
49
-
* The value is converted from GiB to bytes internally (1 GiB = 1024³ bytes).
50
-
* Invalid values or overflow conditions will cause client creation to fail with `AWS_ERROR_INVALID_ARGUMENT`.
51
-
52
-
#### Maximum Parts Pending Read - `AWS_CRT_S3_MAX_PARTS_PENDING_READ`
50
+
##### Maximum Parts Pending Read - `AWS_CRT_S3_MAX_PARTS_PENDING_READ`
53
51
54
52
Controls the maximum number of parts that can be pending read from the input stream during a multipart upload. Higher values may improve upload throughput for large files by allowing more parts to be read in parallel, at the cost of higher memory usage.
55
53
@@ -59,30 +57,30 @@ Example Usage:
59
57
export AWS_CRT_S3_MAX_PARTS_PENDING_READ=20
60
58
```
61
59
62
-
**Notes**:
63
-
* Only affects multipart uploads. Small files that fit in a single part are not affected.
64
-
* Setting this too low may introduce delays between reads, as the meta-request waits for the client to schedule more work.
65
-
* Setting this too high may cause a single upload to hog work tokens, starving other concurrent uploads.
66
-
* The value must be a positive integer (1–4294967295). Invalid or zero values are ignored with a warning, and the default is used.
67
-
* The value is read once on first use and cached for the lifetime of the process.
60
+
> [!NOTE]
61
+
> * Only affects multipart uploads. Small files that fit in a single part are not affected.
62
+
> * Setting this too low may introduce delays between reads, as the meta-request waits for the client to schedule more work.
63
+
> * Setting this too high may cause a single upload to hog work tokens, starving other concurrent uploads.
64
+
> * The value must be a positive integer (1–4294967295). Invalid or zero values are ignored with a warning, and the default is used.
65
+
> * The value is read once on first use and cached for the lifetime of the process.
68
66
69
-
#### Test Bucket - `CRT_S3_TEST_BUCKET_NAME`
67
+
#####Test Bucket - `CRT_S3_TEST_BUCKET_NAME`
70
68
71
69
The S3 bucket name used for running unit tests. See the [test_helper documentation](./tests/test_helper/) for setup instructions.
72
70
73
-
# License
71
+
##License
74
72
75
73
This library is licensed under the Apache 2.0 License.
76
74
77
-
# Usage
75
+
##Usage
78
76
79
-
## Building
77
+
###Building
80
78
81
79
CMake 3.9+ is required to build.
82
80
83
81
`<install-path>` must be an absolute path in the following instructions.
84
82
85
-
### Linux-Only Dependencies
83
+
####Linux-Only Dependencies
86
84
87
85
If you are building on Linux, you will need to build aws-lc and s2n-tls first.
0 commit comments