Merge branch 'main' into get_size

Author: DmitriyMusatkin

.github/workflows/ci.yml

@@ -223,6 +223,8 @@ jobs:
       uses: actions/checkout@v4
     - name: Build ${{ env.PACKAGE_NAME }} + consumers
       run: |
+        python3 -m venv .venv
+        source .venv/bin/activate
         python3 -c "from urllib.request import urlretrieve; urlretrieve('${{ env.BUILDER_HOST }}/${{ env.BUILDER_SOURCE }}/${{ env.BUILDER_VERSION }}/builder.pyz?run=${{ env.RUN }}', 'builder')"
         chmod a+x builder
         ./builder build -p ${{ env.PACKAGE_NAME }} --cmake-extra=-DASSERT_LOCK_HELD=ON

.gitignore

@@ -71,3 +71,4 @@ benchmarks/dashboard-stack/package-lock.json
 
 # virtual environment
 .venv/
+.cache/

CMakeLists.txt

@@ -108,6 +108,7 @@ install(FILES "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}-config.cmake"
 
 include(CTest)
 if (BUILD_TESTING)
+    add_definitions(-DAWS_C_S3_ENABLE_TEST_STUBS)
     add_subdirectory(tests)
     if (NOT BYO_CRYPTO AND NOT CMAKE_CROSSCOMPILING)
         add_subdirectory(samples)

README.md

@@ -3,17 +3,49 @@
 The AWS-C-S3 library is an asynchronous AWS S3 client focused on maximizing throughput and network utilization.
 
 ### Key features:
-- **Automatic Request Splitting**: Improves throughput by automatically splitting the request into part-sized chunks and performing parallel uploads/downloads of these chunks over multiple connections. There's a cap on the throughput of single S3 connection, the only way to go faster is multiple parallel connections.
-- **Automatic Retries**: Increases resilience by retrying individual failed chunks of a file transfer, eliminating the need to restart transfers from scratch after an intermittent error.
-- **DNS Load Balancing**: DNS resolver continuously harvests Amazon S3 IP addresses. When load is spread across the S3 fleet, overall throughput more reliable than if all connections are going to a single IP.
-- **Advanced Network Management**: The client incorporates automatic request parallelization, effective timeouts and retries, and efficient connection reuse. This approach helps to maximize throughput and network utilization, and to avoid network overloads.
-- **Thread Pools and Async I/O**: Avoids bottlenecks associated with single-thread processing.
-- **Parallel Reads**: When uploading a large file from disk, reads from multiple parts of the file in parallel. This is faster than reading the file sequentially from beginning to end.
+
+* **Automatic Request Splitting**: Improves throughput by automatically splitting the request into part-sized chunks and performing parallel uploads/downloads of these chunks over multiple connections. There's a cap on the throughput of single S3 connection, the only way to go faster is multiple parallel connections.
+* **Automatic Retries**: Increases resilience by retrying individual failed chunks of a file transfer, eliminating the need to restart transfers from scratch after an intermittent error.
+* **DNS Load Balancing**: DNS resolver continuously harvests Amazon S3 IP addresses. When load is spread across the S3 fleet, overall throughput more reliable than if all connections are going to a single IP.
+* **Advanced Network Management**: The client incorporates automatic request parallelization, effective timeouts and retries, and efficient connection reuse. This approach helps to maximize throughput and network utilization, and to avoid network overloads.
+* **Thread Pools and Async I/O**: Avoids bottlenecks associated with single-thread processing.
+* **Parallel Reads**: When uploading a large file from disk, reads from multiple parts of the file in parallel. This is faster than reading the file sequentially from beginning to end.
 
 ### Documentation
 
-- [GetObject](docs/GetObject.md): A visual representation of the GetObject request flow.
-- [Memory Aware Requests Execution](docs/memory_aware_request_execution.md): An in-depth guide on optimizing memory usage during request executions.
+* [GetObject](docs/GetObject.md): A visual representation of the GetObject request flow.
+* [Memory Aware Requests Execution](docs/memory_aware_request_execution.md): An in-depth guide on optimizing memory usage during request executions.
+
+### Configuration
+
+#### Memory Limit
+
+The S3 client uses a buffer pool to manage memory for concurrent transfers. You can control the memory limit in two ways:
+
+1. **Via Configuration** (Recommended): Set `memory_limit_in_bytes` in `aws_s3_client_config`:
+
+```c
+   struct aws_s3_client_config config = {
+       .memory_limit_in_bytes = GB_TO_BYTES(4), // 4 GiB limit
+       // ... other configuration
+   };
+   ```
+
+2. **Via Environment Variable**: Set the `AWS_CRT_S3_MEMORY_LIMIT_IN_GIB` environment variable:
+
+```bash
+   export AWS_CRT_S3_MEMORY_LIMIT_IN_GIB=4  # 4 GiB limit
+   ```
+
+**Priority**: The configuration value takes precedence over the environment variable. If `memory_limit_in_bytes` is set to a non-zero value in the config, the environment variable is ignored.
+
+**Default Behavior**: If neither is set (config is 0 and environment variable is not set), the client sets a default memory limit based on the target throughput.
+
+**Notes**:
+* The limit applies per client. If multiple clients created, limit will apply to each separately.
+* The environment variable value must be a valid positive integer representing gigabytes (GiB).
+* The value is converted from GiB to bytes internally (1 GiB = 1024³ bytes).
+* Invalid values or overflow conditions will cause client creation to fail with `AWS_ERROR_INVALID_ARGUMENT`.
 
 ## License
 
@@ -86,14 +118,19 @@ cmake --build aws-c-s3/build --target install
 After installing all the dependencies, and building aws-c-s3, you can run the sample directly from the s3 build directory.
 
 To download:
+
 ```
 aws-c-s3/build/samples/s3/s3 cp s3://<bucket-name>/<object-name> <download-path> --region <region>
 ```
+
 To upload:
+
 ```
 aws-c-s3/build/samples/s3/s3 cp <upload-path> s3://<bucket-name>/<object-name> --region <region>
 ```
+
 To list objects:
+
 ```
 aws-c-s3/build/samples/s3/s3 ls s3://<bucket-name> --region <region>
 ```

include/aws/s3/private/s3_auto_ranged_get.h

@@ -21,6 +21,18 @@ struct aws_s3_auto_ranged_get {
 
     struct aws_string *etag;
 
+    /* Estimated object stored part size based on ETag analysis */
+    uint64_t estimated_object_stored_part_size;
+    /* Number of parts stored in S3. We derive this from ETag, if ETag is not formatted as expected, this will be
+     * default to 1.
+     * Note: For S3Express Append, the object will be treated as a single part, even though, it can be multiple parts
+     * stored in S3.
+     */
+    uint64_t num_stored_parts;
+    /* Part size was set or not from user for this meta request. */
+    bool part_size_set;
+    bool force_dynamic_part_size;
+
     bool initial_message_has_start_range;
     bool initial_message_has_end_range;
     uint64_t initial_range_start;
@@ -74,6 +86,7 @@ AWS_S3_API struct aws_s3_meta_request *aws_s3_meta_request_auto_ranged_get_new(
     struct aws_allocator *allocator,
     struct aws_s3_client *client,
     size_t part_size,
+    bool part_size_set,
     const struct aws_s3_meta_request_options *options);
 
 AWS_EXTERN_C_END

include/aws/s3/private/s3_checksums.h

@@ -62,7 +62,29 @@ struct aws_s3_meta_request_checksum_config_storage {
 };
 
 /**
- * a stream that takes in a stream
+ * Helper stream that takes in a stream and the checksum context to help finalize the checksum from the underlying
+ * stream.
+ * The context will be only finalized when the checksum stream has read to the end of stream.
+ *
+ * Note: seek this stream will immediately fail, as it would prevent an accurate calculation of the
+ * checksum.
+ *
+ * @param allocator
+ * @param existing_stream The real content to read from. Destroying the checksum stream destroys the existing stream.
+ *                        outputs the checksum of existing stream to checksum_output upon destruction. Will be kept
+ *                        alive by the checksum stream
+ * @param context         Checksum context to keep and get checksum requirements from.
+ */
+AWS_S3_API
+struct aws_input_stream *aws_checksum_stream_new_with_context(
+    struct aws_allocator *allocator,
+    struct aws_input_stream *existing_stream,
+    struct aws_s3_upload_request_checksum_context *context);
+
+/**
+ * Helper stream that takes in a stream to keep track of the checksum of the underlying stream during read.
+ * Invoke `aws_checksum_stream_finalize_checksum` to get the checksum of the data has been read so far.
+ *
  * Note: seek this stream will immediately fail, as it would prevent an accurate calculation of the
  * checksum.
  *
@@ -85,15 +107,6 @@ struct aws_input_stream *aws_checksum_stream_new(
 AWS_S3_API
 int aws_checksum_stream_finalize_checksum(struct aws_input_stream *checksum_stream, struct aws_byte_buf *checksum_buf);
 
-/**
- * Finalize the checksum has read so far to the checksum context.
- * Not thread safe.
- */
-AWS_S3_API
-int aws_checksum_stream_finalize_checksum_context(
-    struct aws_input_stream *checksum_stream,
-    struct aws_s3_upload_request_checksum_context *checksum_context);
-
 /**
  * TODO: properly support chunked encoding.
  * Creates a chunked encoding stream that wraps an existing stream and adds checksum trailers.

include/aws/s3/private/s3_client_impl.h

@@ -168,14 +168,20 @@ struct aws_s3_client_vtable {
 
     void (*finish_destroy)(struct aws_s3_client *client);
 
-    struct aws_parallel_input_stream *(
-        *parallel_input_stream_new_from_file)(struct aws_allocator *allocator, struct aws_byte_cursor file_name);
+    struct aws_parallel_input_stream *(*parallel_input_stream_new_from_file)(
+        struct aws_allocator *allocator,
+        struct aws_byte_cursor file_name,
+        struct aws_event_loop_group *reading_elg,
+        bool direct_io_read);
 
     struct aws_http_stream *(*http_connection_make_request)(
         struct aws_http_connection *client_connection,
         const struct aws_http_make_request_options *options);
 
-    void (*after_prepare_upload_part_finish)(struct aws_s3_request *request, struct aws_http_message *message);
+#ifdef AWS_C_S3_ENABLE_TEST_STUBS
+    /********************* TEST ONLY STUB **************************/
+    void (*after_prepare_upload_part_finish_stub)(struct aws_s3_request *request, struct aws_http_message *message);
+#endif
 };
 
 struct aws_s3_upload_part_timeout_stats {
@@ -231,10 +237,21 @@ struct aws_s3_client {
      * to meta requests for use. */
     const size_t part_size;
 
+    bool part_size_set;
+
     /* Size of parts for files when doing gets or puts.  This exists on the client as configurable option that is passed
      * to meta requests for use. */
     const uint64_t max_part_size;
 
+    /* Calculated optimal range size for GET operations based on client configuration (memory limits, throughput
+     * targets). This is used when part_size is not explicitly configured, replacing the default with reasonable
+     * calculation. Value is calculated during client initialization and remains constant for the client's lifetime. */
+    const uint64_t optimal_range_size;
+
+    /* File I/O options. */
+    bool fio_options_set;
+    struct aws_s3_file_io_options fio_opts;
+
     /* The size threshold in bytes for when to use multipart uploads for a AWS_S3_META_REQUEST_TYPE_PUT_OBJECT meta
      * request. Uploads over this size will automatically use a multipart upload strategy, while uploads smaller or
      * equal to this threshold will use a single request to upload the whole object. If not set, `part_size` will be
@@ -351,6 +368,9 @@ struct aws_s3_client {
 
         /* Number of requests currently scheduled to be streamed the response body or are actively being streamed. */
         struct aws_atomic_var num_requests_streaming_response;
+
+        /* Number of overall requests currently streaming the request body instead of buffering. */
+        struct aws_atomic_var num_requests_streaming_request_body;
     } stats;
 
     struct {

include/aws/s3/private/s3_default_buffer_pool.h

@@ -6,6 +6,8 @@
  * SPDX-License-Identifier: Apache-2.0.
  */
 
+#include <aws/common/hash_table.h>
+#include <aws/common/mutex.h>
 #include <aws/s3/s3.h>
 #include <aws/s3/s3_buffer_pool.h>
 
@@ -59,11 +61,79 @@ struct aws_s3_default_buffer_pool_usage_stats {
     /* Secondary memory reserved, but not yet used. Accurate, maps directly to base allocator. */
     size_t secondary_reserved;
 
+    /* Overall memory allocated for special-sized blocks. */
+    size_t special_blocks_allocated;
+    /* Number of special block sizes created. */
+    size_t special_blocks_num;
+    /* Memory reserved in special-sized blocks. */
+    size_t special_blocks_reserved;
+    /* Memory used in special-sized blocks. */
+    size_t special_blocks_used;
+
     /* Bytes used in "forced" buffers (created even if they exceed memory limits).
      * This is always <= primary_used + secondary_used */
     size_t forced_used;
 };
 
+/* Structure to track special-sized blocks */
+struct s3_special_block_list {
+    struct aws_allocator *allocator;
+    size_t buffer_size;           /* Size of buffers in this list */
+    struct aws_array_list blocks; /* Array of uint8_t* pointers to allocated blocks */
+};
+
+struct aws_s3_default_buffer_pool {
+    struct aws_allocator *base_allocator;
+    struct aws_mutex mutex;
+
+    size_t block_size;
+    size_t chunk_size;
+    /* size at which allocations should go to secondary */
+    size_t primary_size_cutoff;
+
+    /* NOTE: See aws_s3_buffer_pool_usage_stats for descriptions of most fields */
+
+    size_t mem_limit;
+
+    size_t primary_allocated;
+    size_t primary_reserved;
+    size_t primary_used;
+
+    size_t special_blocks_allocated;
+    size_t special_blocks_reserved;
+    size_t special_blocks_used;
+
+    size_t secondary_reserved;
+    size_t secondary_used;
+
+    size_t forced_used;
+
+    struct aws_array_list blocks;
+
+    struct aws_linked_list pending_reserves;
+
+    /* Special-sized blocks: hash table mapping size -> struct s3_special_block_list * */
+    /* TODO: let's discuss about the special list lifetime. Should we just keep it with the memory pool? Concern is that
+     * the pool will live with the client, and may result in all sorts of special lists to be around. */
+    struct aws_hash_table special_blocks;
+
+    /* TEST ONLY: to force the special blocks alive during trim. */
+    bool force_keeping_special_blocks;
+};
+
+struct s3_pending_reserve {
+    struct aws_linked_list_node node;
+    struct aws_future_s3_buffer_ticket *ticket_future;
+    struct aws_s3_default_buffer_ticket *ticket;
+    struct aws_s3_buffer_pool_reserve_meta meta;
+};
+
+struct s3_buffer_pool_block {
+    size_t block_size;
+    uint8_t *block_ptr;
+    uint16_t alloc_bit_mask;
+};
+
 /*
  * Create new buffer pool.
  * chunk_size - specifies the size of memory that will most commonly be acquired