Skip to content

Unified write data API#552

Merged
azkrishpy merged 54 commits into
mainfrom
unified-write-data-api
Mar 30, 2026
Merged

Unified write data API#552
azkrishpy merged 54 commits into
mainfrom
unified-write-data-api

Conversation

@azkrishpy
Copy link
Copy Markdown
Contributor

@azkrishpy azkrishpy commented Feb 25, 2026

Issue #, if available:

Description of changes:
A new protocol-agnostic aws_http_stream_write_data() API that lets users write request body data the same way regardless of whether the connection is HTTP/1.1 or HTTP/2.

High level details of changes:
Public API:

  • New aws_http_stream_write_data() function that works for both H1 and H2
  • New use_manual_data_writes flag in request options (unified replacement for the H2-only http2_use_manual_data_writes)
  • Both old APIs (aws_http1_stream_write_chunk, aws_http2_stream_write_data) still work for backward compatibility

H1 Content-Length path:

  • User sets Content-Length header + use_manual_data_writes = true, no body stream
  • Each write_data call creates an aws_h1_data_write that gets queued and processed by the encoder
  • New encoder states (DATA_WRITE_NEXT, DATA_WRITE_BODY) handle reading from queued writes and tracking progress against Content-Length
  • Encoder validates total bytes match Content-Length when end_stream = true

H1 Chunked path:

  • User sets Transfer-Encoding: chunked header + use_manual_data_writes = true
  • write_data internally converts to write_chunk calls (same underlying mechanism as before)
  • User sends a zero-length write_data with end_stream = true to terminate (consistent with write_chunk behavior)

H2 path:

  • use_manual_data_writes now works for H2 (alongside the existing http2_use_manual_data_writes)
  • Internally routes through the same vtable->write_data as before — H2 implementation unchanged

Vtable unification:

  • Single write_data vtable entry replaces the old http2_write_data
  • H1 stream implements it (new), H2 stream implements it (renamed from existing)
  • aws_http2_stream_write_data() casts options and calls the same vtable entry

Elasticurl example:

  • Added --manual-write interactive mode demonstrating the API

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Comment thread include/aws/http/request_response.h
Comment thread include/aws/http/request_response.h Outdated
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Feb 27, 2026

Codecov Report

❌ Patch coverage is 87.13450% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.79%. Comparing base (8bf9e53) to head (6552bb4).

Files with missing lines Patch % Lines
source/h1_stream.c 86.74% 11 Missing ⚠️
source/h1_encoder.c 88.57% 8 Missing ⚠️
source/h1_connection.c 57.14% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #552      +/-   ##
==========================================
+ Coverage   79.70%   79.79%   +0.08%     
==========================================
  Files          28       28              
  Lines       11936    12079     +143     
==========================================
+ Hits         9514     9638     +124     
- Misses       2422     2441      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread source/h1_encoder.c Outdated
Comment thread source/h1_encoder.c Outdated
Comment thread source/request_response.c
Comment thread examples/curl_content_length_example.c Outdated
Comment thread include/aws/http/request_response.h Outdated
Comment thread source/h1_encoder.c Outdated
Comment thread source/h1_encoder.c
Comment thread source/h1_stream.c Outdated
Comment thread source/h1_stream.c
Comment thread include/aws/http/private/h1_stream.h Outdated
Comment thread source/h1_encoder.c
Comment thread source/h2_stream.c
@azkrishpy azkrishpy force-pushed the unified-write-data-api branch from c7b318c to af32dd9 Compare March 2, 2026 22:35
Comment thread tests/test_h1_client.c
Comment thread tests/test_h1_encoder.c
Comment thread tests/test_h2_client.c
Comment thread include/aws/http/request_response.h
Comment thread source/h1_stream.c
Comment on lines +461 to +462
struct aws_http1_chunk_options termination_opts;
AWS_ZERO_STRUCT(termination_opts);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should invoke the callback.
In case of options->data != NULL && options->end_stream
I think more reasonable way is skip the on_complete for the first chunk with data, and then provide the callback to the termination chunk.

Comment thread tests/test_h1_client.c
Comment on lines +5539 to +5542
struct aws_http_stream_write_data_options write_options = {
.data = NULL,
.end_stream = true,
};
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a callback and make sure the callback invoked correctly

Comment thread tests/test_h1_client.c Outdated
Comment thread tests/test_h1_client.c
return AWS_OP_SUCCESS;
}

/* Test: write_data with NULL data and end_stream=true on chunked stream */
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about a case with data and end_stream = true for chunked encoding?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is that not the write_chunk case (the happy path that used to be tested already)?

Comment thread source/h1_stream.c
stream->synced_data.is_cross_thread_work_task_scheduled = true;
}

stream->synced_data.has_final_data_write = options->end_stream;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think we should check when end_stream set, the provided data is mathcing the content-length or not, when options->data is NULL.

Since you skip create data_write when there is no data, the corner case will not be checked. Also, there is no place to invoke the complete callback.

So, maybe another concern case worth to add a test for

Comment thread source/h1_stream.c Outdated
Comment thread source/h1_stream.c Outdated
@azkrishpy azkrishpy force-pushed the unified-write-data-api branch from 16def17 to 944d1e5 Compare March 27, 2026 08:37
Comment thread source/h1_stream.c
Comment thread include/aws/http/private/h1_stream.h Outdated
Comment on lines +127 to +128
/* Whether the stream is using manual data writes instead of input_stream */
bool using_manual_data_writes : 1;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's move this out

Comment thread include/aws/http/request_response.h
Comment thread source/h1_encoder.c
}

if (encoder_message->content_length > 0 && !has_body_stream) {
if (use_manual_data_writes && has_body_stream) {
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually added the check here.. forgot that I had done it.

Comment thread tests/test_h1_client.c
Comment thread source/h1_stream.c Outdated
@azkrishpy azkrishpy merged commit 0de22ec into main Mar 30, 2026
45 checks passed
@azkrishpy azkrishpy deleted the unified-write-data-api branch March 30, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants