S3 client can stall on PutObject after long idle gap (pooled connection reuse / configuration)

[quinnjr]

Summary

S3Client::put_object().send().await can appear to hang when the same client is used for get_object, then a long period of local work (no HTTP I/O), then put_object to the same endpoint. The future may not complete until an operation-level timeout fires (e.g. TimeoutConfig::operation_timeout), with errors like request has timed out.

This is most plausibly explained by HTTP connection pooling: the client reuses an idle pooled connection that the peer or path has already closed (or half-closed), while the pool’s idle retention (Hyper’s default is 90s when pool_timer is configured) may still treat the connection as reusable. That is a client lifecycle / default tuning issue relative to server-side idle behavior, not necessarily incorrect PutObject logic.

get_object can succeed; the problematic phase is reuse after a gap.

Likely classification

• Application configuration: Workloads with large gaps between sequential calls on one S3Client should consider shorter pool_idle_timeout on a custom HTTP connector, operation timeouts, a fresh client (or connector) for the upload phase, or not sharing one client across long CPU-bound sections.
• Documentation gap: The AWS Rust SDK could call out this pattern explicitly (long idle between requests → tune pool idle or avoid reuse).
• Optional product change: Whether default pool idle timeout should be more conservative than Hyper’s 90s is a tradeoff (fewer dead reuse attempts vs more connection churn) for maintainers to decide—not a given “bug fix.”

Environment

• Runtime: Linux x86_64 (Docker: CUDA base image + Rust binary from rust:1-bookworm)
• Region: us-west-2
• Credentials: IAM task role (AWS Batch on EC2)
• Dependencies observed: aws-sdk-s3 1.126.x / 1.127.x when unconstrained; hang also seen with aws-sdk-s3 pinned <1.127 (1.126.x). Client from aws_config::defaults(BehaviorVersion::latest()).load() + aws_sdk_s3::Client::new(&cfg).

An older production image built around 2026-03-11 (older resolved SDK versions) did not show this for the same application pattern.

Reproduction sketch

1. Single aws_sdk_s3::Client from default config.
2. get_object, consume body (success).
3. ~30+ seconds (or longer) of CPU-heavy work without using the S3 client.
4. put_object with ByteStream::from_path(...) and .send().await.

Observed: .await blocks for a long time; with TimeoutConfig::operation_timeout (e.g. 300s), request has timed out.

Expected (from an app perspective): Either successful upload, or a fast, clear connection error and retry on a new connection—without requiring every user to discover pool behavior by production incident.

Additional observations

• aws s3 cp to the same bucket from the same task succeeds quickly, consistent with fresh connection / different stack, not IAM or S3 outage.
• Pinning aws-sdk-s3 < 1.127 alone did not resolve the stall in our tests—consistent with the issue being below the S3 crate (HTTP pool / hyper), not a single crate version regression.
• Mitigations that worked for us: TimeoutConfig on the config loader; performing uploads via aws s3 cp subprocess; optionally new client for upload phase.

Suggested follow-ups for maintainers

1. Docs: Add guidance for long idle gaps between calls on one client (tune pool_idle_timeout via custom HTTP client, or separate clients / phases).
2. Examples: Optional example of aws_config + custom aws_smithy_http_client::Connector with explicit pool_idle_timeout.
3. Defaults: Evaluate whether Smithy’s default when pool_idle_timeout is unset should differ from Hyper’s 90s (tradeoff vs dead reuse)—not assumed without measurement.

References

• Related prototype note (default pool idle): discussion in this thread.
• Internal context: OCR pipeline — download PDF → GPU/CPU processing → upload SQLite + PDF.

Thank you for considering documentation and defaults; we’re happy to help validate doc changes or a minimal repro crate if useful.

[quinnjr]

We prototyped a mitigation in smithy-rs aws-smithy-http-client: when pool_idle_timeout is left unset on the connector, explicitly set 45s (instead of inheriting hyper_util’s 90s default) so idle pooled connections are discarded sooner after gaps between requests. This may reduce reuse of connections the server has already half-closed.

Local branch (not yet pushed upstream): fix/default-pool-idle-timeout-1417 — happy to open a PR against smithy-lang/smithy-rs if maintainers want this direction; the behavioral change (90s → 45s default) may need product/perf review.

[quinnjr]

Update: mitigation attempts and final workaround

We have now exhausted several SDK-level approaches to resolve this in our application and have fallen back to shelling out to the AWS CLI for S3 uploads. Sharing the full list of attempts in case it helps inform the priority/scope of any upstream fix.

Attempts (all failed)

1. Pin aws-sdk-s3 to <1.127
Hypothesis: regression in aws-sdk-s3 v1.127.0. Pinned aws-sdk-s3 = ">=1, <1.127" and aws-config = ">=1, <1.8.10".
Result: uploads still hung after the long GPU-bound gap. Stall reproduced identically with v1.126.x.

2. Custom HTTP client with short pool_idle_timeout (rustls-aws-lc)
Used aws-smithy-http-client with CryptoMode::AwsLc and a 20-second pool idle timeout for a dedicated upload-only S3Client.
Result: PutObject still timed out after 300s (our operation_timeout). The connection pool configuration did not prevent the stall.

3. Custom HTTP client with rustls-ring TLS provider
Switched the custom HTTP client from rustls-aws-lc to rustls-ring in case the TLS provider was involved.
Result: identical 300s timeout on PutObject.

4. Separate default SDK configs (no custom HTTP client)
Removed aws-smithy-http-client entirely. Created two independent aws_config::SdkConfig instances (one for downloads, one for uploads) via separate aws_config::defaults(...).load().await calls, each with their own default HTTP stack.
Result: still timed out at 300s. Even a completely fresh default config, never used for downloads, failed when PutObject was called after the long idle gap.

5. Prototype pool_idle_timeout default change in smithy-rs (our earlier comment)
Local branch lowering the default from 90s → 45s. We did not get to validate this in production because attempts 2–4 showed that even explicit short timeouts on the connector did not resolve the issue, suggesting the problem may not be purely about idle connection reuse timing.

What works

Shelling out to aws s3 cp via tokio::process::Command.

let output = tokio::process::Command::new("aws")
    .args(["s3", "cp", &local_path, &s3_dest, "--content-type", content_type, "--sse", "AES256"])
    .output()
    .await?;

This spawns a fresh process with its own connection stack and completes in ~1 second for the same files that hang indefinitely with the SDK client. We've now deployed this in production and re-processed all affected batches successfully.

Observations

• The stall occurs even when the S3Client used for uploads was never used for any prior request (attempt 4). This rules out connection reuse from a prior get_object as the sole cause.
• The stall occurs regardless of TLS provider (aws-lc vs ring).
• The stall occurs regardless of whether pool_idle_timeout is explicitly configured.
• The stall only manifests after a long period (30s+) of CPU/GPU-bound work with no SDK I/O, in a containerized environment (AWS Batch on EC2 with ECS agent).
• aws s3 cp from the same container at the same time succeeds instantly.
• An older production image from ~2026-03-11 (older resolved SDK versions) did not exhibit this behavior for the same workload.

This pattern suggests the issue may be deeper than connection pool tuning — possibly related to the async runtime state, Hyper's internal connection management after long periods without I/O, or an interaction with the container networking environment. Happy to provide additional diagnostics or a minimal repro if that would help.

[jct-tympanon]

I observed a suspiciously similar issue after recently upgrading to s3 sdk v1.119.0, and was also unable to fix it by pinning to older versions. i'm using Rust 1.93.1 running in AWS Lambda, tokio async runtime with io and time drivers. I execute the lambda from AWS CLI. TL;DR i did find a couple of workarounds that suggest where the problem may lie, see towards the bottom of my comment. Here's my problem code, nothing wild:

let data = ByteStream::from_path(source_file).await?;
self.s3.put_object()
    .bucket(&self.ops_bucket)
    .key(&key)
    .body(data)
    .send().await?;

"source_file" is a &Path pointed at a NamedTemporaryFile that is about 2Mb in lambda temp dir. This worked fine in older versions of the SDK. In the v1.119.0 SDK, it just hangs until there's a timeout. other S3 interactions (get, head, list) all work fine. Notably, I think those don't have a request body. Or the body is quite small.

DEBUG logs from the SDK indicate the client closed the connection due to a lack of activity, which was confusing. example (heavily redacted):

Caused by: [aws_smithy_runtime_api::client::result::SdkError<aws_sdk_s3::operation::put_object::PutObjectError, aws_smithy_runtime_api::http::response::Response>]
--
ServiceError(ServiceError { 
  source: Unhandled(Unhandled { source: ErrorMetadata { 
    code: Some("RequestTimeout"), 
    message: Some("Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed."), [...], 
    meta: ErrorMetadata { code: Some("RequestTimeout"), 
   ... 
   raw: Response { status: StatusCode(400), ...

Workaround 1: send body from memory.

I found that reading the file into memory and providing that as the SdkBody, rather than streaming from disk, resolved the issue. but this isn't an acceptable long-term solution for me. so,

Workaround 2: disable checksums.

After a lot of code diving and trial-and-error, this resolved the issue for me:

// possible issue: the trailing checksum header calculation creates a disagreement between the content length header
// and what is ultimately transmitted to the server, or possibly some interaction between the checksum calculation and aws chunked transfer encoding leading to a mismatch?
let s3_client = aws_sdk_s3::Client::from_conf(
    aws_sdk_s3::Config::new(&sdk_config)
        .to_builder()
        .request_checksum_calculation(
            aws_sdk_s3::config::RequestChecksumCalculation::WhenRequired, // not strictly required for put object, so we skip it.
        )
        .build(),
);

the code comment above is obviously speculative. i'm still researching and trying to produce a failing minimal test case in a fork of smithy-rs. i'm not deep in the SDK, but looking at the history while trying to research this issue, it looks like there's been a bit going on with both an http 04->1.x upgrade and AWS chunked encoding logic.

here's my sdk version as reported by cargo tree -d normal | grep aws:

├── aws-sdk-s3 v1.119.0
│   ├── aws-credential-types v1.2.12 (*)
│   ├── aws-runtime v1.7.0 (*)
│   ├── aws-sigv4 v1.4.0 (*)
│   ├── aws-smithy-async v1.2.12 (*)
│   ├── aws-smithy-checksums v0.63.12
│   │   ├── aws-smithy-http v0.62.6
│   │   │   ├── aws-smithy-eventstream v0.60.19 (*)
│   │   │   ├── aws-smithy-runtime-api v1.11.4 (*)
│   │   │   ├── aws-smithy-types v1.4.7 (*)
│   │   ├── aws-smithy-types v1.4.7 (*)
│   ├── aws-smithy-eventstream v0.60.19 (*)
│   ├── aws-smithy-http v0.62.6 (*)
│   ├── aws-smithy-json v0.61.9
│   │   └── aws-smithy-types v1.4.7 (*)
│   ├── aws-smithy-runtime v1.10.1 (*)
│   ├── aws-smithy-runtime-api v1.11.4 (*)
│   ├── aws-smithy-types v1.4.7 (*)
│   ├── aws-smithy-xml v0.60.14
│   ├── aws-types v1.3.12 (*)

[ysaito1001]

Thank you for reporting this issue along with the detailed description and analysis. To help identify where the issue might lie, here are some notable changes for S3 around the relevant timeframe:

• aws-sdk-s3 1.114.0 (release-2025-11-20) — Refactored the checksum interceptor
• aws-sdk-s3 1.122.0 (release-2026-02-03) — Released http 1.x support
• aws-sdk-s3 1.123.0 (release-2026-02-11) — Released aws-chunked changes

If you're able to bisect which version introduced the issue, that would help us narrow down the root cause:

• ≤ 1.113.0: The issue may be deeper in the codebase and predate these changes.
• ≥ 1.114.0 and < 1.122.0: Likely related to the checksum interceptor refactoring.
• = 1.122.0: Likely related to the http 1.x upgrade.
• ≥ 1.123.0: Likely related to the aws-chunked logic.

[jct-tympanon]

Thank you for reporting this issue along with the detailed description and analysis. To help identify where the issue might lie, here are some notable changes for S3 around the relevant timeframe:

Thanks for the timeline, that's a huge help deciphering the history. From my own deployment history, I can at least report that things were working up until at least 1.95, and then started failing with a bump to 1.119 (aws-s3-sdk).

BUT, In my case, I've found today that rolling forward fixes the issue for me, so the urgency is a lot lower now. I'm unsure whether my problem is the same as OP @quinnjr ; based on the original report and my experience with different versions, they would need to be different issues, even though the observed behavior is almost exactly the same. it seems like the checksum workaround would be a good way to verify on @quinnjr 's end.

I was able to remove my workarounds, and the code worked again as before, after upgrading as follows:

├── aws-sdk-s3 v1.127.0
│   ├── aws-credential-types v1.2.14 (*)
│   ├── aws-runtime v1.7.2 (*)
│   ├── aws-sigv4 v1.4.2 (*)
│   ├── aws-smithy-async v1.2.14 (*)
│   ├── aws-smithy-checksums v0.64.6
│   │   ├── aws-smithy-http v0.63.6 (*)
│   │   ├── aws-smithy-types v1.4.7 (*)
│   ├── aws-smithy-eventstream v0.60.20 (*)
│   ├── aws-smithy-http v0.63.6 (*)
│   ├── aws-smithy-json v0.62.5 (*)
│   ├── aws-smithy-observability v0.2.6 (*)
│   ├── aws-smithy-runtime v1.10.3 (*)
│   ├── aws-smithy-runtime-api v1.11.6 (*)
│   ├── aws-smithy-types v1.4.7 (*)
│   ├── aws-smithy-xml v0.60.15
│   ├── aws-types v1.3.14 (*)

bisect is difficult given that i can only reproduce in the lambda production environment. i'll let you know if i get further.

[quinnjr]

Bisection results: ByteStream::from_path + checksum pipeline in aws-smithy-checksums 0.63.x

Following @ysaito1001's suggested version ranges and @jct-tympanon's checksum workaround, I ran a local bisection against our staging S3 bucket (us-west-2). The test performs HeadBucket → GetObject → 45-second idle gap (simulating GPU OCR) → PutObject with both in-memory ByteStream::from(Vec<u8>) and file-based ByteStream::from_path(...).

Results

aws-sdk-s3	aws-smithy-checksums	ByteStream::from(Vec)	ByteStream::from_path	Verdict
1.95.0	0.63.x	1.4s ✅	30s timeout ❌	BROKEN
1.113.0	0.63.x	1.2s ✅	30s timeout ❌	BROKEN
1.119.0 (our lockfile)	0.63.x	1.1s ✅	30s timeout ❌	BROKEN
1.121.0	0.63.x	1.2s ✅	30s timeout ❌	BROKEN
1.122.0	0.64.x	1.7s ✅	322ms ✅	FIXED
1.123.0	0.64.x	1.2s ✅	277ms ✅	FIXED
1.127.0	0.64.x	1.1s ✅	279ms ✅	FIXED

Checksum workaround

On 1.119.0 with RequestChecksumCalculation::WhenRequired (per @jct-tympanon's suggestion): both in-memory and file-based PutObject pass (file upload completes in 325ms). This confirms the bug is in the checksum calculation path, not connection pooling.

Findings

1. The bug is NOT a recent regression. It exists in every version from at least 1.95.0 through 1.121.0. The reason we didn't notice it before is that our Cargo.lock previously resolved to >= 1.122.0 (the caret "1" dependency picked up the fix automatically); a stale lockfile rebuild pinned us back to 1.119.0.

2. The fix boundary is aws-sdk-s3 1.121.0 → 1.122.0, which corresponds to the http 1.x merge (smithy-rs #4484, released 2026-02-03). The key transitive dependency change is aws-smithy-checksums 0.63.x → 0.64.x.

3. Only ByteStream::from_path (streaming from disk) is affected. In-memory ByteStream::from(bytes) works on every version tested. This points to the checksum interceptor's handling of streaming bodies with unknown-at-start content in the old aws-chunked transfer encoding implementation.

4. Connection pooling / idle timeout is NOT the root cause for our case. Even a freshly-constructed client with no prior requests fails on from_path uploads. Our earlier mitigation attempts (custom HTTP client, pool_idle_timeout, separate SDK configs, TLS provider swap) were chasing the wrong hypothesis.

Why our previous mitigations failed

What we tried	Why it didn't help
Pin aws-sdk-s3 < 1.127	Bug is in all versions below 1.122.0
Separate upload client with short pool_idle_timeout	Not a connection pool issue
Custom HTTP client with ring TLS	Not TLS-related
Fresh default SDK config per operation	Fresh client still fails — bug is in the body/checksum pipeline
aws s3 cp via CLI	Works — fresh process, completely bypasses SDK checksum path

Resolution

Updating to aws-sdk-s3 >= 1.122.0 (we're going to 1.127.0) resolves the issue at the source. We can then revert our aws s3 cp workaround back to native SDK PutObject calls.

Thanks @ysaito1001 for the timeline and @jct-tympanon for the checksum workaround hint — that's what cracked it open. Happy to provide additional data if it helps with documentation or regression tests.

[vcjana]

Thanks for the thorough bisection. This points to the checksum pipeline and how it handles the streaming bodies. Glad that upgrading to 1.127.0 resolves the issue in your end. That said, we would love to understand the exact failure a bit more.

If possible can you share a minimal reproducer? We want to make sure we fully understand what was happening in the older path, even though the current versions have the fix.

[quinnjr]

Minimal reproducer

As requested by @vcjana, here's a standalone repro crate:

https://github.com/Lexmata/aws-sdk-s3-putobject-repro

Quick start

git clone https://github.com/Lexmata/aws-sdk-s3-putobject-repro.git
cd aws-sdk-s3-putobject-repro

export REPRO_BUCKET=your-test-bucket   # any bucket you have read/write access to
export AWS_REGION=us-west-2            # your bucket's region
cargo run

The binary is pinned to aws-sdk-s3 = "=1.119.0" (aws-smithy-checksums 0.63.x, broken) by default. It runs three tests sequentially:

Test	Body source	Checksums	Expected on <= 1.121.0
1	ByteStream::from(Vec<u8>)	default	✅ OK (control)
2	ByteStream::from_path(...)	default	❌ 30s timeout
3	ByteStream::from_path(...)	WhenRequired	✅ OK (workaround)

Each test simulates a 45-second idle gap (configurable via REPRO_IDLE_SECS) before the PutObject call to match the real-world pattern (download → long CPU/GPU work → upload), though the idle gap is not strictly required to trigger the bug.

To verify the fix

Edit Cargo.toml and change the pin to a fixed version:

aws-sdk-s3 = "=1.122.0"   # or any >= 1.122.0

Then rm Cargo.lock && cargo run — all three tests should pass.

Automated bisection

The repo also includes a bisect.sh script that iterates over versions 1.95.0 → 1.127.0, rebuilds for each, and collects a results table:

./bisect.sh

This confirms the boundary at 1.121.0 → 1.122.0 (aws-smithy-checksums 0.63.x → 0.64.x).

[vcjana]

Thanks for the reproducer we reproduced it on our end and confirmed the behavior.

The root cause is in ChecksumBody's poll_trailers() in aws-smithy-checksums 0.63.x. After all body data is streamed for a ByteStream::from_path upload, poll_trailers() never resolves, causing the request to hang until the operation timeout fires. In-memory bodies ByteStream::from(Vec<u8>) are unaffected because they take the header checksum path, bypassing the streaming trailer pipeline entirely.

This was fixed in aws-smithy-checksums 0.64.x as part of the http 1.x migration (smithy-rs #4484), where poll_trailers() was replaced with a unified poll_frame() approach that handles checksum trailer emission inline when the body is exhausted.

Upgrading to aws-sdk-s3 >= 1.122.0 resolves the issue. For users on older versions, setting
RequestChecksumCalculation::WhenRequired is a workaround.

Thanks again for the thorough investigation the bisection table and reproducer made this easier to trace.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.