`CreateBuilder` never returns.

[ryzhyk]

Environment

Delta-rs version:

0.26.2

Binding:

Rust

Environment:

• Cloud provider: AWS
• OS: Ubuntu
• Other:

---

Bug

What happened:

This usually happens when my application is trying to open multiple Delta Tables in parallel. This seems to cause S3 timeouts, which in turn lead to the following behavior:

CreateBuilder::new()
            .with_location(uri)
            .with_save_mode(DeltaTableWriteMode::Append)
            .with_storage_options(storage_options)
            .with_columns(columns)
            .await;

This call never returns (we've seen it stuck for many hours), but I see the following message in the log:

[2m2025-09-17T23:09:44.890623Z�[0m �[33m WARN�[0m �[2maws_config::web_identity_token�[0m�[2m:�[0m STS returned an error assuming web identity role �[3merror�[0m�[2m=�[0mdispatch failure: timeout: minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed (DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: ThroughputBelowMinimum { expected: Throughput { bytes_read: 1, per_time_elapsed: 1s }, actual: Throughput { bytes_read: 0, per_time_elapsed: 1s } }, connection: Unknown } }))
[2m2025-09-17T23:09:44.890681Z�[0m �[33m WARN�[0m �[2maws_config::meta::credentials::chain�[0m�[2m:�[0m provider failed to provide credentials �[3mprovider�[0m�[2m=�[0mWebIdentityToken �[3merror�[0m�[2m=�[0man error occurred while loading credentials: dispatch failure: timeout: minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed (ProviderError(ProviderError { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: ThroughputBelowMinimum { expected: Throughput { bytes_read: 1, per_time_elapsed: 1s }, actual: Throughput { bytes_read: 0, per_time_elapsed: 1s } }, connection: Unknown } }) }))
[2m2025-09-17T23:09:44.890696Z�[0m �[33m WARN�[0m �[2maws_config::meta::credentials::chain�[0m�[2m:�[0m provider failed to provide credentials �[3mprovider�[0m�[2m=�[0mDefaultChain �[3merror�[0m�[2m=�[0man error occurred while loading credentials: dispatch failure: timeout: minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed (ProviderError(ProviderError { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: ThroughputBelowMinimum { expected: Throughput { bytes_read: 1, per_time_elapsed: 1s }, actual: Throughput { bytes_read: 0, per_time_elapsed: 1s } }, connection: Unknown } }) }))

What you expected to happen:

The operations either succeeds or fails.

How to reproduce it:

Run many instances of the above code in parallel. I'll see if I can produce a small repro.

More details:

[rtyler]

How many parallel S3 tables are we talking about?

[ryzhyk]

In the order of 10.

[ryzhyk]

Unfortunately I don't have an environment where I can reproduce this, so I am trying to deduce what's happening from the code. I think that the warning is printed here before returning an error from load_credentials:

https://github.com/smithy-lang/smithy-rs/blob/1c192f836cbf547acb2f4e7d21438635e631d429/aws/rust-runtime/aws-config/src/web_identity_token.rs#L280

This error should be handled here:

delta-rs/crates/aws/src/credentials.rs

Line 87 in 19f19db

.provide_credentials()

This part is a bit suspicious: why is it wrapping the error returned by the credentials provider in ObjectStoreError::NotSupported? This doesn't look like the right type of error to me, although I'm not sure how this can lead to CreateBuilder getting stuck.

[rtyler]

@ryzhyk Lo0oking at the logs that you provided, I don't believe that the code snippet you highlighted around provide_credentials() represents a problem. The mapping of the error is just to convert the error type for the object_store CredentialProvider trait implementation.

The warnings in the logs are really quite interesting. It doesn't look like the underlying STS SDK is able to reach either the STS service or the IMDSv2 daemon (or something like that).

Based on what you've described this smells like a rate limiting or API exhaustion on STS but the logs the AWS SDKs are giving us here are really not sufficient to point to that.

0.26.2 is a bit older now, and if you're able to try 0.29.1, that has some newer AWS SDK dependencies bundled with it

[ryzhyk]

@ryzhyk Lo0oking at the logs that you provided, I don't believe that the code snippet you highlighted around provide_credentials() represents a problem. The mapping of the error is just to convert the error type for the object_store CredentialProvider trait implementation.

Sure, I also don't think it's what causes the issue, but I was still wondering if NotSupported is the right error to return her.

The warnings in the logs are really quite interesting. It doesn't look like the underlying STS SDK is able to reach either the STS service or the IMDSv2 daemon (or something like that).

Based on what you've described this smells like a rate limiting or API exhaustion on STS but the logs the AWS SDKs are giving us here are really not sufficient to point to that.

And it doesn't help that the error happens rarely, non-deterministically, and on a system I don't have direct access to :( Is there some additional logging we can add to help diagnose this in the future?

0.26.2 is a bit older now, and if you're able to try 0.29.1, that has some newer AWS SDK dependencies bundled with it

We've been sticking to 0.26.2, since I saw you mention performance and stability issues in newer releases. Do you reckon those are resolved in 0.29.1?

In the meantime, I added a timeout and retry around the entire CreateBuilder invocation.

[rtyler]

Yes performance issues were resolved, which is why 0.29.x was actually released 😄