Using S3 creds in storage_options results in side effect of setting env vars

[alexr-bq]

Environment

Delta-rs version: 0.14.0

Binding: Rust

Environment: macOS Ventura 13.4

---

Bug

What happened:

When passing in S3 creds via storage options to create a delta table, ie AWS_ACCESS_KEY_ID, this has an unintended side-effect of overriding set environment variables.

What you expected to happen:

S3 values to be used from the storage options map without setting any environment variables.

How to reproduce it:

let storage_options_invalid: HashMap<String, String> = HashMap::from([
(
"AWS_DEFAULT_REGION".to_string(),
env::var("AWS_DEFAULT_REGION").unwrap().to_string(),
),
(
"AWS_SECRET_ACCESS_KEY".to_string(),
env::var("AWS_SECRET_ACCESS_KEY").unwrap().to_string(),
),
("AWS_ACCESS_KEY_ID".to_string(), "INVALID123".to_string()),
(
"AWS_SESSION_TOKEN".to_string(),
env::var("AWS_SESSION_TOKEN")
.unwrap_or("".to_string())
.to_string(),
),
]);

println!("{}",env::var("AWS_ACCESS_KEY_ID").unwrap().to_string()); // Prints correct value

let mut table = deltalake::open_table_with_storage_options(
table_filepath.as_str(),
storage_options_invalid,
)

println!("{}",env::var("AWS_ACCESS_KEY_ID").unwrap().to_string()); // Prints "INVALID123"

[r3stl355]

It happens here and looks like there is a good reason according to comments

delta-rs/rust/src/storage/s3.rs

Line 259 in bdf1c4e

Self::ensure_env_var(options, s3_storage_options::AWS_ACCESS_KEY_ID);

. It should only effect the current execution context - not the global environment (e.g. on running the again the correct values will be picked up from global env again).

It also looks like there is a bit of a delay before the request actually fails (some kind of caching?). I mean running a test with right value of AWS_ACCESS_KEY_ID set in the storage option followed by several runs with invalid value still works.

Actually, other values (AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN) show the same behavoiur

[rtyler]

I am going to close this because I believe we are working much more predictably now by relying on the AWS SDK for credentials which will provide the appropriate passthrough