Skip to content

Commit 166be2a

Browse files
storage: Refresh catalog-vended storage credentials
Vended credentials expire, and nothing refreshed them: `loadTable` hands their access keys to the FileIO once and OpenDAL keeps signing with them until the dataflow restarts. Adds `VendedCredentialLoader`, a `ProvideCredential` implementation that re-fetches from the catalog's `loadCredentials` endpoint. OpenDAL rebuilds its `Operator` for every file operation, so reqsign's own credential cache never survives one call; the loader therefore caches with its own expiry deadline, taken from `s3.session-token-expires-at-ms` where the catalog reports one and a short interval where it does not. On a 401 or 403 it invalidates the catalog token so the next attempt mints a fresh one. Materialize now builds the OAuth2 provider itself rather than passing a `credential` catalog property, so one token object serves both catalog requests and credential refreshes. The catalog client rejects a custom authenticator combined with that property, so the two cannot coexist. `connect` grows a table argument because the credentials endpoint is table-scoped. Installing a loader also means it alone supplies S3 credentials, since OpenDAL replaces its whole provider chain, so one is installed only when the connection asked for delegation.
1 parent 40b7bea commit 166be2a

3 files changed

Lines changed: 411 additions & 28 deletions

File tree

src/sql/src/pure.rs

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -625,8 +625,10 @@ async fn purify_create_sink(
625625
// Now that we've validated the sink's storage creds (if they exist)
626626
// we _could_ use them to build a complete Iceberg client (both catalog and storage).
627627
// TODO(kynan): Actually use those sink-specific creds here instead of ignoring them.
628+
// Purification only proves the catalog is reachable, so it needs no table-scoped
629+
// storage credentials.
628630
let _catalog = connection
629-
.connect(storage_configuration, InTask::No)
631+
.connect(storage_configuration, InTask::No, None)
630632
.await
631633
.map_err(|e| IcebergSinkPurificationError::CatalogError(Arc::new(e)))?;
632634
}

0 commit comments

Comments
 (0)