Commit 166be2a
committed
storage: Refresh catalog-vended storage credentials
Vended credentials expire, and nothing refreshed them: `loadTable` hands
their access keys to the FileIO once and OpenDAL keeps signing with them
until the dataflow restarts.
Adds `VendedCredentialLoader`, a `ProvideCredential` implementation that
re-fetches from the catalog's `loadCredentials` endpoint. OpenDAL rebuilds
its `Operator` for every file operation, so reqsign's own credential cache
never survives one call; the loader therefore caches with its own expiry
deadline, taken from `s3.session-token-expires-at-ms` where the catalog
reports one and a short interval where it does not. On a 401 or 403 it
invalidates the catalog token so the next attempt mints a fresh one.
Materialize now builds the OAuth2 provider itself rather than passing a
`credential` catalog property, so one token object serves both catalog
requests and credential refreshes. The catalog client rejects a custom
authenticator combined with that property, so the two cannot coexist.
`connect` grows a table argument because the credentials endpoint is
table-scoped. Installing a loader also means it alone supplies S3
credentials, since OpenDAL replaces its whole provider chain, so one is
installed only when the connection asked for delegation.1 parent 40b7bea commit 166be2a
3 files changed
Lines changed: 411 additions & 28 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
625 | 625 | | |
626 | 626 | | |
627 | 627 | | |
| 628 | + | |
| 629 | + | |
628 | 630 | | |
629 | | - | |
| 631 | + | |
630 | 632 | | |
631 | 633 | | |
632 | 634 | | |
| |||
0 commit comments