Is your feature request related to a problem or challenge? Please describe what you are trying to do.
One of our production process started failing every GCS upload with 403 storage.objects.create denied and never recovered on its own. Digging in, the trigger was a transient GKE metadata-server fault that handed the pod a perfectly valid token, just for the wrong identity. From that point on every request was rejected, but the 403s did nothing to the cached credential: object_store held onto that wrong-identity token for the full ~1h Workload Identity TTL, and the only way out was restarting the process.
The reason it can't heal itself is that the TokenCache behind the cloud CredentialProviders (AWS / Azure / GCP) is purely expiry-driven, a cached token is reused until it's within min_ttl of expiring, and there's no way for a caller to say "this token just got rejected, drop it and fetch a fresh one." So a single bad fetch is sticky: as long as the token hasn't expired, the cache keeps handing it back no matter how many 401/403s the downstream requests return.
Describe the solution you'd like
Add a defaulted, backwards-compatible invalidation hook to the CredentialProvider trait. Proof-of-concept against the v0.13.1 version tag: r0mdau/arrow-rs-object-store@2961575.
A caller that observes sustained auth failures can then call invalidate(), and the next get_credential() re-resolves identity from the underlying source (metadata server, STS, etc.). Providers that don't cache, e.g. StaticCredentialProvider, inherit the no-op default and are unaffected. The clear() method is feature-gated to aws / azure / gcp, matching the rest of TokenCache.
Describe alternatives you've considered
- Wrap the provider app-side to interpose our own clearable cache. This duplicates the crate's token logic and can't reach the cache inside the built-in
InstanceCredentialProvider, which is exactly the one we need to invalidate.
- Wait for expiry. Up to ~1h of downtime, and it doesn't help when the underlying fault keeps renewing the same bad token.
- A
force_refresh() variant instead of clear. clear() is simpler and lets the normal fetch path repopulate on the next call; a dedicated refresh method adds surface area for no extra benefit here.
Additional context
Happy to open the PR. The change is purely additive, a new defaulted trait method, one pub(crate) cache method, and one provider override. So it should carry no compatibility risk for existing implementors.
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
One of our production process started failing every GCS upload with
403 storage.objects.create deniedand never recovered on its own. Digging in, the trigger was a transient GKE metadata-server fault that handed the pod a perfectly valid token, just for the wrong identity. From that point on every request was rejected, but the 403s did nothing to the cached credential:object_storeheld onto that wrong-identity token for the full ~1h Workload Identity TTL, and the only way out was restarting the process.The reason it can't heal itself is that the
TokenCachebehind the cloudCredentialProviders (AWS / Azure / GCP) is purely expiry-driven, a cached token is reused until it's withinmin_ttlof expiring, and there's no way for a caller to say "this token just got rejected, drop it and fetch a fresh one." So a single bad fetch is sticky: as long as the token hasn't expired, the cache keeps handing it back no matter how many 401/403s the downstream requests return.Describe the solution you'd like
Add a defaulted, backwards-compatible invalidation hook to the
CredentialProvidertrait. Proof-of-concept against thev0.13.1version tag: r0mdau/arrow-rs-object-store@2961575.A caller that observes sustained auth failures can then call
invalidate(), and the nextget_credential()re-resolves identity from the underlying source (metadata server, STS, etc.). Providers that don't cache, e.g.StaticCredentialProvider, inherit the no-op default and are unaffected. Theclear()method is feature-gated toaws/azure/gcp, matching the rest ofTokenCache.Describe alternatives you've considered
InstanceCredentialProvider, which is exactly the one we need to invalidate.force_refresh()variant instead of clear.clear()is simpler and lets the normal fetch path repopulate on the next call; a dedicated refresh method adds surface area for no extra benefit here.Additional context
Happy to open the PR. The change is purely additive, a new defaulted trait method, one
pub(crate)cache method, and one provider override. So it should carry no compatibility risk for existing implementors.