Circuit-break during remote tier outages #21
Description
Is your feature request related to a problem? Please describe.
S3 doesn't have 100.0% availability - sometimes there are outages. The plugin should try to fail gracefully in this scenario.
Describe the solution you'd like
The plugin should attempt to detect when most requests to the remote tier are failing and enter a failure state where remote-tier operations are paused until the event is resolved. That failure state should behave like regular non-tiered-storage streaming. Instead of retrying requests aggressively we should retry a low number of requests infrequently to avoid spamming the remote tier during its recovery.
Describe alternatives you've considered
We could block all streaming during a remote-tier outage. I think having some availability is better than strong consistency during an outage though.