Enable failover to different backend bucket

[jabbrwcky]

Is your feature request related to a problem? Please describe.

I had the problem in the past (and lately again) to move data from one S3 provider to another (or from one bucket to another) while services are using them without interruption.

Migrating the data is out of scope, I usually just set up a cron job that moves data to the new destination using s3sync.

The problem, which s3-proxy could help solve, is that you have to coordinate switching consumers from old to new s3 location without experiencing interruptions.

Describe the solution you'd like

S3-proxy could allow mapping a resource to a list of mounted backend buckets (e.g. new location first, old location second). If an object is not available at the first bucket (new), the request is retried at the second backend location (old). The request only fails if the object is not resolved at the end of the list of configured buckets.

This would facilitate uninterrupted migrations at the cost of eventually slightly longer response times.

[oxyno-zeta]

Hello @jabbrwcky ,

Thanks for this idea.
This will require a huge rework and configuration will have to rework from scratch.

I'm not sure that this will work but maybe yes: maybe 2 targets on different domains with a first one on the first bucket and the second on the second. When hitting a 404, perform a temporary redirect to the second DNS?

Best regards,

[jabbrwcky]

Interesting idea. I'll give it a try and see how it works