Support n-1 version compatibility

[mooreds]

Support n-1 version compatibility

Previously Support zero downtime upgrades, renaming to be more accurate.

This was a feature request from a user. Thanks for the feedback, @trevorr !

Problem

"There is no official support for zero-downtime upgrades. Downtime causes us contractual headaches around SLAs and personal headaches around performing upgrades during the middle of the night on a weekend. I've done this successfully at my own risk, but it will make me increasingly nervous. This feature seems doable, even in the face of schema upgrades, with some constraints (both for FA and its users) on consecutive versions."

Note that the only current downtime if you run multiple nodes is when there are database upgrades and the database after upgrade is not compatible with version N-1. This downtime is typically seconds in duration, but does depend on the size of your data.

Solution

Ensure that version N and version N-1 of the application work with version N and version N-1 of the database, so that you can step from N-1 -> N with both nodes and the database with zero degradation.

Alternatives/workarounds

Automate the upgrade, do it in the middle of the night or other low traffic time.

Remain on an older version.

Provide a 'read only' degraded mode when you are upgrading or figure out some other way for the system to operate in a degraded, but still available mode during a database upgrade. Another option would be for a clear blue/green strategy set of instructions, which would probably impose some version to version limits on database changes.

Additional context

Modified Apr 13 2022 to be more precise. There are many approaches to minimizing downtime of upgrades.

Docs around upgrading currently: https://fusionauth.io/docs/v1/tech/admin-guide/upgrade#downtime

Community guidelines

All issues filed in this repository must abide by the FusionAuth community guidelines.

How to vote

Please give us a thumbs up or thumbs down as a reaction to help us prioritize this feature. Feel free to comment if you have a particular need or comment on how this feature should work.

[mooreds]

Here's one approach (postgresql only): https://fabianlindfors.se/blog/schema-migrations-in-postgres-using-reshape/

[fabianlindfors]

Hi @mooreds! I'm the creator of Reshape and author of that post. Let me know if you'd be interested in using Reshape, I'd be happy to help in any way I can.

The tool is still experimental but I'm eager to work with users and bring it to a production-ready state.

[matthewhartstonge]

Chiming in here around support for zero-downtime upgrades:

• State based regulatory/contractual requirements now means we have to give at least 15 Business Days’ notice in writing before any planned outage for maintenance and upgrades.

Imagine another log4shell~esque debacle, we would have to be in a state of vulnerability for at least 15 days without a zero downtime upgrade strategy/pathway.

To be clear to others, FusionAuth was not affected by log4j/log4shell

[glen-84]

Minutes of downtime for a patch release with no schema changes is really unfortunate. 😐 This shouldn't require HA.

[robotdan]

This issue is primarily focused on official support for n-1 to support true rolling upgrades in a multi-node configuration.

Minutes of downtime for a patch release with no schema changes is really unfortunate. 😐 This shouldn't require HA.

If you are managing your own upgrades and there are no schema changes, upgrades should be seconds. Not sure what you mean about HA. Are you referring to the FusionAuth Cloud offering, or a generic multi-node FusionAuth deployment?

[glen-84]

If you are managing your own upgrades and there are no schema changes, upgrades should be seconds.

I upgraded our staging instance which is on basic cloud (from 1.36.4 to 1.36.7), and I think it took 2-3 minutes.

Not sure what you mean about HA. Are you referring to the FusionAuth Cloud offering, or a generic multi-node FusionAuth deployment?

The cloud offering. We're using Basic for staging, and Business for production. Now I'm a bit worried that production will go down for minutes as well if I upgrade it.

[robotdan]

If you require fast upgrades w/out downtime, you will need to upgrade to HA, that is the only cloud offering with full redundant services. If you want to discuss Cloud Offerings, let's move this discussion to the forum.

[Manouchehri]

@robotdan It is kinda unexpected that even most cloud customers do not have HA (only the highest paid plan appears to offer it). $6,000/year for HA seems insane.

[mooreds]

Hi @Manouchehri ,

Thanks for your feedback! We love to hear from customers and users of FusionAuth. In the spirit of full transparency, you can see our pricing here: https://fusionauth.io/pricing?step=plan&hosting=ha-cloud For new customers, HA would require the Essentials plan, which makes it more than $6k/year (but includes support).

However, sometimes our pricing is confusing, so I wanted to respond with some options. If you need high uptime numbers from your FusionAuth system, you have a couple of choices:

• Pay FusionAuth to run, maintain and operate a multi-node cluster on your behalf (the HA version that @robotdan mentioned). This has multiple server nodes, a database with replication and automated backups, and other supporting infrastructure.
• Run in a less expensive FusionAuth Cloud configuration (say Business Cloud) and minimize the number of times you upgrade. If an upgrade takes 1 hour and you only do it once a year, that is 99.99% uptime. Three 1 hour upgrades is 99.96% uptime. (Not SLAs, just examples.) You control your upgrade cycle with FusionAuth Cloud. We have plenty of customers that stay close to the latest version, but plenty that do not, and might upgrade 1-3 times a year.
• Run it yourself in a multi-node configuration. I'm sure you understand building out infrastructure, processes and personnel to support FusionAuth Cloud isn't cheap. But I've run a startup before. I understand cost pressures, and if you need to minimize out of pocket costs, you can build out a multi-node cluster yourself and run it on any cloud. Here's the documentation to do so: https://fusionauth.io/docs/v1/tech/installation-guide/cluster We have folks running FusionAuth on multi-node clusters who've never paid us a cent, and we love it.

Again, we really appreciate hearing from people considering FusionAuth, so thank you for your feedback. I hope one of the options above will work for you, but we understand that no product is the right fit for everyone.