Skip to content

rocket_db_pools: Allow pool creation to succeed even if DB unavailable (sqlx, maybe others) #2931

Open
@cormacrelf

Description

@cormacrelf

What's missing?

The sqlx driver for rocket_db_pools currently crashes at startup if your database is offline or if anything else fails -- sslrootcert is invalid/missing, user doesn't exist, password wrong, etc etc. This is sometimes useful in development as you are reminded to start your database if you forgot, but a log would work just as well. In production it is a bad choice.

The deadpool crate's second-top-billed feature is that pool creation never fails even if the database is unavailable:

Identical startup and runtime behaviour. When writing long running application there usually should be no difference between startup and runtime if a database connection is temporarily not available. Nobody would expect an application to crash if the database becomes unavailable at runtime. So it should not crash on startup either. Creating the pool never fails and errors are only ever returned when calling Pool::get().

I'd like that, basically for the reasoning given there. It is needlessly crashy to do so only at startup.

To motivate somewhat more, if you are not convinced:

  • I am currently debugging the provision + rotation of database credentials and TLS certificates to a Rocket application, and it is quite annoying having to restart a crash-loop-backoff rocket application deployment to see if one of the changes I have made works. K8s is able to update a rotated sslrootcert certificate file in-place without rebooting a container -- because of the crash behaviour, Rocket does not.
  • It would be much easier to recover from a failed database cluster if a crashed or recovering database didn't completely bring down everything else that used it. The startup behaviour means all the alarms go off during a deployment when in fact your database went down an hour ago and you already know that. It means you have to reboot more things more times, while you're already in an emergency, but also, you must wait until your DB has finished recovering from a backup before rebooting everything else, so this either
    • forces you to be really careful not to deploy to your HTTP servers, or allow them to crash, during a DB meltdown
    • or necessarily slows down the app recovery, by forcing it to happen serially

Ideal Solution

The sqlx implementation of rocket_db_pools' Database trait uses Pool::connect_with instead of Pool::connect_lazy_with. You can make a one-line change to switch to connect_lazy_with, and make it behave like deadpool.

You could also make it configurable, but if people really want their app to crash in this case, they can follow deadpool's advice and add a fairing that tries to make a connection and crashes the app. People writing .unwrap() in a fairing is probably easier than making this configurable.

Why can't this be implemented outside of Rocket?

It has been implemented outside of rocket. The deadpool backend behaves this way, I presume. But rocket_db_pools does not configure sqlx to do it.

Are there workarounds usable today?

To work around this limitation, you could probably use a custom Database or Pool implementation, but you would need to make your own wrapper type for sqlx::Pool in order to get around the fact that rocket_db_pools implements its traits on that. I will probably do this myself for now, if it turns out to be easy I'll post a snippet.

Alternative Solutions

No response

Additional Context

No response

System Checks

  • I do not believe that this feature can or should be implemented outside of Rocket.
  • I was unable to find a previous request for this feature.

Metadata

Metadata

Assignees

No one assigned

    Labels

    requestRequest for new functionality

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions