retries for alembic migration for authn on startup

[swelborn]

added this retry code to startup to try and try again if we have many replicas on startup. this should mitigate having to bootstrap a cluster manually with 1 replica on, then spinning up more.

not sure if the code is in the right place. tried conforming to lazy imports, but moved function outside of deeply nested app startup code. since that is tough to read.

Checklist

• Add a Changelog entry
• Add the ticket number which this PR closes to the comment section

[swelborn]

closes #1447

[danielballan]

Thanks for moving it outside the app startup code; I agree that that has overgrown and become hard to read.

I wonder if, instead of retries, we should aim for CREATE TABLE IF NOT EXISTS. The first to open the transaction will do the work, and the rest will wait their turn and then discover that there is nothing left to do.

[swelborn]

good morning dan

do you mean to move the subprocess in-process? trying to think about which replica grabs the lock to do the table creation + alembic upgrade + role creation + stamp version, and how the lock is held thru that process, without some exception handling like we have in this retry code