Metabase gets stuck in migrations mid deployment

[euanmillar]

Describe the bug
On deloyment, Metabase dashboards were not starting with a 502 Bad Gateway error.

Checking metabase logs led to this error being logged:

"Database has migration lock; cannot run migrations. You can force-release these locks by running `java -jar metabase.jar migrate release-locks."

Checking Slack history I found this discussion:

https://opencrvsworkspace.slack.com/archives/C02LU432JGK/p1680174580010749

But I wasnt clear on how to run the 5 steps that are documented.

This Slack conversation from later had some clearer instructions:

https://opencrvsworkspace.slack.com/archives/C033B06F2EM/p1708609516617079

When running these commands, Metabase no longer logged the migration error, but still had not started. I tried to re-deploy the entire stack.

Still Metabase would not start up:

opencrvs_dashboards logs now look like this. It looks like no major errors but dashboards are not working:

opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,641 INFO jdbcjobstore.JobStoreTX :: Using db table-based data access locking (synchronization).
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,641 INFO jdbcjobstore.JobStoreTX :: JobStoreTX initialized.
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,641 INFO core.QuartzScheduler :: Scheduler meta-data: Quartz Scheduler (v2.3.2) 'MetabaseScheduler' with instanceId '83f3224108861747144428638'
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    |   Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    |   NOT STARTED.
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    |   Currently in standby mode.
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    |   Number of jobs executed: 0
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    |   Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    |   Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is clustered.
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    |
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,641 INFO impl.StdSchedulerFactory :: Quartz scheduler 'MetabaseScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,641 INFO impl.StdSchedulerFactory :: Quartz scheduler version: 2.3.2
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,760 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_83f3224108861747144428638 paused.
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,760 INFO metabase.task :: Task scheduler initialized into standby mode.
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,770 INFO metabase.task :: Deleting job metabase.task.truncate-audit-log.job due to class not found
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,775 INFO metabase.task :: Initializing task Cache 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,776 INFO metabase.task :: Initializing task SyncDatabases 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,783 INFO task.sync-databases :: Updated default schedules for 0 databases
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,783 INFO metabase.task :: Initializing task PersistRefresh 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,796 INFO driver.impl :: Initializing driver :mongo...
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,798 INFO plugins.classloader :: Added URL file:/plugins/mongo.metabase-driver.jar to classpath
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:48,799 DEBUG plugins.init-steps :: Loading plugin namespace metabase.driver.mongo...
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,044 INFO driver.impl :: Registered driver :mongo  🚚
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,053 INFO metabase.util :: Load lazy loading driver :mongo took 253.8 ms
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,071 INFO metabase.task :: Initializing task CheckForNewVersions 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,083 INFO metabase.task :: Initializing task PersistPrune 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,086 INFO metabase.task :: Initializing task SendAnonymousUsageStats 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,098 INFO metabase.task :: Initializing task ModelIndexValues 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,100 INFO metabase.task :: Initializing task RefreshSlackChannelsAndUsers 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,113 INFO metabase.task :: Initializing task TruncateAuditTables 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,116 INFO metabase.task :: Initializing task SendPulses 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,120 INFO metabase.task :: Initializing task SendFollowUpEmails 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,127 INFO metabase.task :: Initializing task SendCreatorSentimentEmails 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,130 INFO metabase.task :: Initializing task SendLegacyNoSelfServiceEmail 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,132 INFO metabase.task :: Initializing task TaskHistoryCleanup 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,138 INFO metabase.task :: Initializing task SendWarnPulseRemovalEmail 📆
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,143 INFO core.QuartzScheduler :: Scheduler MetabaseScheduler_$_83f3224108861747144428638 started.
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,144 INFO metabase.task :: Task scheduler started
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,146 INFO jdbcjobstore.JobStoreTX :: Handling 2 trigger(s) that missed their scheduled fire-time.
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,183 INFO task.refresh-slack-channel-user-cache :: Slack is not configured, not refreshing slack user/channel cache.
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:53:49,186 INFO metabase.core :: Metabase Initialization COMPLETE in 23.2 s
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:54:04,145 INFO jdbcjobstore.JobStoreTX :: ClusterManager: detected 1 failed or restarted instances.
opencrvs_dashboards.1.m905a2t1k9k6@opencrvs-mosip-dev    | 2025-05-13 13:54:04,145 INFO jdbcjobstore.JobStoreTX :: ClusterManager: Scanning for instance "83f3224108861747144426834"'s failed in-progress jobs.
^Ccontext canceled

Expected behaviour

Metabase should deploy without any migration issue requiring any manual steps required to resolve the situation

Screenshots

OpenCRVS Core Version:

• v1.8.0

Country Configuration Version:

• v1.8.0 (Git branch: master / release-v1.1.0)