Clouddriver and Orca services with multiple replicas using MySQL get stuck with DATABASECHANGELOGLOCK

[j771]

While running multiple replicas of clouddriver and orca services and using MySQL storage for them, sometimes if a pod gets deleted that has a DATABASECHANGELOGLOCK set on a MySQL DB, the new pods that come up cannot access the MySQL DB because the the lock is set for the pod that does not exist any more.

The only way to get things running again is to manually delete the lock on the DB.

While it does not happen often, I have seen this issue with both orca and clouddriver (both running their own MySQL DB).

[man0s]

I've seen also the the same behaviour.

[aryabchenko]

Faced the same

[jmartinezfl]

In my case I think the problem is due to the fact that every time a service is updated two ReplicaSets are created. The first one generates a pod that blocks the DB and terminates, and when the second ReplicaSet pods try to connect, the error appears.

Is this a normal behavior?