Reconnect the database after any failed worker cycle - #170
Merged
Conversation
A failed cycle can leave the connection in a state it never recovers from on its own. When a transaction is aborted and the rollback that follows also fails, the connection throws away its own transaction bookkeeping while the driver keeps considering a transaction open. Every later attempt to begin one is then rejected by the driver with "There is already an active transaction", which is not a lost connection, so the worker held on to the connection and failed every single cycle until it was restarted by hand.
firecow
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The worker only reconnected when a cycle failed because of a lost database connection. Any other failure left the connection in place, which is not always safe to do.
A transaction that is aborted by a connection dying mid-flight makes Laravel roll back, and that rollback fails too, because there is nothing left to talk to.
handleRollBackException()then resetstransactionsto0while keeping the PDO instance, so the connection believes it has no transaction open, but the driver still believes it has:rollBack()only clears that flag when it succeeds, and it can never succeed again on a dead socket.From there the connection is permanently unusable.
beginTransaction()seestransactions === 0,reconnectIfMissingConnection()does nothing because the PDO instance is not null, and the driver rejects theBEGINbefore it even reaches the network:That message is not in the lost connection list, so the worker logged it and carried straight on to the next cycle with the same broken connection, failing every cycle at ~10 errors per second until the process was restarted by hand.
Giving up on the connection after any failed cycle covers this, and everything else where a failure leaves state behind that the connection cannot clear itself. The reconnect is cheap next to the 100 ms the worker already sleeps after a failed cycle, and the existing logging policy is unchanged: lost connections stay quiet until they start repeating, anything else is still reported in full.