Backup times-out for no apparent reason

[HadrienKulik]

Here is the setup :

• two app servers : main and secondary (main being slightly more powerful, otherwise same exact Docker setup and codebase)
• two sql servers : main and secondary, perfectly identical, initially created from the same sql dump, secondary being a slave to main
• the app servers are connected to their sql counterparts (main on main, secondary on secondary)
• there are hundreds of databases to backup

We have laravel-backup running on both the main and secondary app servers, at different times of the day.

Both backups used to work perfectly fine for years.

Now, the backup on main fails every day, with the following error:

But the exact same backup, on the slave database (on secondary) works without any problem.

It's not an issue of storage on the server, the database it gets stuck on is not particularly a big one. The database on which it gets stuck did change overtime a bit, leading us to think it was an overall timeout issue. We tried increasing the sql dump timeout from 600 to 900s but it has no effect, it got stuck on the same database after changing the setting. We changed both servers hosting the sql databases, the issue remained the same. The time of the day, and the load on the server does not have an effect on the issue.

Do you have any clue why it could get stuck on one of the two servers (the more powerful one), and how to diagnose further?
Thanks!