Connection stays open randomly after closing/crashing consumer

[Matts]

Hi, I have no idea how this or why this happends. But on our enterprise application (with 2 consumer connections always open) it sometimes happends that the consumer restarts or loses connection. After this RabbitMQ Manager shows that there are now 4 connections (instead of the intended 2). This keeps duplicating over time.
Only solution we have found. Is to manually log into the management page and kill the connections manually.

Do you recognise this issue or have a fix for it available?

[Matts]

Another one of our applications, that uses the Symfony Messenger, does not have this issue. So I don't think this is an architectural IT issue on our side.

[Matts]

Also using SimpleBus/rabbitmq-bundle-bridge but thinking this would be an issue on this level seeing as they don't do anything with connection. And it also happends when just using a normal consumer

I am thinking it may have something to do with the sigkill/sigterm

[MKCG]

One workaround is to defined a heartbeat in order for the rabbitmq server to proactively close any dead tcp connection : https://www.rabbitmq.com/heartbeats.html

[mihaileu]

@Matts is this solving your issue ?

[Matts]

I will check this later today. I remember however something saying in the documentation that the heartbeat should be 2x the time of your running consumer. This can possibly cause an issue as our consumers can sometimes be processing for multiple hours. Will reply with results.

[Matts]

What I found, was that adding a graceful termination period to my k8s helm config solved the issue partially
terminationGracePeriodSeconds https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-handler-execution

Now is just still the issue that the consumer does not detect a closed TCP connection after running for a while. I expect that with heartbeats/keepalive I may be able to solve it.