Connection::connect hangs on first connection, if RabbitMQ is not available

[PetroniDavid]

I've found a possible problem that happens during the very first connection attempt to RabbitMQ.

Let me add a disclaimer upfront: everything has been manually verified, but some analysis has been done with Claude Sonnet 5. I haven't found any policy about use of AI for this repository, but I'd rather be clear upfront. This analysis was done manually - Sonnet helped out with the possible fix. More on that later.

What's wrong

In some test scenarios, we create a RabbitMQ instance and at the same time start a task that tries to use Connection::connect.
Lapin 4.10.0 provides a mechanism to auto-reconnect, and when trying that option, we found out that the connection would be stuck.
Specifically, we're setting up the connection like this

            Connection::connect(
                &uri,
                ConnectionProperties::default()
                    .enable_auto_recover()
                    .with_backoff(
                        ExponentialBuilder::default()
                            .with_min_delay(Duration::from_millis(500))
                            .with_max_delay(Duration::from_secs(5))
                            .without_max_times(),
                    ),
            ).await

and this call hangs forever.
For completeness: we're starting up rabbitmq using docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:4 and the logs we see are

// From RabbitMQ

   accepting AMQP connection 172.17.0.1:X -> 172.17.0.2:5672
   closing AMQP connection ... (duration: '10s'):
   {handshake_timeout,handshake}

// From Lapin

   ERROR io_loop: lapin::io_loop: Socket was readable but we read 0. This usually means that the connection is half closed, thus report it as broken.
   DEBUG io_loop: lapin::io_loop: Throttling 5s before reconnection to avoid flooding

I've used Wireshark to see what's happening, and apparently the header is not sent on the very first reconnection attempt. The tcp socket is opened, yet 0 bytes are transmitted. It seems that in this case, the header is missing. I can provide the pcapng recordings from the failed and successful attempts, in case.

Working solution (but - is this the right one?)

I'm attaching the patch file that seems to solve the problem for me.
At first, I had the feeling that just trying to restart the protocol communication in the re-connection case would be enough,
but I could not find a way to do that.

What Sonnet 5 proposed, though, is a 3-line change.
Briefly:

• in init_connection_recovery, clear_connection_steps(None) was called, which prevents resolver.reject from being called. IIUC, this prevents the error from "bubbling" up; I'd like to note, anyway, that the channel id seems to be hardcoded to 0, which means that this seems to already take into account that this is the very first connection attempt, AFAICT.
• in critical_error, we would only check whether a channel can_recover to init a connection recovery - but in my opinion, we should not.
• in handle_half_closed_connection, same as above: if we're in the first connection, we do not want to retry.

With the attached patch file, the problem disappears, and the very first connection attempt fails when an error is found.

Thanks!

0001-fix-connection-recovery-hangs-on-first-connection.patch

[Keruspe]

Thanks,

Will give it a look!
There's one chunk I'm not sure about that I want to investigate. Will get back to you soon

[Keruspe]

On the AI usage, I could add a section about it but it is welcome as long as it's not just random slop. I switched from absolutely against it to welcoming it - as long as it's used properly - when I actually tried it 😄

For this particular issue, just to make sure I fully grasped the extent of the problem: it only happens for the first connection, and only when reconnect is enabled, right?

[PetroniDavid]

For this particular issue, just to make sure I fully grasped the extent of the problem: it only happens for the first connection, and only when reconnect is enabled, right?

Confirmed. AFAICT, this seems to happen only on the very first connection, and only with the auto-reconnect feature enabled. With auto-reconnect off, there is no problem - in the sense that Lapin will correctly error out as the connection cannot be established. I created a small test binary, I can attach it in case (but it is really trivial :) )