Insufficient exchange.declare permissions during connection recovery trip up topology recovery

[hennevogel]

We are using Bunny 3.10 on Ruby 3.4 to publish to rabbitmq from a bunch of RoR instances. We rely on automatic recovery and observe the following:

Every time we run into a network error that triggers automatic recovery, the first recovery attempt is failing to create a channel and subsequently declare the exchange on it. Once we "just" restart our RoR threads and the connection is made fresh everything works.

It looks like somehow Bunny::Session.recover_exchange and Bunny::Exchange.declare! behave differently somehow, but I can't for the life of me not figure out what's going on... Any idea?

The setup is fairly easy. Just a PORO for publishing and a fairly limited amount of Bunny options...

    host: rabbit.obsmgmt.opensuse.org
    port: 5671
    user: $USER
    pass: $PASS 
    ssl: true
    heartbeat_interval: 10
    verify_peer: false
    recovery_attempts: 75

Logs

Bunny Logs

E, [2026-04-23T08:54:41.307405 #11486] ERROR -- #<Bunny::Session:0x1a740 $USER@rabbit.obsmgmt.opensuse.org:5671, vhost=/, addresses=[rabbit.obsmgmt.opensuse.org:5671]>: Got an exception when receiving data: Connection reset by peer (Errno::ECONNRESET)
W, [2026-04-23T08:54:41.307617 #11486]  WARN -- #<Bunny::Session:0x1a740 $USER@rabbit.obsmgmt.opensuse.org:5671, vhost=/, addresses=[rabbit.obsmgmt.opensuse.org:5671]>: Will recover from a network failure (75 out of 75 left)...
E, [2026-04-23T08:54:51.337585 #11486] ERROR -- #<Bunny::Session:0x1a740 $USER@rabbit.obsmgmt.opensuse.org:5671, vhost=/, addresses=[rabbit.obsmgmt.opensuse.org:5671]>: Caught an exception while recovering exchange pubsub: #<Bunny::AccessRefused: ACCESS_REFUSED - configure access to exchange 'pubsub' in vhost '/' refused for user '$USER'>

RabbitMQ logs

2026-04-23 08:54:51.332284+00:00 [info] <0.7590156.0> accepting AMQP connection <0.7590156.0> (192.168.2.100:42482 -> 192.168.2.130:5671)
2026-04-23 08:54:51.335098+00:00 [info] <0.7590156.0> connection <0.7590156.0> (192.168.2.100:42482 -> 192.168.2.130:5671): user '$USER' authenticated and granted access to vhost '/'
2026-04-23 08:54:51.336793+00:00 [error] <0.7591027.0> Channel error on connection <0.7590156.0> (192.168.2.100:42482 -> 192.168.2.130:5671, vhost: '/', user: '$USER'), channel 1:
2026-04-23 08:54:51.336793+00:00 [error] <0.7591027.0> operation exchange.declare caused a channel exception access_refused: configure access to exchange 'pubsub' in vhost '/' refused for user '$USER'

[michaelklishin]

Connection recovery is well covered with tests and you are the first person to report this.

The code in question is very curiously structured, a method that is supposed to return an exchange runs Bunny::Session#start but a method that supposedly "starts" a connection does not.

[hennevogel]

The code in question is very curiously structured...

That's us! very curiously structured is our middle name ;-)

[michaelklishin]

@hennevogel ahem, the very logs you have shared have a visible red flag:

E, [2026-04-23T08:54:51.337585 #11486] ERROR -- #<Bunny::Session:0x1a740 $USER@rabbit.obsmgmt.opensuse.org:5671, vhost=/, addresses=[rabbit.obsmgmt.opensuse.org:5671]>: Caught an exception while recovering exchange pubsub: #<Bunny::AccessRefused: ACCESS_REFUSED - configure access to exchange 'pubsub' in vhost '/' refused for user '$USER'>

ACCESS_REFUSED - configure access to exchange 'pubsub' in vhost '/' refused for user '$USER'

2026-04-23 08:54:51.336793+00:00 [error] <0.7591027.0> Channel error on connection <0.7590156.0> (192.168.2.100:42482 -> 192.168.2.130:5671, vhost: '/', user: '$USER'), channel 1:
2026-04-23 08:54:51.336793+00:00 [error] <0.7591027.0> operation exchange.declare caused a channel exception access_refused: configure access to exchange 'pubsub' in vhost '/' refused for user '$USER'

The exchange.declare protocol method requires a configure permission on the target exchange.

[hennevogel]

Yes, that is exactly what goes wrong while recovering the exchange and does not go wrong when declaring it. Same host, same Bunny setup, same RabbitMQ configuration. The only difference is recovering vs. declaring.

[hennevogel]

But indeed you are right, the user in question has no configure permissions. Hm so recovering requires more permissions...

[michaelklishin]

@hennevogel recovering does not require more permissions but an authorization failure trips up topology recovery in a way that Bunny (or any other client) cannot do anything about, so it does not even try [to, ahem, retry].

Two related or potentially related topics that are not obvious:

1. Bunny caches declarations, e.g. Bunny::Channel#durable_queue will declare a queue on the first call but then return the same Bunny::Queue on subsequent calls without contacting the server; this is to avoid unintentional synchronous (as in AMQP 0-9-1 method type) requests when you use Bunny's DSL heavily
2. 4.3.0, 4.2.6, 4.1.11 and so on all the way to 3.13.15 now require the same permissions for regular declarations as for passive ones.

[hennevogel]

Why are we running into an authorization failure when recovering but not when declaring? That is what we don't understand.

[michaelklishin]

@hennevogel we do not guess in this community. Enable debug logging on the node, set Bunny log level to debug (it will log protocol traffic) and see what the difference is, or take and inspect a traffic capture.

There isn't anything particular clever in the recovery algorithm and N clients do exactly the same thing.