Federated MQTT exchange is not restored when network cable is unplugged and plugged again

[RPelleg]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.3.1

Erlang version used

27.3.x

Operating system (distribution) used

Windows Server 2022

How is RabbitMQ deployed?

Windows installer

rabbitmq-diagnostics status output

C:\Program Files\RabbitMQ Server\rabbitmq_server-4.3.1\sbin>rabbitmq-diagnostics federation_status

Listing federation links on node rabbit@DIGISTAT_NODE2...
[#{error =>
       {gen_server,call,
           [<13687.2598.0>,
            {call,
                {'exchange.unbind',0,
                    <<"federation: amq.topic -> rabbit@DIGISTAT_NODE2 A">>,
                    <<"amq.topic">>,<<"t.s.dig.dataset.p.*.1.*">>,false,
                    [{<<"x-binding-key">>,longstr,
                      <<"t.s.dig.dataset.p.*.1.*">>},
                     {<<"x-bound-from">>,array,
                      [{table,
                           [{<<"cluster-name">>,longstr,
                             <<"rabbit@DIGISTAT_NODE2">>},
                            {<<"vhost">>,longstr,<<"digistat">>},
                            {<<"exchange">>,longstr,
                             <<"digistat:amq.topic A">>},
                            {<<"hops">>,short,1}]}]},
                     {<<"x-mqtt-subscription-opts">>,table,
                      [{<<"no-local">>,bool,false},
                       {<<"qos">>,unsignedbyte,0},
                       {<<"retain-as-published">>,bool,false},
                       {<<"retain-handling">>,unsignedbyte,0}]}]},
                none,<13687.2541.0>},
            130000]},
   id => <<"2b86c892">>,status => shutdown,type => exchange,
   exchange => <<"amq.topic">>,queue => <<>>,
   uri => <<"amqp://10.30.185.53:5672/digistat">>,vhost => <<"digistat">>,
   last_changed => <<"2026-06-17 10:25:53">>,local_connection => <<>>,
   upstream_exchange => <<"amq.topic">>,upstream_queue => <<>>,
   upstream => <<"to-rabbit3">>},
 #{error =>
       {gen_server,call,
           [<13687.3082.0>,
            {call,
                {'exchange.unbind',0,
                    <<"federation: amq.topic -> rabbit@DIGISTAT_NODE2 B">>,
                    <<"amq.topic">>,<<"t.s.dig.dataset.p.*.1.*">>,false,
                    [{<<"x-binding-key">>,longstr,
                      <<"t.s.dig.dataset.p.*.1.*">>},
                     {<<"x-bound-from">>,array,
                      [{table,
                           [{<<"cluster-name">>,longstr,
                             <<"rabbit@DIGISTAT_NODE2">>},
                            {<<"vhost">>,longstr,<<"digistat">>},
                            {<<"exchange">>,longstr,
                             <<"digistat:amq.topic B">>},
                            {<<"hops">>,short,1}]}]},
                     {<<"x-mqtt-subscription-opts">>,table,
                      [{<<"no-local">>,bool,false},
                       {<<"qos">>,unsignedbyte,0},
                       {<<"retain-as-published">>,bool,false},
                       {<<"retain-handling">>,unsignedbyte,0}]}]},
                none,<13687.3046.0>},
            130000]},
   id => <<"b5e3f26c">>,status => shutdown,type => exchange,
   exchange => <<"amq.topic">>,queue => <<>>,
   uri => <<"amqp://10.30.185.51:5672/digistat">>,vhost => <<"digistat">>,
   last_changed => <<"2026-06-17 10:25:58">>,local_connection => <<>>,
   upstream_exchange => <<"amq.topic">>,upstream_queue => <<>>,
   upstream => <<"to-rabbit2">>}]

C:\Program Files\RabbitMQ Server\rabbitmq_server-4.3.1\sbin>rabbitmq-diagnostics status
Status of node rabbit@DIGISTAT_NODE2 ...
Runtime

OS PID: 4604
OS: Windows
Uptime (seconds): 4212
Is under maintenance?: false
RabbitMQ version: 4.3.1
RabbitMQ release series support status: see https://www.rabbitmq.com/release-information
Node name: rabbit@DIGISTAT_NODE2
Erlang configuration: Erlang/OTP 27 [erts-15.2.7.9] [source] [64-bit] [smp:24:24] [ds:24:24:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 3.5.5 27 Jan 2026
Erlang processes: 633 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: c:/ProgramData/RabbitMQ/enabled_plugins
Enabled plugins:

• rabbitmq_federation_management
• rabbitmq_federation
• rabbitmq_exchange_federation
• rabbitmq_queue_federation
• rabbitmq_federation_common
• rabbitmq_web_mqtt
• rabbitmq_mqtt
• rabbitmq_management
• rabbitmq_management_agent
• rabbitmq_web_dispatch
• cowboy
• oauth2_client
• jose
• amqp_client

Data directory

Node data directory: c:/ProgramData/RabbitMQ/db/rabbit@DIGISTAT_NODE2-mnesia
Raft data directory: c:/ProgramData/RabbitMQ/db/rabbit@DIGISTAT_NODE2-mnesia/quorum/rabbit@DIGISTAT_NODE2

Config files

• c:/ProgramData/RabbitMQ/advanced.config

Log file(s)

• c:/ProgramData/RabbitMQ/log/rabbit@DIGISTAT_NODE2.log

Alarms

(none)

Tags

(none)

Memory

Total memory used: 0.1399 gb
Calculation strategy: rss
Memory high watermark setting: 0.6 of available memory, computed to: 10.3073 gb

other_proc: 0.0337 gb (24.12 %)
other_system: 0.0334 gb (23.86 %)
allocated_unused: 0.0323 gb (23.08 %)
code: 0.0252 gb (18.03 %)
metrics: 0.0028 gb (2.02 %)
plugins: 0.0022 gb (1.58 %)
reserved_unallocated: 0.0021 gb (1.5 %)
binary: 0.002 gb (1.4 %)
other_ets: 0.0011 gb (0.81 %)
atom: 0.0011 gb (0.81 %)
msg_index: 0.001 gb (0.7 %)
mgmt_db: 0.0009 gb (0.68 %)
metadata_store: 0.0007 gb (0.5 %)
connection_other: 0.0005 gb (0.33 %)
connection_readers: 0.0003 gb (0.22 %)
connection_channels: 0.0003 gb (0.22 %)
metadata_store_ets: 0.0001 gb (0.07 %)
queue_procs: 0.0001 gb (0.05 %)
connection_writers: 0.0 gb (0.01 %)
quorum_ets: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
mnesia: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 24.1888 gb

Totals

Connection count: 9
Queue count: 9
Virtual host count: 2

Listeners

Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Interface: 0.0.0.0, port: 1883, protocol: mqtt, purpose: MQTT
Interface: [::], port: 15675, protocol: http/web-mqtt, purpose: MQTT over WebSocket
Interface: 0.0.0.0, port: 15675, protocol: http/web-mqtt, purpose: MQTT over WebSocket
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: 0.0.0.0, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Logs from node 1 (with sensitive values edited out)

rabbit@DIGISTAT_NODE2.log

rabbitmq.conf

rabbitmq.conf not present on the disk, because everything has been configured by command line. Here you have the output of the following commands:

rabbitmqctl.bat export_definitions C:\temp\defs.json
rabbitmqctl.bat list_policies --vhost=digistat > C:\temp\policies.txt
rabbitmqctl.bat list_parameters --vhost=digistat > C:\temp\parameters.txt
policies.txt
defs.json
parameters.txt

Steps to deploy RabbitMQ cluster

It is not a cluster, but multiple federated nodes

Steps to reproduce the behavior in question

Disconnect and reconnect the network cable in the VM settings of VMWare Workstation. Not tested with a physical cable.

advanced.config

Empty

What problem are you trying to solve?

OS: Windows Server 2022
RabbitMQ: 4.3.1, replicated also on 4.2.3

I have three nodes not in a cluster, but with a full-mesh federation (max-hop = 1). I have federated directly MQTT exchange (amq.topic). We are testing the stability of the federation link for a production environment.
Test was made on node 2.

Unfortunately, if I disconnect and reconnect the network cable (virtually from VMWare Workstation VM settings) the federation links goes in shutdown and never restore.
Sadly the 130000 timeout for the unbind is not even applied, because the error is appearing as soon as the network cable is unplugged.
Federation link has heartbeat set to 15, connection timeout = 10000, max-hop = 1, reconnect-delay = 10

Everything is working well in other use cases, like:

• enable and disable the network card from the OS
• restart the VMs
• if no MQTT 5.0 client is connected, also plugging and unplugging the network cable works

I'm trying to understand if it is a bug or an expected behavior. Please let me know if you need further details and if you have suggestion for a workaround.

[mkuratczyk]

Just yesterday, we merged a bunch of improvements in this area: #16662
Quite likely these changes fix your problem. They will ship in RabbitMQ 4.3.3 in a few weeks.

[mkuratczyk]

No idea, sorry. I had never built the windows package before and I have not used Windows since Windows 3.11. :)

We should fix the workflow Michael mentioned so that we build these packages automatically.

[michaelklishin]

rabbitmq/server-packages@4978839 might fix the 4.4 alpha builds, we will find out in the next 12 hours or so.

[RPelleg]

Installed the latest 4.4 alpha. We have tested plug and unplug of the network cable on NODE2.
On Rabbit NODE2 there is a producer in AMQP.
On Rabbit NODE1 there is a consumer in MQTT.
Rabbit NODE3 does not have any client.

We have still the federation with NODE1 in shutdown state, but this time also federation with NODE3 is in shutdown state. The latter wasn't affected in 4.3.1 and 4.2.3.

Output of rabbitmq-diagnostics federation_status:

Listing federation links on node rabbit@DIGISTAT_NODE2...
[#{error =>
       {gen_server,call,
           [<13691.28992.0>,
            {call,
                {'exchange.unbind',0,
                    <<"federation: amq.topic -> rabbit@DIGISTAT_NODE2 A">>,
                    <<"amq.topic">>,<<"ROUNDTRIP.ACK">>,false,
                    [{<<"x-binding-key">>,longstr,<<"ROUNDTRIP.ACK">>},
                     {<<"x-bound-from">>,array,
                      [{table,
                           [{<<"cluster-name">>,longstr,
                             <<"rabbit@DIGISTAT_NODE2">>},
                            {<<"vhost">>,longstr,<<"digistat">>},
                            {<<"exchange">>,longstr,
                             <<"digistat:amq.topic A">>},
                            {<<"hops">>,short,1}]}]},
                     {<<"x-mqtt-subscription-opts">>,table,
                      [{<<"no-local">>,bool,false},
                       {<<"qos">>,unsignedbyte,0},
                       {<<"retain-as-published">>,bool,false},
                       {<<"retain-handling">>,unsignedbyte,0}]}]},
                none,<13691.28950.0>},
            130000]},
   id => <<"2b86c892">>,status => shutdown,type => exchange,
   exchange => <<"amq.topic">>,queue => <<>>,
   uri => <<"amqp://10.30.185.53:5672/digistat">>,vhost => <<"digistat">>,
   last_changed => <<"2026-06-22 15:37:16">>,local_connection => <<>>,
   upstream_exchange => <<"amq.topic">>,upstream_queue => <<>>,
   upstream => <<"to-rabbit3">>},
 #{error =>
       {gen_server,call,
           [<13691.29665.0>,
            {call,
                {'exchange.unbind',0,
                    <<"federation: amq.topic -> rabbit@DIGISTAT_NODE2 A">>,
                    <<"amq.topic">>,<<"ROUNDTRIP.ACK">>,false,
                    [{<<"x-binding-key">>,longstr,<<"ROUNDTRIP.ACK">>},
                     {<<"x-bound-from">>,array,
                      [{table,
                           [{<<"cluster-name">>,longstr,
                             <<"rabbit@DIGISTAT_NODE2">>},
                            {<<"vhost">>,longstr,<<"digistat">>},
                            {<<"exchange">>,longstr,
                             <<"digistat:amq.topic A">>},
                            {<<"hops">>,short,1}]}]},
                     {<<"x-mqtt-subscription-opts">>,table,
                      [{<<"no-local">>,bool,false},
                       {<<"qos">>,unsignedbyte,0},
                       {<<"retain-as-published">>,bool,false},
                       {<<"retain-handling">>,unsignedbyte,0}]}]},
                none,<13691.29619.0>},
            130000]},
   id => <<"b5e3f26c">>,status => shutdown,type => exchange,
   exchange => <<"amq.topic">>,queue => <<>>,
   uri => <<"amqp://10.30.185.51:5672/digistat">>,vhost => <<"digistat">>,
   last_changed => <<"2026-06-22 15:37:19">>,local_connection => <<>>,
   upstream_exchange => <<"amq.topic">>,upstream_queue => <<>>,
   upstream => <<"to-rabbit2">>}]

I see that amq.topic -> rabbit@DIGISTAT_NODE2 is logged in both. If I'm not mistaken, that should be the binding on remote amq.topic in NODE1 and NODE3. So seems like upstream connection is trying to unbind amq.topic on both NODE1 and NODE3, but without being able to communicate over the network the command gives an unrecoverable error.

[michaelklishin]

@RPelleg we need full logs, not a 50 line long snippet.

Use rabbitmq-lqt if you need to obfuscate virtual host names, usernames, queue names.

[RPelleg]

Here it is the log
obfuscate.zip