[Questions] x-expires deletes non-durable classic queue with active consumers (RabbitMQ 4.2.6 / CQv2)

[timur-ND]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

other (please specify)

Erlang version used

27.3.x

Operating system (distribution) used

CentOS 9 (Stream)

How is RabbitMQ deployed?

RPM package

rabbitmq-diagnostics status output

rabbitmq-diagnostics status | sed 's/rabbit@[a-zA-Z0-9._-]*/rabbit@node-REDACTED/g'
Status of node rabbit@node-REDACTED ...
Runtime

OS PID: 2435326
OS: Linux
Uptime (seconds): 91958
Is under maintenance?: false
RabbitMQ version: 4.2.6
RabbitMQ release series support status: see https://www.rabbitmq.com/release-information
Node name: rabbit@node-REDACTED
Erlang configuration: Erlang/OTP 27 [erts-15.2.7.8] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 3.5.5 27 Jan 2026
Erlang processes: 8918 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

• rabbitmq_prometheus
• rabbitmq_shovel_management
• rabbitmq_federation_management
• rabbitmq_federation
• rabbitmq_auth_mechanism_ssl
• rabbitmq_exchange_federation
• rabbitmq_shovel
• amqp10_client
• prometheus
• gun
• rabbitmq_queue_federation
• rabbitmq_federation_common
• ddskerl
• rabbitmq_management
• rabbitmq_management_agent
• rabbitmq_web_dispatch
• amqp_client
• cowboy
• oauth2_client
• jose

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@node-REDACTED
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@node-REDACTED/quorum/rabbit@node-REDACTED

Config files

• /etc/rabbitmq/rabbitmq.config

Log file(s)

• /var/log/rabbitmq/rabbit@node-REDACTED

Alarms

(none)

Tags

(none)

Memory

Total memory used: 0.5987 gb
Calculation strategy: rss
Memory high watermark setting: 0.8 of available memory, computed to: 13.1754 gb

allocated_unused: 0.1126 gb (18.81 %)
reserved_unallocated: 0.092 gb (15.37 %)
quorum_queue_procs: 0.0472 gb (7.88 %)
plugins: 0.0458 gb (7.64 %)
mgmt_db: 0.0408 gb (6.82 %)
binary: 0.0389 gb (6.5 %)
other_proc: 0.033 gb (5.51 %)
other_system: 0.0324 gb (5.41 %)
code: 0.0299 gb (4.99 %)
connection_other: 0.0297 gb (4.96 %)
connection_channels: 0.021 gb (3.51 %)
connection_readers: 0.0187 gb (3.12 %)
quorum_ets: 0.0146 gb (2.44 %)
metadata_store: 0.009 gb (1.5 %)
other_ets: 0.0079 gb (1.32 %)
metrics: 0.0059 gb (0.99 %)
connection_writers: 0.0048 gb (0.8 %)
queue_procs: 0.0047 gb (0.78 %)
msg_index: 0.0041 gb (0.68 %)
metadata_store_ets: 0.0038 gb (0.63 %)
atom: 0.0015 gb (0.25 %)
quorum_queue_dlx_procs: 0.0003 gb (0.06 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
mnesia: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 0, limit: 16287

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 87.0434 gb

Totals

Connection count: 402
Queue count: 918
Virtual host count: 24

Listeners

Interface: [::], port: 15671, protocol: https, purpose: HTTP API over TLS (HTTPS)
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS

Logs from node 1 (with sensitive values edited out)

RabbitMQ node logs at default log level contain no entries about queue deletion. The deletion was only detectable via rabbitmq_event_exchange plugin via the amq.rabbitmq.event exchange (queue.deleted, user_who_performed_action: rmq-internal). But if logs are still needed, I can attach obfuscated logs using this tool (https://github.com/michaelklishin/rabbitmq-lqt)

Logs from node 2 (if applicable, with sensitive values edited out)

---

Logs from node 3 (if applicable, with sensitive values edited out)

---

rabbitmq.conf

[
{rabbit, [
{loopback_users, [<<"guest">>]},
{cluster_nodes, {['rabbit@node-1, 'rabbit@node-2, 'rabbit@node-3'], disc}},
{cluster_partition_handling, pause_minority},
{tcp_listen_options, [
{backlog, 128},
{nodelay, true},
{linger, {true, 0}},
{exit_on_close, false}
]},
{ssl_listeners, [5671]},
{ssl_options, [
{cacertfile,"/etc/ssl/certs/rabbit_ca.pem"},
{certfile,"/etc/ssl/certs/rabbit.pem"},
{keyfile,"/etc/ssl/certs/rabbit.key"},
{secure_renegotiate,true},
{reuse_sessions,true},
{honor_cipher_order,true},
{verify,verify_peer},
{fail_if_no_peer_cert,true}
]},
{auth_mechanisms, ['PLAIN', 'AMQPLAIN', 'EXTERNAL']},
{quorum_cluster_size, 5},
{quorum_commands_soft_limit, 64},
{ssl_cert_login_from, common_name},
{vm_memory_high_watermark, 0.8},
{vm_memory_high_watermark_paging_ratio, 0.6},
{default_user, <<"guest">>},
{default_pass, <<"guest">>}
]},
{kernel, [
]}
,
{rabbitmq_management, [
{listener, [
{port, 15671},
{ssl, true},
{ssl_opts, [
{cacertfile, "/etc/ssl/certs/rabbit_ca.pem"},
{certfile, "/etc/ssl/certs/rabbit.pem"},
{keyfile, "/etc/ssl/certs/rabbit.key"},
{secure_renegotiate,true},
{reuse_sessions,true},
{honor_cipher_order,true},
{verify,verify_none},
{fail_if_no_peer_cert,false}
]}
]}
]}
].

Steps to deploy RabbitMQ cluster

3-node RabbitMQ cluster installed via RPM packages on CentOS 9 (Stream). Nodes are VM hosts, peer discovery configured via classic config (listing node names explicitly in rabbitmq.conf). Cluster formed with rabbitmqctl join_cluster (same erlang-cookie)

Steps to reproduce the behavior in question

1. Declare a non-durable classic queue with x-expires=300000 and auto_delete=false (via k8s pods during startup)
2. Register a consumer on it (consumer stays connected, heartbeats active)
3. Queue receives occasional messages (low-traffic RPC pattern - messages arrive infrequently but the consumer is continuously registered and connected)
4. After ~5m40s (from consumers start) the queue is deleted by rmq-internal despite the active consumer

Note: the queue was NOT idle in the strict sense - consumers were visible in the management UI and messages were being processed. However, the queue showed "Idle" state in the UI between message deliveries. It appears x-expires may be firing based on "no recent message activity" rather than "no consumers", which contradicts the documented behavior: "Unused means the queue has no consumers, the queue has not been recently redeclared (redeclaring renews the lease), and basic.get has not been invoked for a duration of at least the expiration period."

advanced.config

[
{rabbit, [{quorum_cluster_size, 5},
{quorum_commands_soft_limit, 64}]}
]

Application code

Python / nameko framework / kombu ConsumerMixin

Queue declared with these parameters:

queue = Queue(
"rpc-my-service",
exchange=rpc_exchange,
routing_key="my-service.*",
auto_delete=False,
exclusive=False,
durable=False,
queue_arguments={
"x-expires": 300000, # 5 minutes
},
)

Consumer registered via kombu ConsumerMixin (eventlet green threads)

Kubernetes deployment file

---

What problem are you trying to solve?

After upgrading from RabbitMQ 3.13.2 to 4.2.6, non-durable queues with x-expires=300000 (5 min) are being auto-deleted by RabbitMQ itself (not all, that is strange) - even though active consumers are registered and visible in the management UI. This causes UnknownService errors in our application.

We confirmed via the amq.rabbitmq.event exchange that the deletion is performed by rmq-internal (x-expires expiry?), not by any client:

{
    "exchange": "amq.rabbitmq.event",
    "message_count": 14,
    "payload": "",
    "payload_bytes": 0,
    "payload_encoding": "string",
    "properties": {
      "delivery_mode": 2,
      "headers": {
        "name": "v3.rpc-publish",
        "timestamp_in_ms": 1777904684268,
        "type": "rabbit_classic_queue",
        "user_who_performed_action": "rmq-internal",
        "vhost": "waap"
      },
      "timestamp": 1777904684
    },
    "redelivered": false,
    "routing_key": "queue.deleted"
  },

The queue configuration itself:

{
  "arguments": {
    "x-expires": 300000
  },
  "auto_delete": false,
  "durable": false,
  "name": "v3.rpc-publish",
  "type": "classic",
  "vhost": "waap"
}

The strangest part: we have 17 identically declared queues (same vhost, same code path, same durable=false, x-expires=300000, auto_delete=false). Only 5 of them are deleted. The affected queues are low-traffic RPC queues that show as Idle in the management UI, while the other 12 (with higher message throughput) remain alive. However, this low-traffic is about 1 message per minute, not 0 messages per 5 minutes.

In RabbitMQ 3.13.2 the same setup seems worked fine because we had an ha-all mirroring policy applied. After upgrading to 4.x (which dropped mirroring), the deletions started. I checked https://stackoverflow.com/questions/67425780/rabbitmq-queue-expires-while-in-use but seems not our case.

Some screenshots (can't provide how it looked like during deletion, but consumers were active in UI and there were some activity):

After ~5 minutes:

Queue metric (custom metric) is missing because queue is being deleted constantly every 5 minutes (app doesn't re-declare it automatically, so pods restart were performed):

Questions:

1. Is it expected behavior in RabbitMQ /CQv2 that x-expires fires on a non-durable queue while a consumer is connected but the queue is Idle (no message activity)?
2. What are all the scenarios where RabbitMQ itself (rmq-internal) can delete a queue? I'm aware of x-expires and auto_delete but are there others (e.g. CQv2 internal GC, memory pressure)?
3. Could a half-open TCP connection explain this behavior? The management UI showed active consumers, but if the underlying TCP connection was stale (no heartbeat traffic reaching the broker despite the client believing it is connected), would RabbitMQ consider the queue as having no consumers and start the x-expires timer? If so, how can this be confirmed?
4. Is the "Idle" state in the management UI a reliable indicator of how RabbitMQ evaluates the x-expires timer i.e., does "idle" mean "no messages flowing" rather than "no consumers"?

[michaelklishin]

If this is the case, then it's surprising that we are not flooded with such reports.
Non-durable queues can be deleted in other scenarios, so my guess is that the queue is deleted not by the TTL but something else, including disconnection of the node hosting said non-replicated queue.

Non-durable non-exclusive a deprecated combination that's rejected by default in RabbitMQ 4.3.0.

Switch to durable queues (they can have TTL and message TTL) or an exclusive queues.

[timur-ND]

Thank you for the explanation.

Follow-up questions

In our case, the management UI was showing active consumers on the queue right up until the moment of deletion. If the queue was deleted due to the node disconnecting, is it expected that the UI shows healthy consumers while the underlying node hosting the non-replicated non-durable queue is already in a disconnected/problematic state? It feels like edge case where the broker state and the management view diverge. The queue is gone, but consumer connections (possible re-established to a different node or cached in the UI) still appear active. Is there a way to distinguish deletion caused by x-expires vs deletion caused by node disconnection?

[michaelklishin]

Management stats DB is updated once every 5 seconds by default.