[Question] Quorum queues left without a leader (noproc, unresposive state) after high churn with concurrent deletions and re-declarations in RabbitMQ 4.x
#16758
-
Community Support Policy
RabbitMQ version usedother (please specify) Erlang version used27.3.x Operating system (distribution) usedKubernetes-hosted RabbitMQ clusters running Linux containers. How is RabbitMQ deployed?Bitnami Helm chart rabbitmq-diagnostics status outputSee https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics DetailsLogs from node 1 (with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs DetailsWhile there are also a bunch of other errors it feels like they were likely the result of queues in the process of being deleted. Ultimately, they were cleaned up by the applied policy and our application was able to redeclare the queues. Logs from node 2 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs DetailsLogs from node 3 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Detailsrabbitmq.confSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location DetailsSteps to deploy RabbitMQ clusterWe have a standardized helm chart that was based on the Bitnami Helm Chart and then modified to work with our OCI environments. It is mostly a vanilla rabbit deployment. Steps to reproduce the behavior in questionWe are in the process of upgrading from RabbitMQ 3.13.7 to 4.2.2. During testing in one of our non-production environments, we observed an issue where numerous quorum queues were left in a bad state. The affected queues appeared as Unfortunately, we do not have logs or metrics from those original events, so we could only infer the likely trigger. Our working theory was that the combination of a large queue count, roughly 30,000 total queues with about 50% quorum queues, and batched service cycling caused a high amount of connection churn and potentially queue churn as new service instances came online while older instances were terminated. To investigate, we created an intentionally severe reproduction test that:
On RabbitMQ 3.13.7, we only observed transient queue declaration failures, and the queues ultimately recovered. On the RabbitMQ 4.x versions, we were consistently able to leave several quorum queues in a permanently bad state that required manual deletion. advanced.configSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location DetailsApplication codeWe use the java library: com.rabbitmq:amqp-client:5.26.0. I'm not sure what level of code is of interest but this represents an outline of the overall approach. We are creating a lot of queues concurrently and as a result when the policy expiration hits all of the queues will qualify at more or less the same time. Detailstest1.start();
declare(queue1001, durable=true, exclusive=false, autoDelete=false, xQueueType="quorum");
declare(queue2000, durable=true, exclusive=false, autoDelete=false, xQueueType="quorum");
bindAndConsume(queue1001);
bindAndConsume(queue2000);
test2.start();
declareSameQueueNamesAgain();
bindAndConsumeSameQueueNamesAgain();
test1.end()
basicCancelConsumers();
leaveSharedDurableQueuesInRabbit();
disconnectClient();
test2.end();
//Server side policy cleanup begins deleting queues
test3.start();
declareSameQueueNamesAgain();
basicConsume(queue1001); // observed Rabbit internal_error/noproc
basicConsume(queue2000); // observed Rabbit internal_error/noprocKubernetes deployment fileDetailsapiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
annotations:
meta.helm.sh/release-name: rabbitmq-release
meta.helm.sh/release-namespace: rabbitmq-namespace
finalizers:
- deletion.finalizers.rabbitmqclusters.rabbitmq.com
generation: 2
labels:
app.kubernetes.io/managed-by: Helm
example.com/external-internet: "true"
example.com/external-consume-apiserver: "true"
example.com/external-global-whitelist: "true"
example.com/stack: test-env
name: rabbitmq-cluster
namespace: rabbitmq-namespace
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-type
operator: In
values:
- rabbitmq
delayStartSeconds: 30
image: registry.example.com/example/rabbitmq:4.3.2-management-alpine
imagePullSecrets:
- name: registry-credentials
override:
service:
spec:
ports:
- appProtocol: amqp
name: amqp
port: 5672
protocol: TCP
targetPort: 5672
- appProtocol: amqps
name: amqps
port: 5671
protocol: TCP
targetPort: 5671
- appProtocol: http
name: management
port: 15672
protocol: TCP
targetPort: 15672
- appProtocol: prometheus.io/metrics
name: prometheus
port: 15692
protocol: TCP
targetPort: 15692
statefulSet:
metadata:
labels:
example.com/external-internet: "true"
example.com/external-consume-apiserver: "true"
example.com/external-global-whitelist: "true"
example.com/stack: test-env
spec:
template:
metadata:
labels:
example.com/external-internet: "true"
example.com/external-consume-apiserver: "true"
example.com/external-global-whitelist: "true"
example.com/stack: test-env
spec:
containers:
- lifecycle:
postStart:
exec:
command:
- bash
- -c
- cd /opt/rabbitmq/ && ./poststart-monitor.sh
name: rabbitmq
resources: {}
volumeMounts:
- mountPath: /opt/rabbitmq/definitions.json
name: rabbitmq-definitions
subPath: definitions.json
- mountPath: /opt/rabbitmq/poststart-monitor.sh
name: rabbitmq-poststart-configmap
subPath: poststart-monitor.sh
- mountPath: /opt/rabbitmq/ssl-certs
name: tls-secret-volume
- image: registry.example.com/example/oci-management-agent:1.12.0
lifecycle:
postStart:
exec:
command:
- bash
- -c
- cd /opt/oracle/ && ./poststart-monitor.sh
preStop:
exec:
command:
- bash
- -c
- pkill -u 1000
name: oracle-mgmt-agent
resources: {}
securityContext:
allowPrivilegeEscalation: true
readOnlyRootFilesystem: false
runAsGroup: 2000
runAsUser: 1000
volumeMounts:
- mountPath: /opt/oracle/
name: oracle-mgmt-agent-data
- mountPath: /opt/oracle/mgmtagent_secret/input.rsp
name: mgmt-agent-install-config
subPath: config.rsp
- mountPath: /opt/oracle/metrics_configuration/rabbitmq-metrics.properties
name: mgmt-agent-metrics-config
subPath: rabbitmq-metrics.properties
- mountPath: /opt/oracle/metrics_configuration/rabbitmq-metrics-detailed.properties
name: mgmt-agent-metrics-detailed-config
subPath: rabbitmq-metrics-detailed.properties
- mountPath: /opt/oracle/poststart-monitor.sh
name: mgmt-agent-poststart-configmap
subPath: poststart-monitor.sh
volumes:
- name: tls-secret-volume
secret:
secretName: tls-secret
- configMap:
defaultMode: 511
name: rabbitmq-poststart-configmap
name: rabbitmq-poststart-configmap
- configMap:
name: definitions-configmap
name: rabbitmq-definitions
- configMap:
name: metrics-configmap
name: mgmt-agent-metrics-config
- configMap:
name: metrics-configmap
name: mgmt-agent-metrics-detailed-config
- configMap:
name: mgmt-agent-install-configmap
name: mgmt-agent-install-config
- configMap:
defaultMode: 511
name: mgmt-agent-poststart-configmap
name: mgmt-agent-poststart-configmap
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app.kubernetes.io/component: rabbitmq
app.kubernetes.io/name: rabbitmq-cluster
app.kubernetes.io/part-of: rabbitmq
name: persistence
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Gi
storageClassName: block-storage-class
volumeMode: Filesystem
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app.kubernetes.io/component: rabbitmq
app.kubernetes.io/name: rabbitmq-cluster
app.kubernetes.io/part-of: rabbitmq
name: oracle-mgmt-agent-data
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: block-storage-class
volumeMode: Filesystem
persistence:
storage: 500Gi
storageClassName: block-storage-class
rabbitmq:
additionalConfig: |
collect_statistics_interval = 30000
vm_memory_high_watermark.relative = 0.6
cluster_name=<redacted-cluster-name>
listeners.ssl.default=5671
ssl_options.cacertfile=/opt/rabbitmq/ssl-certs/ca.crt
ssl_options.certfile=/opt/rabbitmq/ssl-certs/tls.crt
ssl_options.keyfile=/opt/rabbitmq/ssl-certs/tls.key
ssl_options.verify=verify_none
ssl_options.fail_if_no_peer_cert=false
replicas: 3
resources:
limits:
cpu: "8"
memory: 32Gi
secretBackend:
externalSecret: {}
service:
type: ClusterIP
terminationGracePeriodSeconds: 604800
tls: {}
tolerations:
- effect: NoSchedule
key: node-type
operator: Equal
value: rabbitmqWhat problem are you trying to solve?We are in the process of upgrading from RabbitMQ 3.13.7 to 4.2.2. During testing in one of our non-production environments, we observed an issue where numerous quorum queues were left in a bad state. The affected queues appeared as Unfortunately, we do not have logs or metrics from those original events, so we could only infer the likely trigger. Our working theory was that the combination of a large queue count, roughly 30,000 total queues with about 50% quorum queues, and batched service cycling caused a high amount of connection churn and potentially queue churn as new service instances came online while older instances were terminated. To investigate, we created an intentionally severe reproduction test and attempted to reproduce the behavior seen. On RabbitMQ 3.13.7, we only observed transient queue declaration failures, and the queues ultimately recovered. On the RabbitMQ 4.x versions, we were consistently able to leave several quorum queues in a permanently bad state that required manual deletion. Ex. 
In RabbitMQ 4.3.2, we also cannot run We understand that this test case is severe. However, we are concerned because we cannot reproduce this behavior on our current RabbitMQ 3.13.7 version, and we believe we have already seen a variant of this corruption in our largest non-production RabbitMQ cluster. We're hoping to get some insight into why this is occurring and any potential fixes. All of the logs and details provided are specific to 4.3.2 since it is the latest and also had some fixes that sounded like they may be relevant to this sort of failure case. |
Beta Was this translation helpful? Give feedback.
Replies: 7 comments 17 replies
-
@cwlowe2007 if you don't have any logs or metrics, we cannot help beyond a list of relevant discussions and changes in recent months. We do not guess in this community. Quorum queues are not supposed to be churned and some of the above issues are directly related to churn.
RabbitMQ 4.2 will happily trigger elections of all QQs at once e.g. upon a node restart, even for thousands of quorum queues, while 4.3 does that in batches. |
Beta Was this translation helpful? Give feedback.
-
|
For the 4.3.2 cluster I don't see a way to safely cycle any nodes without just removing the queue. I can't safely stop the running node I get this error if I try to add new members to the queue: I am also unable to get the current status of the queue: Interestingly, the queue does show in the admin console and says it is running with two online members. |
Beta Was this translation helpful? Give feedback.
-
|
@cwlowe2007 your descriptions are still very superficial, what do you expect the core team to do with "I can reproduce" without a specific steps to reproduce? We don't know what you are doing with a We do not guess in this community and you are not a paying customer or a regular contributor, so please help us help you. |
Beta Was this translation helpful? Give feedback.
-
|
If you can reproduce with then please do that and share the logs from all nodes. Quorum queues were not designed for this kind of churn but we'd at least be able to see what's going on using more data than a few sentences. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I wasn't intentionally being vague. With those steps I have been able to reproduce this issue pretty consistently. I specifically tested on 4.2.2, 4.2.8, and 4.3.2. I should have all the logs for 4.3.2 and I will work on sanitizing and attaching them. If there is any debug logging or metrics worth gathering let me know and I can rerun the scenario against any of those versions to gather that. |
Beta Was this translation helpful? Give feedback.
-
|
Debug logging can increase log files substantially for this kind of tests but See how to enabled debug logging. Most likely even regular logs would give us enough. |
Beta Was this translation helpful? Give feedback.
-
Hypothesis One: Khepri Deletion Times OutEnvironment Comparison
HypothesisThose The expiry-driven delete code path stops the Ra cluster first and deletes the Khepri record. When thousands of QQs expire at once and the Khepri (coordination) cluster is saturated, that second step can time out. A timeout leaves the queue record in place forever, even though the QQ replaces were 3.13 uses Mnesia, where a delete is a cheaper local transaction vs. a cluster-wide Raft Ra 2.x and 3.x evolution optimizes many things that are not "churn resistance" and even when The metadata store does not have a "sweeper" fail safe that would clean up such deleted queues. Some potentially relevant changes #14672 #14902. Much less likely to be related but still: #15554 #15025. @cwlowe2007 so those "bad queues" are not actually queues, they are leftover metadata store keys that are a result of your test combining peak short term load with queue TTL. Quorum queues that have a TTL is a curious combination (so you replicate your data and willing to throw it away at the same time?). Perhaps try the test without the TTL on the quorum queues. |
Beta Was this translation helpful? Give feedback.
-
|
Full logs. Timeline 06-19:
Queues For the issue we saw in non-prod both times it was when we were batch cycling all of our services. Given the number of queues we had in that one cluster I wouldn't be surprised if ultimately it was undersized for the number of queues it was managing. In that we cluster we ended up with 100s of queues that were in this same state where they weren't deleted but also couldn't be used. Ultimately, we split that load across two new clusters and haven't seen a repeat of the issue. However, this was not something we had seen on that same cluster using 3.13.7 so that led us down the path of creating this high churn scenario to test the behavior. Regarding the TTL that is mostly a failsafe to clean up left over queues if a service is shutdown long term. I only set it this low to try and cause as much churn as possible to reproduce the end state we saw in our non-prod. |
Beta Was this translation helpful? Give feedback.
-
|
The logs do not seem to confirm this hypothesis: there are no Khepri operation timeout traces. |
Beta Was this translation helpful? Give feedback.
-
|
To anyone reading the logs: |
Beta Was this translation helpful? Give feedback.
-
|
See Hypothesis Two below. |
Beta Was this translation helpful? Give feedback.
-
|
To anyone reading the logs: In the logs, 27,757 queues log that they are terminating with a reason set to However, queues Obfuscated logs do not share the number of schedulers (CPU cores), so it can be a system completely deprived of resources or something else. In theory it's possible that the node has 30 GiB of RAM but only a couple of cores, wouldn't be the first time people assume that "any number of cores should be fine", which is completely wrong. @cwlowe2007 I do not know what "cycling your services" really means. My conclusion is that your workload leads to concurrent declaration and deletion of enough QQs that some observe cascading failures they cannot recover from even before even attempting a leader election. Like those 2 queues out of close to 28K that did terminate as expected after being deleted (by TTL or explicitly). My best guess is that this is a form of #15615 (originally submitted as #14241). Which took months to dig out and find a reasonable solution for. Such extreme churn workloads is the opposite of what quorum queues are designed for. |
Beta Was this translation helpful? Give feedback.
-
|
Regarding CPU I never saw it over ~60% when running this test. That said, it is possible there were spikes I missed. This cluster is configured to have 8 cores for each node. Thanks for looking through this and providing thoughts. If debug logs, or anything else, has any potential value let me know and I can get that attached. |
Beta Was this translation helpful? Give feedback.
-
|
A large number of inexpensive runtime operations can produce a lot of context switching but not reach per-core utilization of 100%. Think of it as the same issue as memory fragmentation but for CPU time allocated to Erlang processes by the runtime: some free resources are there, they are just not used perfectly efficiently. |
Beta Was this translation helpful? Give feedback.
-
Hypothesis TwoI have dug up enough details to form another hypothesis, Hypothesis Two. This is a variation of Hypothesis One in the sense that the metadata store no longer The Basics and Key ObservationsA quorum queue is a Ra cluster (a Raft cluster implemented using a library called When a Ra cluster member is removed from the cluster voluntarily stops with exit reason set to Some Ra relevant code paths in Ra only log at A Set of Pre-conditions For a Non-Recovering QQThis is just one scenario, obviously there can be other pre-conditions such as a permanently lost cluster majority. Consider the following conditions together:
So we have evidence of conditions 1 and 4 (per problem definition), 2 is invisible in the logs, Possible ScenarioNow on to afeasible scenario that can lead to the four conditions above. Under high churn the same queue name is deleted and redeclared multiple times due to the interaction Because the Ra cluster name is derived deterministically from the queue name, such Next, due to the churn a membership or stop operation belonging to incarnation A lands on members of incarnation B (after re-declaration) and removes these "new" (redeclared) incarnation B replicas. Meanwhile the churn-produced Raft cluster member discrepancy is not known to Khepri, which only How to Prove This HypothesisRe-run the test using debug logging. That means that Ra will log more than it did in the first round logs. This might include enough evidence to track down those "mixed incarnation" clusters. Can Quorum Queues Be "Fixed" to Handle Such Scenarios?In theory yes, by introducing a member identifier that would grow between restarts, be consistent across all replicas and not depend on the Raft leader term. The per-incarnation member UID already exists in 4.3 but Ra deletes, membership transition ops only use This will take a while to get right for the general case of high churn, so how about we stop pretending that heavy churn of quorum queues is a RabbitMQ bug and consider it to be a bug of the overall system that uses quorum queues |
Beta Was this translation helpful? Give feedback.
-
|
@cwlowe2007 debug logs would help verify this scenario but the chances of our team spending time on this are slim. The entire problem comes down to heavy churn of exactly the queue types were document as not designed for churn. #15615 addresses just one scenario where churn can wreck havoc for a replicated entity. I am not sure our team has the cycles and enough interest to address the rest. The fundamental problem is churn, not our Raft or quorum queue implementation, even though they could be even more defensive than they are today, about 9 years in. |
Beta Was this translation helpful? Give feedback.
-
|
@cwlowe2007 Queue TTLs can be a bit troublesome in this regard when there are lots of other concurrent activities. Do you have to use them? If the queues are re-declared with the same name can you just leave them instead of deleting them? We don't recommend churning quorum queues like this as cluster deletion / creation is quite resource heavy. |
Beta Was this translation helpful? Give feedback.
-
|
I don't think we can get rid of them anytime in the near future. I also believe in a less contrived workflow they shouldn't be as impactful as we won't have that many queues all expiring at once. My testing was simply an effort to replicate the sort of behavior we saw in our non-production environment that led to queues still showing as declared but not being useable. The workflow that we saw leading to that was essentially:
Most of our quorum queues have multiple consumers and as we cycle our services new consumers will be added\removed as instances start\stop. In general I would expect far more connection churn and classic queue churn in that scenario. However, there will still be some amount of quorum queues that end up being declared and deleted during this process. |
Beta Was this translation helpful? Give feedback.
-
|
As long as TTL on quorum queues does not result in moderate or high churn, you should be fine in practice. |
Beta Was this translation helpful? Give feedback.
-
|
queue12000-relevant-log-lines-by-node-sanitized-v2.txt Attached is debug logging around the initial creation, begin of deletion, attempt to redeclare, and the corresponding errors for a queue that was left in this state. The debug logs are massive so I don't think I can really attach them in their entirety, but happy to provide other snippets if there is anything else that can provide value. |
Beta Was this translation helpful? Give feedback.
-
|
This short log confirms both of the above hypotheses, at least partially. The deterministic Ra cluster name lets a delete and a redeclare of the same queue interleave across QQ incarnations caused by the churn:
All caused by extremely aggressive QQ churn. Remove the churn and you will be good to go. |
Beta Was this translation helpful? Give feedback.
-
|
There are certain improvements around QQs that can make this less likely to happen. Some are fairly complex. The detailed sequence of events speaks for itself. This is exactly what quorum queues were NOT built to withstand: churn high enough that not two but three incarnations can co-exist at the same time. A quick look at the Ra also has those but they are not used as widely. |
Beta Was this translation helpful? Give feedback.
-
|
This is not a Ra issue but rather how RabbitMQ re-uses the same naming strategy for the registered process names. Names are atoms and atoms are a limited resource in RabbitMQ, if we provided unique names for each declaration and you delete and redeclare the same queue names often you will eventually run out of atoms and your RabbitMQ nodes will crash, probably in close succession. |
Beta Was this translation helpful? Give feedback.
This short log confirms both of the above hypotheses, at least partially.
The deterministic Ra cluster name lets a delete and a redeclare of the same queue interleave across QQ incarnations caused by the churn:
QUEUE_12000has members on nodes {n0, n1, n2}. TTL-drive removal kicks in, all three members terminating with reason 'delete', removing their data directories{n2, n1, n3}(son0is gone,n3is introduced)n3already runs a Ra member for this name from an earlier incarnation, so the new member fails to start with{already_started, Pid}. As a result, the new QQ clu…