Allow configuring per-mount-point per-queue-type disk alarms by the-mikedavis · Pull Request #14815 · rabbitmq/rabbitmq-server

the-mikedavis · 2025-10-24T19:23:20Z

This is an extension of the free disk space alarm which allows configuring additional mount points to monitor and which queue type(s) to block when they are near full. For example with a config like so:

stream.data_dir = /mnt/data/streams
# Directory where the file system is mounted.
disk_free_limits.stream.mount_point = /mnt/data/streams
# Alarm threshold: if free space falls under this absolute
# limit then an alarm fires per queue type.
disk_free_limits.stream.absolute = 2GB
# Queue types to block when the threshold is breached.
disk_free_limits.stream.queue_types = stream

Publishers to streams would be blocked once the free space of /data/stream-data falls under 2GB. Publishers to classic or quorum queues could continue though.

The motivation of this feature is that you may want to use separate disks for different queue types. For example for higher throughput you may want to use volume(s) with better throughput and/or IOPS for streaming but use standard disks for queue data. Also, alarms are currently fairly aggressive by blocking all publishing. Ideally you should be able to continue using queues when the space you have allocated for streams fills up, or vice versa.

This is a different approach than #14086. Instead of measuring disk usage under a directory like du(1), rabbit_disk_monitor is updated to measure free space of all mounts at once with disksup:get_disk_info/0. Under the hood this performs the same df(1) check as rabbit_disk_monitor had been doing previously - measuring mount-point free space is much cheaper than measuring directory disk footprint. Monitoring mount points is also quite flexible: you can use multiple disks on one mount point with RAID-0 striping or split up a single disk with partitions.

This is a draft - it needs tests and currently only AMQP 0-9-1 is updated to perform selecting blocking. All other protocols currently block for any alarm.

Some of the commits in this branch are refactors that could be cherry-picked out. #14814 is pretty trivial and the refactors to use maps instead of dict in rabbit_alarm and use disksup instead of the custom df code in rabbit_disk_alarm are not strictly related to the feature here.

Discussed in #14590

the-mikedavis · 2025-10-24T19:52:14Z

deps/amqp_client/src/amqp_gen_connection.erl

+        false ->
+            {noreply, State1}
+    end;
+handle_cast({channel_published_to_queue_type, _ChPid, QT},


This feature might need a feature flag. Here for direct connections if old client code is used on a newer server then it would error after publishing since it isn't expecting this cast. I think it would be unlikely to happen in practice but the mixed-version test suite will probably run into this.

samuelmasse · 2025-10-24T20:13:40Z

What would the config setup be for having a main disk that contains quorum and classic queues and a secondary disk that contains streams. Would we specify the same mount point for quorum and classic with each defining queue_types as quorum and classic respectively? Would that result in a common alarm for both or two alarms looking at the same thing.

the-mikedavis · 2025-10-24T20:57:18Z

Ah yeah, in that scenario you could have a config like so:

disk_free_limits.streaming.mount_point = /mnt/data/streams
disk_free_limits.streaming.absolute = 2GB
disk_free_limits.streaming.queue_types = stream

disk_free_limits.messaging.mount_point = /mnt/data/queues
disk_free_limits.messaging.absolute = 2GB
disk_free_limits.messaging.queue_types = classic,quorum

And if /mnt/data/queues went under its configured limit it would set two alarms (disk for classic and disk for quorum queue types) but wouldn't affect streams.

samuelmasse · 2025-10-24T22:38:34Z

Ah I see thanks! So if I understand correctly the name here disk_free_limits.[name].mount_point can be any name we want to set it to, it wouldn't have to be the name of a queue type. So I could set disk_free_limits.bob.mount_point for example.

Taking that thought further, what would the process of adding that "bob" disk alarm to an existing broker look like. If node A thinks the "bob" alarm exists but node B doesn't can there be issues that comes from the disagreement? When node A restarts with the new configuration do all nodes now know of the "bob" alarm or just node A until all other nodes also restart.

Also after I added my new "bob" disk alarm, what ways do I have as a user to then monitor the "bob" alarm to see if it's currently alarming, what value it is configured to and how close it's getting to the alarm point. For MQ's use case currently we are getting this information from the /api/nodes endpoint using the disk_free_limit, disk_free and disk_free_alarm values. Are we thinking about adding an alarm name map to the output of this API to give those values for each disk alarm. So I would maybe access disk_free_map.bob.disk_free to know the status of my "bob" disk alarm.

Lastly for the RabbitMQ console we currently have a column named "Disk space" that displays the information of the as of now only disk alarm. When I add this "bob" disk alarm would we want to dynamically add a new column to that table named something like "Disk space (bob)". In that case would we also support defining the ordering of those columns. For example if we consider that it's most relevant to display the disk alarm of quorum queues on the left, then streams, then classic and at the end the disk alarm for non queue storage, would we be able to define that order manually in some console config.

the-mikedavis · 2025-10-27T14:50:45Z

The [name] part is only used to name the disk. Different members of the cluster could use different names if they wanted: alarms are set by queue-type rather than by disk name. So if you added a node to a cluster and it set an alarm, the other nodes would know to block those queue types regardless of the name of the disk.

Yeah before this change is finished we should really expose the configured disks in the API (and maybe prometheus as well?) and the UI. For the ordering part maybe we can use the order they are listed in the config file? There's an ETS table holding the info of the last-known available bytes and configured limit which can be queried cheaply for the metrics. And we should extend the CLI commands so that the limits can be updated dynamically.

This is the same kind of toggle as the Raft data directory and stream data directory but for controlling classic queue data location.

This is not a functional change, just a refactor to eliminate dicts and use maps instead. This cleans up some helper functions like dict_append/3, and we can use map comprehensions in some places to avoid intermediary lists.

Previously we set `start_disksup` to `false` to avoid OTP's automatic monitoring of disk space. `disksup`'s gen_server starts a port (which runs `df` on Unix) which measures disk usage and sets an alarm through OTP's `alarm_handler` when usage exceeds the configured `disk_almost_full_threshold`. We can set this threshold to 1.0 to effectively turn off disksup's monitoring (i.e. the alarm will never be set). By enabling disksup we have access to `get_disk_data/0` and `get_disk_info/0,1` which can be used to replace the copied versions in `rabbit_disk_monitor`.

`disksup` now exposes the calculation for available disk space for a given path using the same `df` mechanism on Unix. We can use this directly and drop the custom code which reimplements that.

Co-authored-by: Sunny Katkuri <skatkur@amazon.com>

This introduces a new variant of `rabbit_alarm:resource_alarm_source()`: `{disk, QueueType}` which triggers when the configured mount for queue type(s) fall under their limit of available space.

This covers both network and direct connections for 0-9-1. We store a set of the queue types which have been published into on both a channel and connection level since blocking is done on the connection level but only the channel knows what queue types have been published. Then when the published queue types or the set of alarms changes, the connection evaluates whether it is affected by the alarm. If not it may publish but once a channel publishes to an alarmed queue type the connection then blocks until the channel exits or the alarm clears.

This adds two gauge metrics which are emitted per configured mount, one for available bytes and the other for the low watermark. The label `"disk=<name>"` is attached to both gauges to distinguish which mount the gauge applies to.

This adds the configured mounts, if there are any, to the API JSON response for the `/api/nodes` and `/api/node/<node>` endpoints and to the overview and node-detail UI pages. Time series data is not collected for these metrics - that should be scraped from the Prometheus endpoint instead.

The polling interval (min, max and fast-rate) should be tuned for use on different hardware. For example high-end machines with strong network bandwidth should be tuning the fast-rate higher so that disk space is checked more often, as with stronger resources the disk space could fill up more rapidly than the default 250MB/sec predicts.

With this change you can say: rabbitmqctl set_disk_limit mount Streaming 2GiB This applies the limit only to the "Streaming" mount.

This includes regular MQTT and MQTT-over-WebSockets.

When disk information is unavailable, the 'available' field in mount records is set to 'NaN'. Due to Erlang term ordering, 'NaN' < Limit evaluates to true, which triggers alarms for those mounts. This is correct fail-safe behavior - when we cannot determine available disk space, we block publishing to prevent potential disk exhaustion. Add comments to alarmed_mounts/1 and alarmed_queue_types/1 explaining this intentional behavior.

the-mikedavis · 2026-02-19T22:23:10Z

Looks like we'll have to hold off on using disksup in rabbit_disk_monitor as it currently doesn't handle Unicode correctly erlang/otp#10721. unicode_SUITE fails because of that change.

the-mikedavis requested review from SimonUnge and lukebakken October 24, 2025 19:29

the-mikedavis self-assigned this Oct 24, 2025

the-mikedavis commented Oct 24, 2025

View reviewed changes

the-mikedavis force-pushed the mount-point-limits branch from c96fb3a to 44016e6 Compare October 31, 2025 19:25

the-mikedavis force-pushed the mount-point-limits branch 2 times, most recently from cc9a3f4 to d8b19d3 Compare November 11, 2025 19:25

the-mikedavis force-pushed the mount-point-limits branch from d8b19d3 to cbc49e5 Compare December 23, 2025 23:02

the-mikedavis force-pushed the mount-point-limits branch 3 times, most recently from 3d9dbb5 to f47572d Compare February 5, 2026 18:36

the-mikedavis force-pushed the mount-point-limits branch from 48f8492 to 6678cce Compare February 17, 2026 16:15

the-mikedavis and others added 14 commits February 19, 2026 17:21

Allow configuring classic queue data dir in Cuttlefish config

a4963c4

This is the same kind of toggle as the Raft data directory and stream data directory but for controlling classic queue data location.

rabbit_alarm: Prefer maps to dicts

f995799

This is not a functional change, just a refactor to eliminate dicts and use maps instead. This cleans up some helper functions like dict_append/3, and we can use map comprehensions in some places to avoid intermediary lists.

rabbit_disk_monitor: Use disksup to determine available bytes

6df3f54

`disksup` now exposes the calculation for available disk space for a given path using the same `df` mechanism on Unix. We can use this directly and drop the custom code which reimplements that.

rabbit.schema: Add config options for per-queue-type disk limits

85ca18c

rabbit_disk_monitor: Monitor per-queue-type mounts

9065c11

rabbit_alarm: Add a helper to format resource alarm sources

b305992

Co-authored-by: Sunny Katkuri <skatkur@amazon.com>

Set per-queue-type disk alarms for configured mounts

a111204

This introduces a new variant of `rabbit_alarm:resource_alarm_source()`: `{disk, QueueType}` which triggers when the configured mount for queue type(s) fall under their limit of available space.

CLI: Extend set_disk_free_limit to set mount limits

8eab077

With this change you can say: rabbitmqctl set_disk_limit mount Streaming 2GiB This applies the limit only to the "Streaming" mount.

rabbit_stream_reader: Block during stream queue-type disk alarm

d509301

the-mikedavis and others added 3 commits February 19, 2026 17:21

MQTT: Handle per-queue-type disk alarms

d63375f

This includes regular MQTT and MQTT-over-WebSockets.

AMQP 1.0: Handle per-queue-type disk alarms

2c0ae89

the-mikedavis force-pushed the mount-point-limits branch from 7926d1c to f534680 Compare February 19, 2026 22:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow configuring per-mount-point per-queue-type disk alarms#14815

Allow configuring per-mount-point per-queue-type disk alarms#14815
the-mikedavis wants to merge 17 commits intorabbitmq:mainfrom
amazon-mq:mount-point-limits

the-mikedavis commented Oct 24, 2025

Uh oh!

the-mikedavis Oct 24, 2025

Uh oh!

samuelmasse commented Oct 24, 2025

Uh oh!

the-mikedavis commented Oct 24, 2025 •

edited

Loading

Uh oh!

samuelmasse commented Oct 24, 2025

Uh oh!

the-mikedavis commented Oct 27, 2025

Uh oh!

the-mikedavis commented Feb 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

the-mikedavis commented Oct 24, 2025

Uh oh!

the-mikedavis Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

samuelmasse commented Oct 24, 2025

Uh oh!

the-mikedavis commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samuelmasse commented Oct 24, 2025

Uh oh!

the-mikedavis commented Oct 27, 2025

Uh oh!

the-mikedavis commented Feb 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

the-mikedavis commented Oct 24, 2025 •

edited

Loading