Leaders write 30-200x more to disk than they receive over the network

[viktorerlingsson]

Investigation by Claude (Claude Code), driven by @chadknutson . Numbers below are from our two production clusters plus a controlled 3-node AWS test cluster. This is independent of clustering.

TL;DR

Disk writes are dominated by a term that has nothing to do with message rate or message size:

disk_write  ~  hot_files x folio_size x syncfs_rate   +   message_bytes
               \____________ this dominates ____________/

We append messages by memcpy into MAP_SHARED mmaps, so the unit of a write is a page or folio, never a message. syncfs(2) is filesystem-wide and is paced by publisher confirms. And every queue has an acks.* file that takes 4 bytes per ack into that same page-sized unit. Multiply those together and you get 200x on a cluster doing 0.15 MiB/s.

It is not a leak, not replication, and not fan-out. Nothing is being written twice at the application level - we write each enqueued message exactly once.

What we measured

Cluster A (prod), from Grafana + node tracing:

network in         6.16 MiB/s
disk write          204 MiB/s (2h mean); 76.4 MiB/s in a live 30s sample
disk read              0 B/s          <- rules out resync / re-read entirely
disk usage          flat at 2.4 GiB   <- steady-state churn, not accumulation
queues                    78
syncfs                10.4 /s
confirms       170.6/s of 3177 publishes  (~5% of publishers)
avg write req       72.9 KiB

Cluster B (prod): 167 KiB/s in -> 32.6 MiB/s written. Same fingerprint: zero reads, flat usage.

The three multipliers

1. The write unit is a page/folio, not a message

MessageStore#write_to_disk (src/lavinmq/message_store.cr:333-344) is a memcpy into an mmap. There is no write(2) and no msync anywhere on the publish path (MFile#flush, the only msync, has no callers). When anything flushes, the kernel writes the whole dirty unit. On ext4 that's 4 KiB. On XFS with a modern kernel (cluster A is on 6.17) the page cache uses large folios up to 64 KiB - we measured 57 KiB per dirty event and 72.9 KiB average write request. So a 2 KB message can dirty 64 KiB.

2. syncfs is filesystem-wide and is paced by publisher confirms

Persister#sync (src/lavinmq/persister.cr:47-54) is the leader's only durability barrier, reached only from the publish-confirm drain (Persister#drain_pending_acks) and tx_commit (src/lavinmq/amqp/channel.cr:832). Each call flushes every dirty folio of every queue file on the volume, not just the file being confirmed. publish_confirm_loop re-arms the instant syncfs returns, so it runs as fast as the disk allows.

The uncomfortable part: only ~5% of cluster A's publishes use confirms. That minority sets the flush cadence for the whole filesystem, and every other queue on the node pays for it.

3. The ack files are the pathological case

Each ack appends 4 bytes to a separate mmap'd acks.* file (src/lavinmq/message_store.cr:176-203). A 64 KiB folio holds 16384 acks, so at a few hundred acks/s it takes minutes to fill - meaning it gets rewritten nearly in full on every flush. When we traced dirtied folios by inode on cluster A, acks.* files were all over the top of the list. 78 queues x 64 KiB x 10.4/s ~ 52 MiB/s from ack files alone.

Arithmetic check for cluster A: 78 queues x 64 KiB x 10.4/s ~ 52 MiB/s, measured 76 MiB/s, against ~12 MiB/s of actual enqueued message data.

Controlled reproduction

On a clean 3-node test cluster, 100 queues, 500 B messages, no confirms, sweeping the publish rate 10 -> 1000 msg/s:

msg/s	net MiB/s	disk MiB/s	amplification
10	0.0054	0.321	59x
30	0.0163	0.418	25.6x
100	0.0545	0.532	9.8x
300	0.1633	0.532	3.3x
1000	0.5443	0.960	1.8x

fit: disk_MiB_s = 602 bytes/msg x rate + 397 KiB/s

Note the disk column barely moves across a 100x change in rate. The 602 B/msg term is just the message written once. The 397 KiB/s constant floor is the amplification - ~4 KiB/s per queue, paid whether the page holds 4 bytes or 4 KiB. "Amplification ratio" is that floor divided by a shrinking denominator, which is why our slower cluster looks worse than our faster one.

Then the same run with confirms turned on (-c 1), everything else identical:

	no confirms	with confirms
disk write	0.416 MiB/s*	6.269 MiB/s	<- 15x
amplification	3.3x	235x
bytes written/msg	1.8 KiB	131 KiB

* model prediction at the same 48.8 msg/s the confirm run achieved

Why RabbitMQ sits at ~2x

Quorum queues append to a WAL with buffered write(2) plus periodic fsync. The unit is the byte range, so a partially-filled block is never rewritten. We pay the page/folio every time because we go through mmap.

Measurement note

One gotcha for anyone digging in: our own /api/nodes io_write_count is ru_oublock, which Linux charges per page-dirty transition. It happens to track physical writes well here, but it is not a physical I/O counter and it cannot see journal or metadata traffic.

[viktorerlingsson]

Independent verification on a single node

I reproduced this on a single node (no clustering) with a purpose-built harness
that boots a fresh broker per scenario and samples three counters over the load
window only: /proc/<pid>/io write_bytes (page-dirty transitions),
/proc/diskstats sectors-written for the data-dir device, and du of the data
dir. Environment is ext4 with 4 KiB pages, so absolute magnitudes run ~10x under
the production numbers above. Treat the ratios, not the absolute MiB/s.

The core model holds. Rate sweep, 100 queues, 500 B, no confirms, fitted:

this harness:  585 bytes/msg x rate +  35 KiB/s
issue above:   602 bytes/msg x rate + 397 KiB/s

The per-message slope matches within 3%, which is the main reason I trust the
harness is measuring the same phenomenon the production tracing found.

The ack term, quantified

Isolating it by running the same confirm load with and without consumers:

scenario	dirty MiB/s	phys MiB/s	p-amp	KiB dirtied/msg
confirm-off	0.071	0.137	2.9x	0.7
confirm-on, no consumers	0.443	2.396	50.3x	4.5
confirm-on, consume + ack	0.834	3.994	83.8x	8.5
confirm-on, --no-sync	0.073	0.141	3.0x	0.7

Acks are 47% of dirtied bytes. Against the real ack data being persisted:

410 KiB/s written / 400 B/s of ack data  =  1025x
page_size / 4 bytes per ack              =  1024x   <- predicted

Matching 1024x to three digits leaves no slack in the explanation. --no-sync
collapsing the whole thing to 3.0x confirms the syncfs cadence is the driver
and nothing else. confirm-on replicated across two independent runs to three
significant figures.

The bystander cost is the part I'd emphasise

Queue 1 publishes at 50/s with confirms; queues 2-100 publish at 100/s total
and never use confirms. Identical message count and identical bytes published in
both rows:

scenario	pub MiB/s	dirty MiB/s	phys MiB/s	p-amp
queue 1 does not confirm	0.0715	0.089	0.150	2.1x
queue 1 confirms	0.0715	0.662	2.351	32.9x

One publisher on 1 of 100 queues turning on confirms costs the node 15.7x more
physical writes. The 99 uninvolved queues absorb 100 msg/s x 4 KiB = 0.391 MiB/s,
exactly one full page per bystander message. This is the "only ~5%
of publishes use confirms and everyone pays" observation, measured directly.

Two corrections

1. The floor is not a fixed per-queue cost. The writeup reads it as
"~4 KiB/s per queue, paid whether the page holds 4 bytes or 4 KiB". I measure
~350 B/s per queue, and 16x is exactly the 4 KiB page vs 64 KiB folio ratio. So
the floor is folio_size x flush_rate, which also means there are two regimes:

• syncfs faster than the publish rate (everything in my tests): cost =
  messages x folio_size, giving a flat 4.5 KiB/msg for a 500 B payload.
• syncfs saturated (cluster A at 10.4/s): cost =
  hot_files x folio_size x syncfs_rate, which is the 78 x 64 KiB x 10.4
  arithmetic above.

This matters for fix selection: rate-limiting syncfs helps the first regime a
lot and cluster A not at all. It also explains something that initially looked
like a contradiction in my data - my queue-count sweep was nearly flat (1.4x at
1 queue to 2.0x at 100) because without confirms the 30s flush interval is
longer than the page fill time, so pages fill before being written.

2. The RabbitMQ comparison isn't quite right. "The unit is the byte range,
so a partially-filled block is never rewritten" does not hold at the block
layer: an fsync after appending 4 bytes writes the block, and the next
fsync writes it again. Quorum queues' real advantage is that there is one
WAL rather than two files per queue, so flush cost is O(1) in queue count and
the fsync is not filesystem-wide. Worth getting right, since that sentence is
effectively the design target for a WAL.

One thing the measurement note undersells

ru_oublock and /proc/pid/io are both blind to journal and metadata traffic,
and that blind spot is large: physical writes run 4.8x above page-dirty
accounting (83.8x vs 17.5x). Every syncfs forces an ext4 journal commit, and
because segments are sparse ftruncate (message_store.cr:354) each first
write to a page also allocates a block. Any fix that keeps a per-flush journal
commit inherits this multiplier.

Two more amplifiers, from reading rather than measuring

Neither is exercised by the scenarios above (all 500 B single-frame, no
transactions), so these are code-reading findings only:

• tx_commit bypasses the confirm loop's batching. channel.cr:832 calls
  vhost.sync -> Persister#sync directly, so every transaction commit is an
  unbatched filesystem-wide syncfs, synchronously on the client's fiber.
• Multi-frame messages are written to disk twice. Bodies larger than one
  frame go to a per-channel tempfile in the data dir (channel.cr:808, opened
  sync = true, then unlinked) and then to the segment. The tempfile is rewound
  and reused, so every large message re-dirties the same pages and syncfs
  flushes them. All tx-mode publishes take this path regardless of size.

Where the numbers point

Two independent fixes, each with a measured share, and neither subsumes the
other:

• Buffer ack writes in memory - 47% of dirtied bytes, takes ack data from
  1024x to ~1x. Works even with syncfs left filesystem-wide, since a clean
  page is never written. Ack durability is not required: a lost ack means
  redelivery, and prune_orphaned_acks (message_store.cr:570) exists
  precisely because msgs/acks can already diverge after an unclean shutdown.
• Targeted per-file sync instead of syncfs - the only fix for the
  bystander term. Caveat worth checking before committing to it: N sequential
  fdatasync calls are N journal commits, so a publish fanning out to 78 queues
  could trade 1 syscall and 1 commit for 78 of each. Probably needs a fallback
  to syncfs above some dirty-set size. Note MFile closes its fd after mmap
  (mfile.cr:60), so it would have to keep one.