Add JMH benchmarks comparing read I/O strategies under memory pressure

[neoremind]

Adds JMH benchmarks to compare read I/O strategies in memory constrained scenario, related to #16044.

I/O strategies tested:

• mmap no madvise
• mmap + MADV_NORMAL + MADV_WILLNEED
• mmap + MADV_RANDOM
• mmap + MADV_RANDOM + MADV_WILLNEED
• FFI pread(2) via Panama
• FileChannel ( with NIOFSDirectory)
• O_DIRECT

Thread counts: 1, 4, 8, 16.

How to run

dd if=/dev/urandom of=/path/to/pread-bench-16G.dat bs=1M count=16384

java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \
  -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/path/to/pread-bench-16G.dat -Dbench.fileSizeMB=16384" \
  -p readSize=16384 -p readsPerOp=16

[jimczi]

madvise needs a page-aligned start address, so passing the raw random offset here makes it return EINVAL and do nothing — and since rc is discarded, it fails silently. On a c6id.4xlarge strace shows ~5117/5181 of these calls returning -1 EINVAL, so the prefetch never actually runs and the prefetch rows end up identical to plain mmap. The real Directory avoids this because MemorySegmentIndexInput#advise rounds the start down to the page first.

Suggested fix (mirrors what the Directory does):

Suggested change

	MemorySegment slice = mmapSegmentNormal.asSlice(offsets[i], readSize);
	int rc = (int) POSIX_MADVISE.invokeExact(slice, (long) readSize, MADV_WILLNEED);
	// madvise needs a page-aligned start address, otherwise it returns EINVAL and is a no-op.
	long offsetInPage = (mmapSegmentNormal.address() + offsets[i]) % ALIGNMENT;
	long aoff = offsets[i] - offsetInPage;
	long alen = readSize + offsetInPage;
	MemorySegment slice = mmapSegmentNormal.asSlice(aoff, alen);
	int rc = (int) POSIX_MADVISE.invokeExact(slice, alen, MADV_WILLNEED);
	assert rc == 0 : "posix_madvise failed: " + rc;

Same change is needed in doMmapMadvRandomBatchedPrefetch (against mmapSegmentMadvRandom). With this, single-threaded mmap+prefetch goes from ~0.15 → ~4.2 ops/ms cold (≈7× pread at T01, ~device saturation). Might be worth asserting on rc at the other madvise call sites too so this can't silently regress again.

[mikemccand]

I love that we are building tooling for benchmarking cold/warm cases -- they are so tricky to properly test because at a "whole search system" level you should replay query traffic with accurate actual production arrival times (ideally, or simulate w/ Poisson process) for real measures. I'd love to see Lucene eventually be able to saturate modern NVMe SSDs when executing a single query on all CPU cores on a larger-than-RAM index but I don't think we are there yet (we don't have enough async/concurrent/prefetch IO? and we don't strongly separate dependent IO into their own paths so non-dependent IOPs don't block one another).

I also want to test if Lucene is anywhere near the linux kernel bottleneck limit on page faults / sec. This is a potential risk we face with memory-mapped IO (described here, but I think there's been good progress in very modern kernels, something about maple trees -- aha, here!). I think this benchmark should be able to tease out that bottleneck by comparing mmap IO vs NIO cold concurrent IOPs.

How do you simulate the memory pressure? I see you drop OS's IO cache (at the start of each run?). Is it just that the user is expected to test on a large enough file exceeding their free RAM. You could also use @rmuir's awesome ramhog.c in luceneutil: https://github.com/mikemccand/luceneutil/blob/main/src/python/ramhog.c ... it just allocates ans pins that amount of RAM so OS cannot use it anymore.

Have you tried comparing this benchy to other IO benchy tools e.g. fio? Then we could gain some confidence in our benchy... (or, maybe lose some confidence in fio!).

[neoremind]

I ran the benchmark across three different hardware profiles to get quantitative measurements of how different I/O strategies work under different memory pressure. Here are the results.

Platforms

Platform	CPU	RAM	Storage	4K random read latency (QD=1)	Saturated random read throughput
Mac (M3 Pro)	12 CPU	36G	512G Apple SSD	~77us	~840 MB/s
Linux EBS (EC2 c5.4xlarge)	16 vCPU	32G	EBS io2, 200G, 20K provisioned IOPS	~334us	~80 MB/s
Linux NVME (EC2 c6id.4xlarge)	16 vCPU	32G	NVME SSD	~103us	~1.25 GB/s

The latency and throughput numbers are calculated via fio with rand read 16k + different iodepth/numjobs to find the saturation point.

Benchmark Setup

16k reads (4 pages) x 16 reads/op = 256KB per op.

Throughput (MB/s) = ops/ms × 256k x 1000. For example, with O_DIRECT on c6id.4xlarge (Linux NVME SSD), JMH at 16 threads yields 4.9 ops/ms → 4.9 × 256k x 1000 ≈ 1250 MB/s, matching the peak random read throughput from fio.

I removed the O_DIRECT results from the table charts below to focus on mmap (with MMapDirectory), pread (via FFI), and NIOFSDirectory (backed by FileChannel).

Random Read Results

CASE 1: Fully warm (all data in page cache)

16G file fits in RAM, pre-warmed with cat file > /dev/null.

• When everything is in RAM, mmap removes syscall overhead, it's pure page-table lookup with direct pointer to cached page, also no intermediate offheap buffer copy in NIO FileChannel. At T8, mmap is 1.5-2x faster than pread and nio FileChannel.
• Observed that batched WILLNEED prefetch adds ~25–30% overhead since the posix_madvise syscall is not necessary when pages are already in RAM (see pure mmap w/o using MMapDirectory in Lucene). But here MemorySegmentIndexInput#prefetch uses a power-of-two backoff counter to skip prefetch when pages are loaded, this avoids the slightly posix_madvise penalty for hot data and high IOPS scenarios, so we can see there is no difference in performance between all variants of mmap.
• NIOFSDirectory (backed by FileChannel) hits a thread-scaling wall, past ~4 threads, the NativeThreadSet monitor contention bottlenecked performance as found in Introduce a pread Directory based on Panama-FFI ? #16044. Pread doesn't suffer from this.

Detailed JMH results (ops/ms, higher is better)

Mac M3 Pro (36G RAM, unified memory, Apple SSD)

Benchmark	T01	T04	T08	T16
mmap (NORMAL)	77.98	224.69	285.09	296.12
mmap (RANDOM)	77.99	224.72	285.42	295.74
mmap + batchedPrefetch	77.62	223.26	284.43	294.45
mmap (RANDOM) + batchedPrefetch	77.78	222.51	284.75	293.86
ffiPread	37.43	93.84	73.71	65.64
fileChannelNIOFS	28.05	80.32	67.77	66.43

Linux c5.4xlarge (16 vCPU, 32G RAM, EBS io2 20K IOPS)

Benchmark	T01	T04	T08	T16
mmap (NORMAL)	37.46	141.00	261.55	282.78
mmap (RANDOM)	36.93	141.52	261.13	282.63
mmap + batchedPrefetch	37.00	138.04	257.00	280.10
mmap (RANDOM) + batchedPrefetch	36.54	138.23	256.77	280.09
ffiPread	22.38	77.67	134.53	185.46
fileChannelNIOFS	18.38	58.24	82.17	77.16
ffiPreadDirectIO (O_DIRECT)	0.15	0.63	1.27	1.25

Linux c6id.4xlarge (16 vCPU, 32G RAM, local NVME SSD)

Benchmark	T01	T04	T08	T16
mmap (NORMAL)	45.16	174.57	316.09	332.71
mmap (RANDOM)	46.53	173.58	316.22	332.91
mmap + batchedPrefetch	46.22	171.98	312.48	330.43
mmap (RANDOM) + batchedPrefetch	45.25	171.93	312.14	330.34
ffiPread	31.53	114.05	204.93	261.52
fileChannelNIOFS	24.57	78.74	102.25	97.01
ffiPreadDirectIO (O_DIRECT)	0.66	2.47	4.46	4.80

CASE 2: Memory pressure (working set close to RAM)

32G file barely exceeds RAM, pre-warmed.

• On Mac, mmap is the best strategy.
• NIOFSDirectory thread contention disappears when there are quite amount of I/O, since the I/O latency dwarfs the monitor overhead, so threads are no longer hot-contending on NativeThreadSet.
• On Linux, pread beats plain mmap at all thread counts with memory pressure not matter how big. I think mmap's page-fault path is more expensive than pread's syscall on Linux, especially under memory pressure. This paper from Andy Pavlo "Are You Sure You Want to Use MMAP in Your Database Management System?" argues mmap is not the best option for larger-than-memory DBMS workloads if not used carefully. It notes InfluxDB, MongoDB, and SingleStore deprecated mmap in their latest releases. Section 3.4 (Problem-4: Performance Issues) in the paper points to OS page eviction mechanisms that don't scale beyond a few threads for larger-than-memory workloads on high-bandwidth storage with bottlenecks like page table contention, page eviction overhead, and TLB shootdowns. @mikemccand also pointed out the mmap lock bottleneck above. There's also a counter-argument worth reading for balance.
• This is a case where we still hit hot pages most of the time, but whenever we encounter I/O, mmap is not as good as pread, and what's worse, the backoff mechanism weakens mmap + prefetch. Batched prefetch should theoretically be the most competitive here (as demonstrated in #16044's benchmarks using customized mmap + prefetch without using MMapDirectory), but the power-of-two backoff counter causes it to skip most madvise calls, so its benefit diminishes or gets emptied away. All in all, without prefetch, mmap alone is not competitive with pread under memory pressure on Linux, prefetch is the silver bullet that makes mmap performant in this scenario.

Detailed JMH results (ops/ms, higher is better)

Mac M3 Pro (36G RAM, unified memory, Apple SSD)

Benchmark	T01	T04	T08	T16
mmap (NORMAL)	4.76	15.73	23.50	34.65
mmap (RANDOM)	4.80	15.02	24.42	33.40
mmap + batchedPrefetch	2.54	19.52	28.09	38.22
mmap (RANDOM) + batchedPrefetch	5.28	16.54	23.17	32.46
ffiPread	2.20	7.60	12.10	16.94
fileChannelNIOFS	1.87	6.86	11.11	16.02

Linux c5.4xlarge (16 vCPU, 32G RAM, EBS io2 20K IOPS)

Benchmark	T01	T04	T08	T16
mmap (NORMAL)	0.85	3.46	3.62	3.82
mmap (RANDOM)	0.50	2.03	3.38	3.35
mmap + batchedPrefetch	0.93	3.81	4.50	4.33
mmap (RANDOM) + batchedPrefetch	0.50	2.00	3.33	3.38
ffiPread	1.48	5.15	5.48	5.12
fileChannelNIOFS	1.12	4.40	4.84	4.78
ffiPreadDirectIO (O_DIRECT)	0.15	0.63	1.28	1.25

Linux c6id.4xlarge (16 vCPU, 32G RAM, local NVME SSD)

Benchmark	T01	T04	T08	T16
mmap (NORMAL)	3.29	11.51	18.86	24.14
mmap (RANDOM)	2.14	8.45	15.29	26.36
mmap + batchedPrefetch	3.96	13.21	16.49	20.07
mmap (RANDOM) + batchedPrefetch	2.25	8.58	15.52	25.87
ffiPread	6.04	19.05	31.47	33.87
fileChannelNIOFS	4.19	15.32	26.14	29.54
ffiPreadDirectIO (O_DIRECT)	0.66	2.46	4.45	4.80

CASE 3: Many cold reads (~50% cold reads)

64G file (2x available RAM), ~50% cold reads, pre-warmed.

• mmap + batched prefetch catches up or surpasses pread here. With ~50% of pages cold, there's a much higher chance for willneed prefetch to actually trigger async I/O operations that deepen iodepth and fill the I/O queue. Even at T01, the prefetch benefit is already very obvious, throughput climbs up high with just one thread due to batched async I/O.
• Interestingly, RANDOM is particularly useful under high-concurrency on fast NVME, likely because it avoids wasted readahead pages that would otherwise be evicted before use. On EBS however, NORMAL wins, the kernel's sequential readahead amortizes the high per-request latency of remote block fetches.

Detailed JMH results (ops/ms, higher is better)

Mac M3 Pro (36G RAM, unified memory, Apple SSD)

Benchmark	T01	T04	T08	T16
mmap (NORMAL)	0.45	1.55	2.58	3.27
mmap (RANDOM)	0.45	1.97	3.17	4.28
mmap + batchedPrefetch	1.07	2.72	3.66	4.32
mmap (RANDOM) + batchedPrefetch	0.98	2.83	3.84	4.61
ffiPread	0.58	2.53	4.16	5.53
fileChannelNIOFS	0.78	2.64	4.02	5.44

Linux c5.4xlarge (16 vCPU, 32G RAM, EBS io2 20K IOPS)

Benchmark	T01	T04	T08	T16
mmap (NORMAL)	0.14	0.49	0.49	0.52
mmap (RANDOM)	0.07	0.28	0.46	0.46
mmap + batchedPrefetch	0.89	1.51	1.52	1.50
mmap (RANDOM) + batchedPrefetch	0.66	1.50	1.49	1.48
ffiPread	0.25	0.96	1.67	1.52
fileChannelNIOFS	0.22	0.88	1.42	1.38
ffiPreadDirectIO (O_DIRECT)	0.15	0.62	1.27	1.25

Linux c6id.4xlarge (16 vCPU, 32G RAM, local NVME SSD)

Benchmark	T01	T04	T08	T16
mmap (NORMAL)	0.64	1.21	1.22	1.23
mmap (RANDOM)	0.28	1.09	2.01	3.41
mmap + batchedPrefetch	3.63	5.75	6.14	6.26
mmap (RANDOM) + batchedPrefetch	3.99	7.21	7.20	7.21
ffiPread	1.06	3.59	5.39	5.15
fileChannelNIOFS	0.88	3.23	5.15	4.98
ffiPreadDirectIO (O_DIRECT)	0.66	2.47	4.45	4.80

CASE 4: Almost all cold reads

64G file (2x available RAM), clear page cache before each iteration, cold start.

• Same story as Case 3, but the gap widens since mmap + batchedPrefetch wins more on both EBS and NVME.
• The current MMapDirectory + prefetch backoff strategy is well-designed for the fully warm and almost all code reads, but when data is mix of warm and cold, the backoff counter ramps up from hot-page hits and skips the batched WILLNEED hints diminishing the effect. So whether to enable prefetch all the time is not one-size-fits-all, it depends on query pattern like warm/cold read ratio and what if warm loads are accessed first. This aligns with my finding in MemorySegmentIndexInput: always prefetch on RANDOM mode #16145 (comment).

Detailed JMH results (ops/ms, higher is better)

Mac M3 Pro (36G RAM, unified memory, Apple SSD)

Benchmark	T01	T04	T08	T16
mmap (NORMAL)	0.30	0.67	1.97	2.29
mmap (RANDOM)	0.19	0.98	1.84	2.85
mmap + batchedPrefetch	0.40	1.49	2.56	3.45
mmap (RANDOM) + batchedPrefetch	0.42	1.51	2.54	3.38
ffiPread	0.41	1.58	2.61	3.38
fileChannelNIOFS	0.38	1.55	2.56	3.41
ffiPreadDirectIO (F_NOCACHE)	0.44	1.97	3.56	5.68

Linux c5.4xlarge (16 vCPU, 32G RAM, EBS io2 20K IOPS)

Benchmark	T01	T04	T08	T16
mmap (NORMAL)	0.08	0.33	0.32	0.33
mmap (RANDOM)	0.04	0.16	0.26	0.26
mmap + batchedPrefetch	1.36	1.35	1.35	1.38
mmap (RANDOM) + batchedPrefetch	1.34	1.35	1.35	1.37
ffiPread	0.14	0.57	1.15	1.36
fileChannelNIOFS	0.14	0.58	1.15	1.36
ffiPreadDirectIO (O_DIRECT)	0.16	0.63	1.27	1.28

Linux c6id.4xlarge (16 vCPU, 32G RAM, local NVME SSD)

Benchmark	T01	T04	T08	T16
mmap (NORMAL)	0.45	0.87	0.87	0.94
mmap (RANDOM)	0.16	0.62	1.18	2.11
mmap + batchedPrefetch	4.60	6.07	6.08	6.17
mmap (RANDOM) + batchedPrefetch	4.53	6.26	6.26	6.64
ffiPread	0.63	2.35	4.24	4.65
fileChannelNIOFS	0.62	2.32	4.20	4.65
ffiPreadDirectIO (O_DIRECT)	0.66	2.47	4.45	4.95

Sequential Read Results

64G file, page cache dropped before each iteration. 128 sequential reads per op at varying read sizes (16KB, 64KB, 128KB).

Look at Linux, mmap (NORMAL) with default kernel readahead performs well across the board for sequential access. mmap (RANDOM) + batchedPrefetch is as good as mmap (NORMAL) with readahead, again, the power-of-two backoff diminishes the effect.

Detailed JMH results (ops/ms, higher is better)

Mac M3 Pro (36G RAM, unified memory, Apple SSD)

Benchmark	16KB	64KB	128KB
mmap (NORMAL)	4.032	0.993	0.503
mmap (SEQUENTIAL)	2.292	0.601	0.288
mmap (RANDOM)	1.839	0.452	0.231
mmap (RANDOM) + batchedPrefetch	5.391	1.546	0.871
ffiPread	2.009	0.661	0.577
fileChannelNIOFS	2.123	0.510	0.261
ffiPreadDirectIO (F_NOCACHE)	0.860	0.218	0.152

Linux c5.4xlarge (16 vCPU, 32G RAM, EBS io2 20K IOPS)

Benchmark	16KB	64KB	128KB
mmap (NORMAL)	2.223	0.601	0.272
mmap (SEQUENTIAL)	2.236	0.563	0.281
mmap (RANDOM)	0.101	0.024	0.011
mmap (RANDOM) + batchedPrefetch	2.205	1.136	0.565
ffiPread	1.922	0.553	0.275
fileChannelNIOFS	2.276	0.491	0.266
ffiPreadDirectIO (O_DIRECT)	0.163	0.072	0.035

Linux c6id.4xlarge (16 vCPU, 32G RAM, local NVME SSD)

Benchmark	16KB	64KB	128KB
mmap (NORMAL)	9.437	2.374	1.189
mmap (SEQUENTIAL)	7.405	1.848	0.925
mmap (RANDOM)	0.722	0.180	0.090
mmap (RANDOM) + batchedPrefetch	9.816	2.454	1.224
ffiPread	9.694	2.388	1.200
fileChannelNIOFS	11.027	2.720	1.386
ffiPreadDirectIO (O_DIRECT)	0.703	0.153	0.076

fio vs. JMH benchmarks

I cross-validated against fio on c6id.4xlarge (NVME). The JMH numbers match fio within about 4% overhead (like JVM indirections, jmh blackhole?):

• Sequential 1MB reads: JMH pread/NIOFS/mmap 12.01K IOPS vs. fio (sync/psync/libaio/mmap) 12.5K IOPS, both saturate disk bandwidth

Command used

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar "SequentialReadIOBenchmark\.(fileChannelNIOFS)" \
  -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \
  -p readSize=1048576 -p readsPerOp=1 -r 30

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar "SequentialReadIOBenchmark\.(ffiPread)" \
  -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \
  -p readSize=1048576 -p readsPerOp=1 -r 30

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar "SequentialReadIOBenchmark\.(mmap)" \
  -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \
  -p readSize=1048576 -p readsPerOp=1 -r 30

fio --name=seqread \
--rw=read \
--bs=1m \
--size=64g \
--numjobs=1 \
--ioengine=psync \
--direct=1 \
--runtime=30 \
--time_based \
--group_reporting \
--filename=/mnt/local/bench-64G.dat

fio --name=seqread \
--rw=read \
--bs=1m \
--size=64g \
--numjobs=1 \
--ioengine=mmap \
--direct=1 \
--runtime=30 \
--time_based \
--group_reporting \
--filename=/mnt/local/bench-64G.dat

fio --name=seqread \
--rw=read \
--bs=1m \
--size=64g \
--numjobs=1 \
--ioengine=libaio \
--direct=1 \
--runtime=30 \
--time_based \
--group_reporting \
--filename=/mnt/local/bench-64G.dat

• Random 16KB reads (4 threads, QD=1): JMH pread 40.5K IOPS vs. fio libaio 42.8K IOPS

Command used

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar "RandomReadIOBenchmark\.(ffiPread_T04)" \
  -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \
  -p readSize=16384 -p readsPerOp=1 -f 1 -wi 1 -w 3 -i 1 -r 30

## no direct set, so some page caches hit

fio --name=randread \
--rw=randread \
--bs=16k \
--size=64g \
--iodepth=x \
--numjobs=4 \
--ioengine=libaio \
--runtime=30 \
--time_based \
--group_reporting \
--filename=/mnt/local/bench-64G.dat

• Random 16KB with mmap: JMH and fio mmap (QD=1) can both achieve 12K IOPS

Command used

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar "RandomReadIOBenchmark\.(mmap_T04)" \
  -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \
  -p readSize=16384 -p readsPerOp=1 -f 1 -wi 1 -w 3 -i 1 -r 10
  
fio --name=randread \
--rw=randread \
--bs=16k \
--size=64g \
--iodepth=1 \
--numjobs=4 \
--ioengine=mmap \
--runtime=30 \
--time_based \
--group_reporting \
--filename=/mnt/local/bench-64G.dat

• Random 16KB with mmap: JMH (mmap + batched prefetch) and fio libaio (QD=16, numjobs=4) can both saturate I/O with 80K IOPS and 1.25G throughput.

Command used

# convert result by 5.0 ops/ms x 16 ops x 1000 = 80K IOPS
# JMH can go even higher if we extend running time from 10 seconds to more because of more hot page cache hit
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar "RandomReadIOBenchmark\.(mmapRandom_T04)" \
  -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \
  -p readSize=16384 -p readsPerOp=16 -f 1 -wi 1 -w 1 -i 1 -r 10

fio --name=randread \
--rw=randread \
--bs=16k \
--size=64g \
--iodepth=16 \
--numjobs=4 \
--ioengine=libaio \
--direct=1 \
--runtime=30 \
--time_based \
--group_reporting \
--filename=/mnt/local/bench-64G.dat

[neoremind]

Thank you @mikemccand! I've updated the benchmark results and analysis above, please see my comments inline below.

I love that we are building tooling for benchmarking cold/warm cases -- they are so tricky to properly test because at a "whole search system" level you should replay query traffic with accurate actual production arrival times (ideally, or simulate w/ Poisson process) for real measures. I'd love to see Lucene eventually be able to saturate modern NVMe SSDs when executing a single query on all CPU cores on a larger-than-RAM index but I don't think we are there yet (we don't have enough async/concurrent/prefetch IO? and we don't strongly separate dependent IO into their own paths so non-dependent IOPs don't block one another).

I fully agree on the vision. I think it's genuinely possible to saturate I/O as long as we interleave computation and I/O with parallelization. I've practiced this strategy before, during the 1st PolarDB database performance competition in 2018, I implemented a KV engine saturating an Intel Optane SSD with 2.2 GB/s write and 2.5 GB/s random read (see write-up). I also built a fast InnoDB checksum tool in C++ that fully exploits hardware by separating I/O from computation, using big-block I/O to amortize kernel overhead, and making full use of multi-core. But indeed, these are either contest simplified workloads or easy utilities, Lucene's query path is much much more complex. One thing I learned from @jimczi is how MMapDirectory's prefetch is already working good now in StoredFields#prefetch(docID) prefetching, and KnnVectorValues#prefetch(int[] ords, int numOrds) in PrefetchableFlatVectorScorer does the same for vector search, and more.

I also want to test if Lucene is anywhere near the linux kernel bottleneck limit on page faults / sec. This is a potential risk we face with memory-mapped IO (described here, but I think there's been good progress in very modern kernels, something about maple trees -- aha, here!). I think this benchmark should be able to tease out that bottleneck by comparing mmap IO vs NIO cold concurrent IOPs.

Thanks for sharing these. The per-VMA lock optimization should be available after Linux 6.4, but my EC2 hosts are still running old kernels. I will give it try once available.

With the data shows above, mmap is the most powerful one when data is hot in RAM, but pread/NIOFS outperforms plain mmap under memory pressure with some degree of cold load (case 2: file size nearly RAM, case 3: 50% cold reads, case 4: almost all cold). On NVME at T16 in the all-cold case: plain mmap can only do 0.94 ops/ms while pread reaches 4.65 ops/ms, however, mmap (RANDOM) + batched WILLNEED prefetch (simulate QD=16) goes to 6.64 ops/ms, beating both pread and NIOFS. This somehow shows that I could hit the mmap bottleneck (page table contention, TLB shootdowns, mmap lock contention) on the cold path, and batched prefetch makes mmap more performant by pipelining I/O and computation.

How do you simulate the memory pressure? I see you drop OS's IO cache (at the start of each run?). Is it just that the user is expected to test on a large enough file exceeding their free RAM. You could also use @rmuir's awesome ramhog.c in luceneutil: https://github.com/mikemccand/luceneutil/blob/main/src/python/ramhog.c ... it just allocates and pins that amount of RAM so OS cannot use it anymore.

I use files that go beyond available RAM, 32G/64G file on a 32G host. Just for the "almost all cold" case 4, I drop page cache before each JMH iteration, others no, just let program contended on memory. The dropping page cache is a switch as param. I also tried using cgroup to limit memory, but ran into a tricky issue, dropping page cache within the cgroup doesn't fully remove pages from host memory, anyway, I didn't go deeper the why. So, I ended up using a host with less RAM instead. I think @rmuir's ramhog.c would be a cleaner approach for memory control, I will take a look.

Have you tried comparing this benchy to other IO benchy tools e.g. fio? Then we could gain some confidence in our benchy... (or, maybe lose some confidence in fio!).

Yes, please see the fio section in the results above. In short: JMH numbers match fio with tiny overhead.