Getting no read parallelism on macOS (kernel mutex reading same file in parallel threads)

[mhansen]

On #118 I removed a read mutex from ripunzip. From the benchmarking I did on Linux, it made a big difference. But I just tried it at home on macOS on my external SSD formatted with APFS and I found I wasn't getting parallelism.

If I sample from Activity Monitor, I clearly see the 10 threads are spending >90% of their time in read. We can't see kernel stacks in the sample app so I switched to Instruments.app

I don't have full kernel symbols but it seems somewhat clear what's going on from the symbols we do have:

• We are calling read
• The kernel delegates through a few functions (probably generic syscall handlers and then routers to figure out what type of filesystem this is)
• apfs_vnop_read is some apfs read function; probably the APFS implementation of read
• cluster_read_ext is "documented" here: https://developer.apple.com/documentation/kernel/1463712-cluster_read_ext
• through a few more unsymbolised functions we end up at lck_mutex_sleep.

It's surprising. We've opened the source file 10x for read-only, so I thought APFS might use a shared-reader/exclusive-writer lock, but it seems to be assuming all these reads can conflict and using a regular mutex.

There's a 2018 blogpost about APFS not having read parallelism in the specific context of apfs_vnop_readdir: https://gregoryszorc.com/blog/2018/10/29/global-kernel-locks-in-apfs/. It's worth a read. They lament that OS filesystem abstractions are holding back SSDs which can do ludicrous IOPS.

At this time, I haven't conducted a comprehensive analysis of APFS to determine what other filesystem operations seem to acquire global kernel locks: all I know is readdir() does.

Well, I think my profile indicates that read holds a lock too. Though I'd guess it's probably not a global kernel lock, but rather a per-file lock.

What options do we have to "fix" this?

1. Make a benchmark to investigate more where this happens (any reads on the same file? overlapping reads only?) and send it to Apple and hope they allow parallel reads in some future macOS version
2. Use mmap? Use memmap2 crate and wrap a Reader around it. The story about mmap in rust is complicated, due to it being unsafe if other apps change the file under you.
3. Try sharing a single file handle, but don't mutate the seek position, use pread. I'm not sure if this would also hit a lock
4. Triple-check we're opening the file for 'read' not 'read+write'. I think we are:

   ripunzip/src/unzip/multi_file_seeker.rs

   Line 25 in 236ed25

   let file = File::open(path)?;

   , and https://doc.rust-lang.org/std/fs/struct.File.html#method.open "Attempts to open a file in read-only mode."
5. Disable some parallelism in macOS? Maybe set it to '2' so we can both read and write in parallel? But macOS does support other filesystems, so maybe that's not right. Can we detect APFS?
6. Investigate some more about where the lock is held; perhaps it's held only for overlapping reads, which would only be reading the index metadata; perhaps we could read the zip index metadata once? See https://docs.rs/zip/latest/zip/read/struct.ZipArchive.html#method.unsafe_new_with_metadata