Use-after-free in epoll muxnote bookkeeping under concurrent `FileHandle.readabilityHandler` teardown (Linux)

[rintaro]

Reproducer

Summary

On Linux, rapidly setting up and tearing down FileHandle.readabilityHandler from multiple threads corrupts libdispatch's epoll muxnote state, and the libdispatch manager thread ("DispatchWorker") crashes with SIGSEGV dereferencing a freed/garbage dispatch_muxnote. It reproduces with pure Foundation (no subprocess) — readabilityHandler installs a libdispatch read source on a dup() of the fd, and the concurrent register/teardown churn leaves a dispatch_muxnote freed while it is still reachable (via the _dispatch_sources hash bucket list and/or a live epoll registration). This is not a client-side close-before-cancel misuse: FileHandle monitors a private dup() and closes it only in the source's cancel handler, so the dup guard is correct.

Environment

• Swift 6.5-dev (nightly main), +assertions — Docker image swiftlang/swift:nightly-main-noble
• aarch64-unknown-linux-gnu, Ubuntu 24.04, kernel 6.14.9, running in an aarch64 Linux VM (10 CPUs / 16 GB)

Reproducer

Pure Foundation, no subprocess. Each iteration installs a readabilityHandler (which dups the fd and arms a read source), reads via availableData and clears the handler on EOF, then writes a few bytes and closes the write end so the source fires and tears down. Many of these run concurrently under CPU pressure.

import Foundation
#if canImport(Glibc)
import Glibc
#elseif canImport(Musl)
import Musl
#endif

let durationSeconds = Double(CommandLine.arguments.dropFirst().first ?? "600") ?? 600
let workerThreads = Int(CommandLine.arguments.dropFirst(2).first ?? "24") ?? 24

let deadline = Date().addingTimeInterval(durationSeconds)
let statsLock = NSLock()
var completedCycles = 0

@inline(never)
func stressOneReadabilityHandler() {
  let pipe = Pipe()
  let readEnd = pipe.fileHandleForReading
  let writeEnd = pipe.fileHandleForWriting

  readEnd.readabilityHandler = { handle in
    let data = handle.availableData
    if data.isEmpty {
      handle.readabilityHandler = nil   // EOF: cancel the read source
    }
  }

  try? writeEnd.write(contentsOf: Data([0x41, 0x42, 0x43, 0x0a]))
  try? writeEnd.close()
}

var workers: [Thread] = []
for _ in 0..<workerThreads {
  let worker = Thread {
    while Date() < deadline {
      stressOneReadabilityHandler()
      statsLock.lock(); completedCycles += 1; statsLock.unlock()
      usleep(20)   // let async teardown drain so live fds stay bounded
    }
  }
  worker.stackSize = 4 << 20
  workers.append(worker)
  worker.start()
}

while Date() < deadline {
  sleep(5)
  statsLock.lock(); let cycles = completedCycles; statsLock.unlock()
  FileHandle.standardError.write(Data("cycles=\(cycles)\n".utf8))
}

print("Finished \(completedCycles) cycles without crashing.")

Steps to reproduce

swiftc -O ReadabilityHandlerCrash.swift -o readability-crash
for i in $(seq "$(nproc)"); do yes >/dev/null & done   # saturate CPU
ulimit -c unlimited; ulimit -n 200000
./readability-crash 600 24

Or fully containerized against the nightly image (this is how it was reproduced; works with docker run in place of container run, dropping --platform if the host is already aarch64):

container run --rm \
  --platform linux/arm64 --cpus 10 --memory 16g \
  --volume "$PWD:/repro" \
  swiftlang/swift:nightly-main-noble \
  bash -c '
    ulimit -c unlimited
    ulimit -n 200000
    echo "/repro/core.%e.%p" > /proc/sys/kernel/core_pattern
    export SWIFT_BACKTRACE=enable=no
    cd /repro
    swiftc -O ReadabilityHandlerCrash.swift -o readability-crash
    for i in $(seq "$(nproc)"); do yes >/dev/null & done
    ./readability-crash 1800 24
  '

The process exits with code 139 (SIGSEGV) and writes core.DispatchWorker.<pid> into the mounted directory. Capturing the core requires the container to permit writing /proc/sys/kernel/core_pattern (may need --privileged under some Docker setups, or set core_pattern on the host); the crash reproduces regardless.

It reproduces far more reliably under CPU pressure. In two runs it crashed at ~36.5M cycles (~9 minutes) and ~72.5M cycles (~18 minutes) with 24 worker threads plus nproc CPU burners; budget 15–30 minutes under load. With an idle machine it may not reproduce.

Backtraces

The crash lands on whichever muxnote access loses the race; both sites dereference a non-canonical/garbage pointer on the manager thread.

Registration path (this reproducer), inside _dispatch_muxnote_find's bucket walk:

Thread "DispatchWorker": SIGSEGV, fault address 0x0000fff07ff83ed7
  #0 _dispatch_unote_register_muxed + 220   libdispatch.so
  swiftlang/swift-corelibs-foundation#1 _dispatch_source_invoke        + 480
  swiftlang/swift-corelibs-foundation#2 _dispatch_lane_serial_drain    + 272
  swiftlang/swift-corelibs-foundation#3 _dispatch_mgr_invoke           + 164
  swiftlang/swift-corelibs-foundation#4 _dispatch_mgr_thread           + 132
  swiftlang/swift-corelibs-foundation#5 _dispatch_worker_thread        + 644

+220 is the dmn->dmn_ident load inside LIST_FOREACH(dmn, dmb, dmn_list) in _dispatch_muxnote_find (src/event/event_epoll.c): the walk starts from a valid bucket head and follows dmn_list.le_next into a freed/garbage muxnote.

Delivery path (originally observed in CI while tearing down a subprocess readabilityHandler), inside _dispatch_event_merge_fd inlined into the epoll drain:

Thread "DispatchWorker": SIGSEGV
  #0 _dispatch_event_loop_drain + ~1228     libdispatch.so
  swiftlang/swift-corelibs-foundation#1 _dispatch_mgr_invoke
  swiftlang/swift-corelibs-foundation#2 _dispatch_mgr_thread

Here epoll_wait returned an event whose data.ptr is a muxnote that had already been unlinked from _dispatch_sources and free()d (confirmed via core dump: the muxnote was absent from the live _dispatch_sources[] buckets, and its fields were reused-heap garbage); _dispatch_retain_unote_owner then ~-decodes the reused du_owner_wref to a garbage owner and faults incrementing its refcount.

Analysis

Concurrent read-source registration together with EOF-driven teardown leaves the epoll muxnote lifecycle inconsistent: a dispatch_muxnote is freed while still reachable. On EOF the pipe read end is persistently readable, so the EV_DISPATCH source re-arms and epoll keeps delivering, while the handler concurrently does readabilityHandler = nil (cancel → _dispatch_muxnote_dispose). The registration-path backtrace additionally shows the _dispatch_sources bucket list itself being corrupted (a freed muxnote left linked), so the failure is not limited to stale epoll delivery. The specific in-handler read call is irrelevant — read(upToCount:) reproduces the same crash — the defect is in the read source's registration/teardown, not in how bytes are consumed.