|
| 1 | +macOS accept()/shutdown() quirk in the DASH ingest server |
| 2 | +========================================================== |
| 3 | + |
| 4 | +Symptom |
| 5 | +------- |
| 6 | +The macOS CI job (build-and-test macos-latest) appeared to time out or fail on |
| 7 | +every push to feature/ctedash, taking ~26 minutes. The real failure was the |
| 8 | +openmoq-publisher-live-dash-tests binary hanging until ctest killed it at the |
| 9 | +1500 second per-test timeout (exit code 8). Linux and Windows were unaffected. |
| 10 | + |
| 11 | +Root cause |
| 12 | +---------- |
| 13 | +Commit 6c08575 changed LiveDashIngestServer::stop() to keep the listening file |
| 14 | +descriptor open (only shutdown() it) until the accept thread joins. That is |
| 15 | +correct for the fd-reuse race it fixed, but it relies on shutdown() waking a |
| 16 | +blocked accept(). |
| 17 | + |
| 18 | +Platform difference: |
| 19 | +- Linux: shutdown(listen_fd, SHUT_RDWR) on a listening socket wakes a blocked |
| 20 | + accept(), which returns with EINVAL. Tests pass. |
| 21 | +- macOS/BSD: shutdown() on a listening (unconnected) socket fails with ENOTCONN |
| 22 | + and does NOT wake accept(). The accept thread sleeps forever, stop() blocks |
| 23 | + in accept_thread.join(), and the first server start/stop test hangs. |
| 24 | + |
| 25 | +Historically the code closed the listening fd during stop(), and on BSD-derived |
| 26 | +systems close() does wake accept() - which is why the hang only appeared once |
| 27 | +the close was deferred until after the join. |
| 28 | + |
| 29 | +A follow-up commit (8b4b99b, "make DASH ingest shutdown responsive on macOS") |
| 30 | +only fixed the client recv() path with SO_RCVTIMEO polling; it did not touch |
| 31 | +accept(), so the hang persisted. |
| 32 | + |
| 33 | +Fix (commit 65c5a4f) |
| 34 | +-------------------- |
| 35 | +In src/live_dash_ingest.cpp: |
| 36 | +1. The accept loop never parks in accept(). It waits in poll() on the listening |
| 37 | + fd with a bounded 100 ms timeout and re-checks stop_requested each cycle, |
| 38 | + mirroring the client workers' SO_RCVTIMEO receive cadence. |
| 39 | +2. The listening socket is set O_NONBLOCK after listen(), so a connection that |
| 40 | + is aborted between the poll() wakeup and the accept() call cannot re-block |
| 41 | + the loop. |
| 42 | +3. handle_client() clears O_NONBLOCK on the accepted client fd before applying |
| 43 | + SO_RCVTIMEO. BSD-derived systems (macOS) make accepted sockets inherit the |
| 44 | + listener's O_NONBLOCK (Linux does not); without clearing it the worker recv |
| 45 | + loop would hot-spin on EAGAIN instead of honoring the timed cadence. |
| 46 | + |
| 47 | +Portable-shutdown rules worth remembering |
| 48 | +----------------------------------------- |
| 49 | +- Never rely on shutdown() of a listening socket to unblock accept() - it is |
| 50 | + Linux-only behavior. Use a bounded poll()+stop-flag loop, a self-pipe, or a |
| 51 | + wake connection instead. |
| 52 | +- Accepted sockets inherit O_NONBLOCK from the listener on BSD/macOS but not |
| 53 | + on Linux; normalize the flag explicitly on the accepted fd. |
| 54 | +- Keeping a listening fd open until its accept thread joins avoids fd-reuse |
| 55 | + races, but the wake mechanism must then be something other than close(). |
| 56 | + |
| 57 | +Verification: after 65c5a4f, macOS CI run 28627497050 passed in 43 seconds. |
0 commit comments