epoll_wait race condition on mac os

[wirerhino]

On Mac OS 15.5 (24F74) I have the following C wrappers:

int b_epoll_ctl(int epfd, int op, int fd, struct epoll_event *event) {
    int ret = epoll_ctl(epfd, op, fd, event);
    if (fd == 0) {
        fprintf(stderr, "b_epoll_ctl: fd == 0\n");
    }
    if (op == EPOLL_CTL_ADD) {
        fprintf(stderr, "b_epoll_ctl: registered fd == %d\n", event->data.fd);
    }
    return ret == -1 ? -errno : ret;
}

int b_epoll_wait(int epfd, struct epoll_event *events, int maxevents, int timeout) {
    int ret = epoll_wait(epfd, events, maxevents, timeout);

    for (int i = 0; i < ret; i++) {
        if (events[i].data.fd == 0) {
            fprintf(stderr, "b_epoll_wait: fd == 0\n");
        }
    }

    int mask = ret >> (sizeof(int) * 8 - 1);
    return (mask & -errno) | (~mask & ret);
}

Then I set up a test where multiple threads create their own Epoll FD and start dealing with sockets.
Sometimes, under massive pressure (around one thousand concurrent instances of openssl s_time ) , the above code does not record any registration for FD 0, but epoll_wait returns 1 and reports FD 0 as closed even if it was never added to the epoll_set

It looks like the issue is triggered if I perform a close() after epoll_ctl DEL, but from another thread

It mostly read 0, but it can happen to read also other "ghost" fd that have never been added.
I ran the test suite, this is the result:

The following tests FAILED:
9 - epoll-test.epoll__event_size (Failed)
53 - epoll-test-interpose.epoll__event_size (Failed)
96 - epoll-test-rdhup-linux-def.epoll__event_size (Failed)
138 - timerfd-test.timerfd__expire_five (Failed)
157 - timerfd-test-interpose.timerfd__expire_five (Failed)
Errors while running CTest

[jiixyj]

From a quick look at src/epollfd_ctx.c I can't see how events[i].data.fd == 0 can be returned if it wasn't put in earlier by some registration.

Can you create a minimal reproducer? This would help in tracking down this problem.

The test failures don't seem too relevant here. See also #41. Can you provide detailed logs for the failing timerfd tests?

[wirerhino]

Looks indeed a similar issue the one regarding the size:
-- result: 1, failed: /Users/epoll-shim/test/epoll-test.c:381: sizeof(event) == 12 not met
CMake Error at /Users/epoll-shim/external/microatf/cmake/ATFRunTest.cmake:69 (message):
/Users/epoll-shim/test/epoll-test.c:381: sizeof(event) == 12
not met

The timer failure:
138/313 Test #138: timerfd-test.timerfd__expire_five ...............................................***Failed 7.10 sec
timer expired 1 times
timer expired 1 times
timer expired 1 times
timer expired 1 times
-- result: 1, failed: /Users/epoll-shim/test/timerfd-test.c:90: -1 35
CMake Error at /Users/epoll-shim/external/microatf/cmake/ATFRunTest.cmake:69 (message):
/Users/epoll-shim/test/timerfd-test.c:90: -1 35

    Start 157: timerfd-test-interpose.timerfd__expire_five

157/313 Test #157: timerfd-test-interpose.timerfd__expire_five .....................................***Failed 7.11 sec
timer expired 1 times
timer expired 1 times
timer expired 1 times
timer expired 1 times
-- result: 1, failed: /Users/epoll-shim/test/timerfd-test.c:90: -1 35
CMake Error at /Users/epoll-shim/external/microatf/cmake/ATFRunTest.cmake:69 (message):
/Users/epoll-shim/test/timerfd-test.c:90: -1 35

I am using an Apple silicon M3

[jiixyj]

Thanks!

-- result: 1, failed: /Users/epoll-shim/test/timerfd-test.c:90: -1 35

It looks like some timing stuff doesn't work as expected on newer macOS version, returning -1 and EAGAIN instead of reading a value from a timerfd. But I think this is a separate issue.

[wirerhino]

I cannot provide the code or a minimal setup, BUT I tried adding first a file lock and later a shared pthread lock and it did not change the outcome so might not be a race. Interestingly, printing all the triggered file descriptors after a locked epoll_wait looks like when the rogue value is returned is always at the first position of the array:

b_epoll_wait: fd == 0 124 176 106 80
b_epoll_wait: fd == 0 126 174 110
b_epoll_wait: fd == 0 191 94 151
b_epoll_wait: fd == 0
b_epoll_wait: fd == 0 135

[jiixyj]

Have you looked at possible EPOLL_CTL_MOD calls? Maybe something calls EPOLL_CTL_MOD with data.fd == 0?

[wirerhino]

I had the same idea, so I began logging every request that is not a DEL

`int b_epoll_ctl(int epfd, int op, int fd, struct epoll_event *event) {

int ret = epoll_ctl(epfd, op, fd, event);

if (op != EPOLL_CTL_DEL) {
    fprintf(stderr, "b_epoll_ctl: registered fd (%d) == %d\n", fd, event->data.fd);
    if (fd != event->data.fd) {
        fprintf(stderr, "MISMATCH!\n");
    }
}
return ret == -1 ? -errno : ret;

}
`

and randomly epoll_wait returns a never registered FD, almost often 0

[jiixyj]

Is there any fork()ing done in the program, and the epoll fd accessed by a child process?

[wirerhino]

No:

T: Main ---- spawns ---> T: Acceptor---------------|
|------- spawns ---> T(N): epoll server <-------| Sends the accepted socket fd value to the epoll_server thread
|------- spawns ---> T: fd closer <---------| Sends the socked fd value that needs to be closed to the closer thread

Acceptor accept()s a new connection and sends it to a given epoll_server thread for management (fd % epoll_server_threads_num)
If I have only 1 epoll_server Thread everything works, as long as multiple epoll_server are created epoll_wait starts returning some garbage. All the epoll_server data is local, not shared.
If instead of doing close() in the fd closer Thread I do it within the epoll server thread where the fd was managed, it works

I just tried using a semaphore as well (with 1 permit available): I have been able to reproduce the issue synchronizing the epoll_wait, but I was NOT able to reproduce it synchronizing epoll_wait and epoll_ctl OR epoll_wait and close()

Update: I uploaded the source code to Gemini 2.5pro and it claims to have found several race conditions, deadlocks, funnels, use after free, improper rwlocks implementation and interaction with kqueue.
I attached the patch with the suggested changes for reference

omnifix.patch

[jiixyj]

You could also try linking against epoll-shim-interpose if you aren't doing that already. With this, calling close() will properly remove an fd from all epoll instances.

[wirerhino]

I already remove the fd before closing the fd to avoid being triggered by myself. So far the only solutions I've seen working are the patched kqueue/rwlocks from Gemini + the two semaphores mentioned above or another completely refactored Gemini version, but that has quite poor performances