Skip to content

Commit 2cf952d

Browse files
tests: disable gRPC fork handlers to fix flaky fork() abort
The suite intermittently fails in CI due to (pexpect spawned) labgrid-client being killed during teardown by SIGABRT and gRPC C-core aborting after a fork: fork_posix.cc:71] Other threads are currently calling into gRPC, skipping fork() handlers ev_epoll1_linux.cc:1125] Check failed: next_worker->state == KICKED gRPC documents that fork() is only safe if gRPC objects are created in the child after the fork [1]. The tests do the opposite: the pytest process spawns the labgrid-client commands via pexpect (using forkpty) while gRPC is already initialized in it (imported at collection, and used in-process by the coordinator tests). To partially support this, gRPC installs a pthread_atfork handler that tries to quiesce its poller across the fork. When another thread is in the C-core at fork time the handler loses that race, skips the reset, and the forked child aborts on the inconsistent poller before it can exec the client. The problem could never be reproduced locally; it likely occurs more often in CI because scheduling pressure makes the race more likely to trigger. Set GRPC_ENABLE_FORK_SUPPORT=0 to disable the handler. This is the documented remedy for callers that always exec() after fork() and never reuse a channel across fork(), which is what the test suite does. Set it in conftest.py before pytest's collection imports grpc. [1] https://github.com/grpc/grpc/blob/master/doc/fork_support.md Signed-off-by: Bastian Krause <bst@pengutronix.de>
1 parent 04bb2b6 commit 2cf952d

1 file changed

Lines changed: 10 additions & 0 deletions

File tree

tests/conftest.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import logging
2+
import os
23
from signal import SIGTERM
34
import sys
45
import threading
@@ -13,6 +14,15 @@
1314

1415
psutil = pytest.importorskip("psutil")
1516

17+
# gRPC is only safe to fork() if gRPC objects are created in the child after the fork. The tests do
18+
# the opposite: the pytest process spawns labgrid-client via pexpect (forkpty) while gRPC is
19+
# already initialized in it, so gRPC's pthread_atfork handler can lose a race and abort the child
20+
# before it execs. Set GRPC_ENABLE_FORK_SUPPORT=0 to disable the handler; the spawned clients
21+
# fork+exec and never reuse an inherited channel, so it is not needed anyway.
22+
# See also https://github.com/grpc/grpc/blob/master/doc/fork_support.md
23+
os.environ.setdefault("GRPC_ENABLE_FORK_SUPPORT", "0")
24+
25+
1626
@pytest.fixture(scope="session")
1727
def curses_init():
1828
""" curses only reads the terminfo DB once on the first import, so make

0 commit comments

Comments
 (0)