Commit 2cf952d
committed
tests: disable gRPC fork handlers to fix flaky fork() abort
The suite intermittently fails in CI due to (pexpect spawned)
labgrid-client being killed during teardown by SIGABRT and gRPC C-core
aborting after a fork:
fork_posix.cc:71] Other threads are currently calling into gRPC, skipping fork() handlers
ev_epoll1_linux.cc:1125] Check failed: next_worker->state == KICKED
gRPC documents that fork() is only safe if gRPC objects are created in
the child after the fork [1]. The tests do the opposite: the pytest
process spawns the labgrid-client commands via pexpect (using forkpty)
while gRPC is already initialized in it (imported at collection, and used
in-process by the coordinator tests). To partially support this, gRPC
installs a pthread_atfork handler that tries to quiesce its poller across
the fork. When another thread is in the C-core at fork time the handler
loses that race, skips the reset, and the forked child aborts on the
inconsistent poller before it can exec the client.
The problem could never be reproduced locally; it likely occurs more
often in CI because scheduling pressure makes the race more likely to
trigger.
Set GRPC_ENABLE_FORK_SUPPORT=0 to disable the handler. This is the
documented remedy for callers that always exec() after fork() and never
reuse a channel across fork(), which is what the test suite does. Set it
in conftest.py before pytest's collection imports grpc.
[1] https://github.com/grpc/grpc/blob/master/doc/fork_support.md
Signed-off-by: Bastian Krause <bst@pengutronix.de>1 parent 04bb2b6 commit 2cf952d
1 file changed
Lines changed: 10 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
| 2 | + | |
2 | 3 | | |
3 | 4 | | |
4 | 5 | | |
| |||
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
16 | 26 | | |
17 | 27 | | |
18 | 28 | | |
| |||
0 commit comments