Skip to content

tests: disable gRPC fork handlers to fix flaky fork() abort#1905

Merged
Emantor merged 1 commit into
labgrid-project:masterfrom
Bastian-Krause:bst/grpc-fork
Jun 29, 2026
Merged

tests: disable gRPC fork handlers to fix flaky fork() abort#1905
Emantor merged 1 commit into
labgrid-project:masterfrom
Bastian-Krause:bst/grpc-fork

Conversation

@Bastian-Krause

Copy link
Copy Markdown
Member

Description
The test suite intermittently fails in CI due to (pexpect spawned) labgrid-client being killed during teardown by SIGABRT and gRPC C-core aborting after a fork:

fork_posix.cc:71] Other threads are currently calling into gRPC, skipping fork() handlers
ev_epoll1_linux.cc:1125] Check failed: next_worker->state == KICKED

gRPC documents that fork() is only safe if gRPC objects are created in the child after the fork [1]. The tests do the opposite: the pytest process spawns the labgrid-client commands via pexpect (using forkpty) while gRPC is already initialized in it (imported at collection, and used in-process by the coordinator tests). To partially support this, gRPC installs a pthread_atfork handler that tries to quiesce its poller across the fork. When another thread is in the C-core at fork time the handler loses that race, skips the reset, and the forked child aborts on the inconsistent poller before it can exec the client.

The problem could never be reproduced locally; it likely occurs more often in CI because scheduling pressure makes the race more likely to trigger.

Set GRPC_ENABLE_FORK_SUPPORT=0 to disable the handler. This is the documented remedy for callers that always exec() after fork() and never reuse a channel across fork(), which is what the test suite does. Set it in conftest.py before pytest's collection imports grpc.

[1] https://github.com/grpc/grpc/blob/master/doc/fork_support.md

Checklist

  • PR has been tested

The suite intermittently fails in CI due to (pexpect spawned)
labgrid-client being killed during teardown by SIGABRT and gRPC C-core
aborting after a fork:

  fork_posix.cc:71] Other threads are currently calling into gRPC, skipping fork() handlers
  ev_epoll1_linux.cc:1125] Check failed: next_worker->state == KICKED

gRPC documents that fork() is only safe if gRPC objects are created in
the child after the fork [1]. The tests do the opposite: the pytest
process spawns the labgrid-client commands via pexpect (using forkpty)
while gRPC is already initialized in it (imported at collection, and used
in-process by the coordinator tests). To partially support this, gRPC
installs a pthread_atfork handler that tries to quiesce its poller across
the fork. When another thread is in the C-core at fork time the handler
loses that race, skips the reset, and the forked child aborts on the
inconsistent poller before it can exec the client.

The problem could never be reproduced locally; it likely occurs more
often in CI because scheduling pressure makes the race more likely to
trigger.

Set GRPC_ENABLE_FORK_SUPPORT=0 to disable the handler. This is the
documented remedy for callers that always exec() after fork() and never
reuse a channel across fork(), which is what the test suite does. Set it
in conftest.py before pytest's collection imports grpc.

[1] https://github.com/grpc/grpc/blob/master/doc/fork_support.md

Signed-off-by: Bastian Krause <bst@pengutronix.de>
@codecov

codecov Bot commented Jun 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 45.9%. Comparing base (04bb2b6) to head (2cf952d).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@          Coverage Diff           @@
##           master   #1905   +/-   ##
======================================
  Coverage    45.9%   45.9%           
======================================
  Files         180     180           
  Lines       14500   14500           
======================================
  Hits         6666    6666           
  Misses       7834    7834           
Flag Coverage Δ
3.10 45.9% <ø> (ø)
3.11 45.9% <ø> (ø)
3.12 45.9% <ø> (ø)
3.13 45.9% <ø> (ø)
3.14 45.9% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@Emantor
Emantor merged commit 6b40400 into labgrid-project:master Jun 29, 2026
11 checks passed
@Emantor

Emantor commented Jun 29, 2026

Copy link
Copy Markdown
Member

I looked at this briefly on friday too when wondering about the test failures, thanks for the deep dive and fixing the tests 👍

@Bastian-Krause Bastian-Krause added the pick to stable Needs a pick to the latest stable branch label Jun 29, 2026
@Bastian-Krause

Copy link
Copy Markdown
Member Author

Although this is not really material for "stable", having a stable CI (pun intended) would be worth it :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pick to stable Needs a pick to the latest stable branch tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants