Skip to content

v2.336.0 on macOS (arm64): Runner.Worker deadlocks in Process.Start — forked child never execs, spins at 100% CPU; every job hangs seconds after start #4570

Description

@vscolear

Describe the bug

After the self-hosted runner auto-updated 2.335.1 → 2.336.0 (2026-07-22), every job on our macOS runner hangs a few seconds after starting, during an early process spawn (checkout/bash steps). 6 out of 6 jobs since the update wedged; all jobs before the update passed. Rolling back to 2.335.1 (same machine, same OS) fixed it immediately.

Mechanism, observed live on a wedged job:

  • _diag/Worker_*.log ends with a Starting process: block that is never followed by
    Process started with process id N — the worker is stuck inside Process.Start.
  • ps shows a second Runner.Worker spawnclient ... process — a forked child that never
    reached exec
    — burning ~100% of a core (physical footprint ~2 MB), child of the real Worker.
  • sample (macOS profiler) on that child shows a single thread spinning in managed CoreCLR
    code
    (ThreadNative::KickOffThread → ManagedThreadBase::KickOff → ... into JIT-ed frames) —
    i.e. a managed thread continued running in the fork child, which is undefined post-fork state.
  • The parent Worker waits forever; the job shows the current step "in progress" indefinitely.
  • On job cancellation / 60-min timeout the spinning fork child survives, so the runner stays
    wedged (pegged core, listener busy) until the processes are killed manually. Reboot doesn't
    help — the next job hits the same spawn deadlock.

To Reproduce

  1. Self-hosted runner v2.336.0 on macOS 26.5 (arm64), installed as a launchd service (./svc.sh install).
  2. Trigger any workflow job with a few consecutive steps (checkout + bash steps).
  3. Within seconds, a step never starts its process; job hangs until timeout.

Hit rate on our machine: 6/6 jobs after updating; 0 occurrences on 2.335.1 before and after (rolled back via the on-disk bin.2.335.1/externals.2.335.1 + disableUpdate: true).

Expected behavior

Process spawn completes (or fails); no fork-child spin; cancellation kills the worker cleanly.

Runner Version and Platform

  • Runner: 2.336.0 (self-update from 2.335.1 on 2026-07-22)
  • OS: macOS Tahoe 26.5.2 (25F84), arm64 (Apple M5 Pro)
  • Service: launchd user agent via svc.sh

Job Log Output

[...] INFO ProcessInvokerWrapper] Finished process 21919 with exit code 0, and elapsed time 00:00:00.3804644.
[...] INFO ProcessInvokerWrapper] Starting process:
[...] INFO ProcessInvokerWrapper]   File name: '/bin/bash'
[...] INFO ProcessInvokerWrapper]   Arguments: '-e /Users/.../_work/_temp/<uuid>.sh'
[...] INFO ProcessInvokerWrapper]   Force kill process on cancellation: 'False'
<nothing further; only HostContext heartbeat lines every 10s>

sample output of the spinning fork child available on request (full stack).

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions