Skip to content

Fix a crash caused by renaming a live worker thread from another thread - #4722

Open
saran2020 wants to merge 1 commit into
Kotlin:developfrom
saran2020:fix-worker-cross-thread-rename-1
Open

Fix a crash caused by renaming a live worker thread from another thread#4722
saran2020 wants to merge 1 commit into
Kotlin:developfrom
saran2020:fix-worker-cross-thread-rename-1

Conversation

@saran2020

Copy link
Copy Markdown

Problem

CoroutineScheduler.Worker.indexInArray's setter unconditionally renames the underlying Thread on every assignment:

var indexInArray = 0
    set(index) {
        name = "$schedulerName-worker-${if (index == 0) "TERMINATED" else index.toString()}"
        field = index
    }

Two of its three call sites are safe: renaming a Worker before start() (no live peer yet), and a worker renaming itself during self-termination. The third, in tryTerminateWorker(), renames a different, live worker as part of keeping the workers array dense after a pool shrink:

val lastWorker = workers[lastIndex]!!
workers.setSynchronized(oldIndex, lastWorker)
lastWorker.indexInArray = oldIndex // <- cross-thread rename of a live thread

On Android, renaming a live thread you don't own requires the ART runtime to suspend it first (Thread.setNameThread_setNativeNameSuspendThreadByPeer). If that thread doesn't reach a safepoint quickly enough — under GC pressure, device load, or anything else that delays it — the suspend attempt times out and ART aborts the whole process with SIGABRT. This matches the crash reports in #2234 exactly, including the CoroutineScheduler$Worker.tryTerminateWorker/setIndexInArray frames in the native backtrace.

I confirmed this is exploitable on-demand: forcing worker-pool churn while deliberately starving one worker (CPU priority + core-affinity pinning + blocking I/O + GC pressure) reproduces the exact crash reliably on an emulator, with the abort backtrace showing this precise call chain.

Fix

Only rename the thread synchronously when it's safe to do so — the thread hasn't started yet, or the calling thread is the thread being renamed:

var indexInArray = 0
    set(index) {
        field = index
        if (!isAlive || Thread.currentThread() === this) {
            name = "$schedulerName-worker-${if (index == 0) "TERMINATED" else index.toString()}"
        }
    }

When neither holds (the cross-thread case), the rename is deferred to the worker's own next loop iteration in runWorker(), which is always a same-thread — and therefore always safe — rename:

private var appliedNameIndex = indexInArray

private fun runWorker() {
    var rescanned = false
    while (!isTerminated && state != WorkerState.TERMINATED) {
        if (appliedNameIndex != indexInArray) {
            appliedNameIndex = indexInArray
            name = "$schedulerName-worker-${if (appliedNameIndex == 0) "TERMINATED" else appliedNameIndex.toString()}"
        }
        ...

The numeric indexInArray field still updates immediately and unconditionally in all cases, so scheduling/parking/work-stealing correctness is unaffected — only the display name can lag briefly after a pool shrink, until the affected worker next runs.

Also updated SchedulerTestBase.maxSequenceNumber() to read indexInArray directly instead of parsing it out of Thread.name, since the name can now be briefly stale right after a shrink event.

Drawbacks/trade-offs

  • Worker names can be briefly stale after a pool shrink. When a cross-thread rename is deferred, the affected worker keeps showing its old index in Thread.getName() — visible in thread dumps, jstack, Android Studio's profiler, ANR reports, or any logging that tags by thread name — until that worker next loops around in runWorker(). In practice, this window is bounded by whatever the worker is currently doing (its current task, or until it wakes from parking), so it's on the order of the same task-scheduling latency that's already normal for this pool, not an unbounded delay. This is a real, if minor, change in observability: previously, the name was always immediately accurate; now it's eventually accurate.

  • The crash itself isn't reproducible in a JVM unit test. The bug only manifests through ART's native thread-suspension machinery on Android, which doesn't exist in a plain JVM test environment — confirmed directly while working on this fix, since neither a JNI critical section nor CPU/IO starvation could force the equivalent failure outside of a real Android runtime. The added tests verify the actual safety property this fix relies on (a live thread is never renamed except by itself), not the crash/abort itself. If a future change reintroduces a similar cross-thread rename somewhere else in the scheduler, these tests won't catch it unless it goes through this exact code path.

Testing

  • Added WorkerRenameTest covering both live-thread branches: a cross-thread indexInArray change never renames synchronously (and the worker self-heals its name on its next run), and a worker renaming itself is applied synchronously.
  • Full existing scheduling test suite passes with no regressions.
  • Verified against the real crash: built a reproduction harness that reliably crashes on the unpatched code within a handful of attempts under the conditions described above, and confirmed 20/20 clean runs (repeated across multiple full passes) against this fix with no SIGABRT.

Fixes:
#2234

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant