Skip to content

1.12.5+: worker/thread reboot (opcache_reset, watcher, workers/restart) deadlocks or corrupts the heap with pecl_http loaded #2568

Description

@Arkounay

What happened?

(disclaimer: I have used claude to help me making the ticket)

Since v1.12.5, restarting workers no longer resets opcache. It reboots every PHP thread and the PHP main thread, which makes the main thread run php_module_shutdown() / sapi_shutdown() / tsrm_shutdown() and then boot a brand new engine inside the same process.

With pecl_http loaded, that reboot never succeeds once. It ends in one of two ways, seemingly at random:

  • Mode A, permanent deadlock. All PHP threads stay in rebooting (force), the 6 s grace period expires, force-killing thread on reboot timeout is logged once per thread, and rebootAllThreads() blocks forever on rebootWg.Wait(). The process stays alive and the Caddy admin endpoint keeps answering 200, so Docker keeps reporting the container as healthy while every HTTP request hangs forever (no timeout, no 502).
  • Mode B, heap corruption. glibc aborts with double free or corruption (fasttop), corrupted size vs. prev_size in fastbins or malloc_consolidate(): invalid chunk size, and the process dies with SIGSEGV/SIGABRT (exit 139).

thread reboot finished is never logged in either mode.

v1.12.4 is unaffected. v1.12.5 and v1.12.6 failed 18/18 attempts (13x mode B, 5x mode A).

Three independent triggers reach the same code path, so this is not watcher-specific and not worker-mode-specific:

Trigger Code path
opcache_reset() from PHP, and opcache's own scheduled restart (SHM exhausted / max wasted memory) frankenphp.c:1712 zend_accel_schedule_restart_hook -> go_schedule_opcache_reset -> frankenphp.go:786 -> rebootAllThreads()
POST /frankenphp/workers/restart (admin API) caddy/admin.go:37 -> RestartWorkers() -> rebootAllThreads()
File watcher (worker { watch … }) watcher.go:36 -> RestartWorkers() -> rebootAllThreads()

The first one means a production server with opcache enabled can hit this with no watcher and no admin API involved.

Steps to reproduce

Three files. No framework, no load, no watcher needed.

Dockerfile

ARG FP_VERSION=1.12.6
FROM dunglas/frankenphp:${FP_VERSION}-php8.5

RUN install-php-extensions http

Caddyfile

{
	auto_https off
	admin :2019
	skip_install_trust

	frankenphp {
		max_threads auto

		worker {
			file /app/public/index.php
		}
	}
}

:80 {
	root /app/public
	rewrite * /index.php
	php /index.php
}

public/index.php

<?php

$handler = static function (): void {
    echo "ok\n";
};

while (frankenphp_handle_request($handler)) {
    gc_collect_cycles();
}

Run:

$ docker build -t fp-http:1.12.6 .
$ docker run -d --name fp -v "$PWD:/app" -w /app fp-http:1.12.6 \
    frankenphp run --config /app/Caddyfile
$ sleep 6

# one request, then one worker restart
$ docker exec fp curl -s http://localhost/
ok
$ docker exec fp curl -s -X POST http://localhost:2019/frankenphp/workers/restart
workers restarted successfully

# the server never serves another request
$ docker exec fp curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://localhost/
000

Rebuild with --build-arg FP_VERSION=1.12.4 and the same script keeps answering 200.

The single warm-up request matters: the reboot only breaks after the worker script has reached frankenphp_handle_request() at least once.

Three variants of the same failure, all verified:

  • Watcher instead of admin API. Add watch /app/**/*.php to the worker block, then after the warm-up request run echo '<?php // touched' > watched-trigger.php. Log: filesystem changes detected -> rebooting all PHP threads -> double free or corruption -> exit 139.
  • opcache_reset() from the worker script itself. A worker script that calls opcache_reset() on one route: the response is delivered (reset ok), then rebooting all PHP threads -> double free or corruption -> exit 139.
  • No worker mode at all. Replace the whole worker { … } block with plain php_server, then request a script containing var_dump(opcache_reset());. It prints bool(true), logs rebooting all PHP threads, and the server either dies with malloc_consolidate(): invalid chunk size (exit 139) or hangs forever with no further log line at all, not even a reboot timeout warning, since there are no worker threads to force-kill.

Results

Same reproducer, only the base image changes:

Image Version Outcome rebooting all PHP threads thread reboot finished force-killing thread… glibc corruption HTTP after restart
fp-http v1.12.4 works 0 (no reboot at all) n/a 0 0 200
fp-http v1.12.5 mode A or B 1 0 40 (mode A) 2 (mode B) dead
fp-http v1.12.6 mode A or B 1 0 40 (mode A) 2 (mode B) dead
dunglas/frankenphp:1.12.6-php8.5 (no pecl_http) v1.12.6 works 1 1 0 0 200

Failure rate on 1.12.5/1.12.6 with pecl_http: 18/18 runs (13x mode B, 5x mode A), across the 6 line worker script above, an allocation-heavy variant of it, an idle server and 30 concurrent requests, worker and non-worker mode, and with thread autoscaling on and off. Without pecl_http the same setup survived 16 to 21 consecutive restarts under load, so the reboot machinery itself is fine until that extension is loaded.

What was ruled out

Same trivial worker script, same image, only the enabled extension .ini files change (/usr/local/etc/php/conf.d), 16+ workers/restart calls under 30 concurrent requests each:

Extensions enabled Result
none works, 16/16 reboots finished
apcu gd imagick intl pdo_mysql works, 16/16
redis sodium zip works, 16/16
xdebug only works, 17/17
raphf only works, 16/16
all 11 except xdebug dies on reboot #1
raphf + http dies on reboot #1

So: not thread autoscaling, not Xdebug, not Imagick, not the framework (6 lines of PHP are enough), not concurrency, not the watcher, not worker mode. raphf alone is fine; http on top of it breaks every reboot.

Evidence, mode A (deadlock)

GET /debug/pprof/goroutine?debug=2 on the wedged process (140 goroutines, 40 worker threads):

goroutine 88 [sync.WaitGroup.Wait]:
sync.(*WaitGroup).Wait(0xe7ca0a33a20)
github.com/dunglas/frankenphp.(*phpMainThread).rebootAllThreads(0x0?)
	/go/src/app/phpmainthread.go:183 +0x396
github.com/dunglas/frankenphp.RestartWorkers(...)
	/go/src/app/worker.go:176
github.com/dunglas/frankenphp.initWatchers.func2({0x3af37f0?, 0x1?, 0x1?})
	/go/src/app/watcher.go:36 +0x2f
github.com/dunglas/frankenphp/internal/watcher.(*globalWatcher).scheduleReload(…)
	/go/src/app/internal/watcher/watcher.go:212 +0x83

x40, the per-thread reboot waiters, blocked on the unbounded WaitFor right after sendKillSignal():

goroutine 435 [chan receive]:
github.com/dunglas/frankenphp/internal/state.(*ThreadState).WaitFor(0xe7c9ff78dc0, {0xe7c9ffa0460, 0x1, 0x1})
	/go/src/app/internal/state/state.go:167 +0x158
github.com/dunglas/frankenphp.(*phpMainThread).rebootAllThreads.func1()
	/go/src/app/phpmainthread.go:179 +0x2a5

x41, forceReboot goroutines waiting for RebootReady, which never comes:

goroutine 218 [chan receive]:
github.com/dunglas/frankenphp/internal/state.(*ThreadState).WaitFor(0xe7c9ff78e00, {0xe7ca0682cc8, 0x1, 0x1})
	/go/src/app/internal/state/state.go:167 +0x158
github.com/dunglas/frankenphp.(*phpThread).forceReboot.func1()
	/go/src/app/phpthread.go:106 +0x4a

Both autoscaling goroutines are blocked behind scalingMu, which rebootAllThreads holds (phpmainthread.go:142):

goroutine 89 [sync.Mutex.Lock]:
sync.(*RWMutex).Lock(0xe7ca0b87f10?)
github.com/dunglas/frankenphp.startUpscalingThreads(0x15, …)
	/go/src/app/scaling.go:143 +0x49

goroutine 90 [sync.Mutex.Lock]:
github.com/dunglas/frankenphp.deactivateThreads()
	/go/src/app/scaling.go:206 +0x38

The watcher is wedged too, its C callback can no longer hand off events:

goroutine 203 [chan send, locked to thread]:
github.com/dunglas/frankenphp/internal/watcher.(*pattern).handle(0xe7c9fede050, 0xe7ca00283c0)
	/go/src/app/internal/watcher/pattern.go:104 +0x9f
github.com/e-dant/watcher/watcher-go.go_callback(…)

In-flight requests hang with no timeout at all, which is why clients see a hang rather than an error:

goroutine 366 [select]:
github.com/dunglas/frankenphp.(*worker).handleRequest(...)
	/go/src/app/worker.go:251
github.com/dunglas/frankenphp.ServeHTTP({0x2525950, 0xe7ca07c8ab0?}, 0xe7c9fc2b728)
	/go/src/app/frankenphp.go:423 +0x9c5

Notably zero goroutines are parked in workerThread.waitForWorkerRequest, while 41 are [syscall, locked to thread]: every PHP thread left the Go select after close(drainChan) and is now stuck inside C, so sendKillSignal() cannot help (the Zend VM is not executing opcodes) and state.WaitFor(YieldingForReboot) at phpmainthread.go:179 never returns.

Two side effects worth noting:

  • The Caddy admin endpoint stays fully responsive (/metrics -> 200) while the app serves nothing, so HEALTHCHECK CMD curl -f http://localhost:2019/metrics (the health check shipped in the Symfony Docker recipe) keeps the container healthy indefinitely. Verified: docker ps -> Up About a minute (healthy) while every app request returned 000.
  • Later POST /frankenphp/workers/restart calls still answer workers restarted successfully, because isRebooting.CompareAndSwap at phpmainthread.go:134 short-circuits. The API reports success on a permanently wedged server.

Evidence, mode B (heap corruption)

The corruption starts immediately after rebooting all PHP threads, with no force-killing warning in between, so the force-kill signal is not involved in this mode.

Backtrace of the first aborting thread (gdb with libc6-dbg; the shipped frankenphp binary and libphp.so are stripped, hence ?? for their frames). This is a worker thread that has already returned from its start routine and is aborting in glibc's per-thread malloc teardown:

#5  malloc_printerr (str="double free or corruption (fasttop)") at ./malloc/malloc.c:5832
#6  _int_free_chunk (av=0x70625c000030, p=0x70625c455770, …) at ./malloc/malloc.c:4628
#7  _int_free (av=0x70625c000030, p=0x70625c455770, have_lock=0) at ./malloc/malloc.c:4699
#8  __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3476
#9  tcache_thread_shutdown () at ./malloc/malloc.c:3299
#10 __malloc_arena_thread_freeres () at ./malloc/arena.c:897
#11 __libc_thread_freeres () at ./malloc/thread-freeres.c:41
#12 start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:459
#13 clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Re-running with tcache disabled (GLIBC_TUNABLES=glibc.malloc.tcache_count=0) moves the abort to the allocation that trips over the corrupted chunk, and that one lands squarely in the main thread's engine teardown (thread name php-main):

#5  malloc_printerr (str="corrupted size vs. prev_size in fastbins") at ./malloc/malloc.c:5832
#6  malloc_consolidate (av=0x7b2ec8000030) at ./malloc/malloc.c:4924
#7  _int_malloc (av=0x7b2ec8000030, bytes=1136) at ./malloc/malloc.c:4099
#8  _int_memalign (av=0x7b2ec8000030, alignment=64, bytes=1024) at ./malloc/malloc.c:5129
#9  _mid_memalign (alignment=64, bytes=1024, …) at ./malloc/malloc.c:3724
#10 __posix_memalign (memptr=0x7b2ecfffe7b0, alignment=64, size=1024) at ./malloc/malloc.c:5858
#12 CRYPTO_aligned_alloc () from /lib/x86_64-linux-gnu/libcrypto.so.3
#18 OSSL_LIB_CTX_free () from /lib/x86_64-linux-gnu/libcrypto.so.3
#19 ?? () from /usr/local/lib/libphp.so
#20 ts_free_id () from /usr/local/lib/libphp.so
#22 zend_hash_graceful_reverse_destroy () from /usr/local/lib/libphp.so
#24 php_module_shutdown () from /usr/local/lib/libphp.so
#25 php_module_shutdown_wrapper () from /usr/local/lib/libphp.so
#27 start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:448

So the main thread is running a full php_module_shutdown() as part of a reboot, not a process shutdown.

Analysis (hypothesis)

What changed between 1.12.4 and 1.12.5 (commit 39b9692, PR #2364 "fix: reboot threads on opcache reset and watcher"):

  • 1.12.4 RestartWorkers() (worker.go:239) drained the worker threads and set them back to Ready. The C threads and the engine were never torn down.
  • 1.12.5+ RestartWorkers() -> mainThread.rebootAllThreads() (phpmainthread.go:133): every PHP C thread is force-rebooted (each runs ts_free_thread() and exits its pthread), then the main thread is set to Rebooting, which makes go_frankenphp_main_thread_is_ready() return (phpmainthread.go:238) and php_main() fall through to what is otherwise the process shutdown sequence (frankenphp.c:1727-1732):
  go_frankenphp_main_thread_is_ready();

  /* channel closed, shutdown gracefully. drainPHPThreads has already
   * waited for every PHP thread to exit (state.Done), so SAPI/TSRM
   * teardown here is safe. */
  frankenphp_sapi_module.shutdown(&frankenphp_sapi_module);
  sapi_shutdown();
#ifdef ZTS
  tsrm_shutdown();
#endif

The invariant asserted by that comment holds for a real shutdown but not for a reboot: on the reboot path the threads are in YieldingForReboot, not Done. That state is published by go_frankenphp_on_thread_shutdown() (phpthread.go:283-295), which C calls at frankenphp.c:1612, before return NULL and therefore before glibc has run __libc_thread_freeres / tcache_thread_shutdown for that pthread.

So as soon as the last thread reports YieldingForReboot, the Go side proceeds (phpmainthread.go:185-190) and the main thread starts php_module_shutdown() -> ts_free_id() -> tsrm_shutdown(), freeing per-thread and module-global storage, while the worker pthreads are still finishing their own teardown in glibc. That matches both backtraces exactly: one crash inside a worker thread's tcache_thread_shutdown, one inside the main thread's ts_free_id -> OSSL_LIB_CTX_free.

Mode A looks like the same race landing the other way: a thread's teardown never completes, it never reaches YieldingForReboot, sendKillSignal() is a no-op because the thread is not in the Zend VM, and the unbounded state.WaitFor(state.YieldingForReboot) at phpmainthread.go:179, hence rebootWg.Wait() at phpmainthread.go:183, blocks forever while scalingMu stays locked.

Why pecl_http makes it deterministic is not established here. It plausibly has per-thread or persistent-handle globals (via raphf) plus libcurl/OpenSSL state whose destruction order matters, but the observed corruption is inside php_module_shutdown itself, so the extension may only be widening an existing race window rather than being the sole offender. raphf alone is safe; it only breaks with http on top. And the same extension versions are stable on 1.12.4, where no engine teardown happens.

Independent of the root cause, two smaller things stand out:

  1. state.WaitFor(state.YieldingForReboot) after sendKillSignal() (phpmainthread.go:179) is unbounded, so a thread that cannot yield hangs the whole server silently and permanently. A bounded wait plus a hard failure (or a deliberate abort) would at least let an orchestrator restart the container.
  2. POST /frankenphp/workers/restart returns workers restarted successfully when a reboot is already in progress (phpmainthread.go:134), even if that reboot has been stuck for minutes.

Impact

  • Development with the watcher is unusable on 1.12.5/1.12.6. In a Symfony app the first request warms the dev cache and writes thousands of .php files inside the watched tree, which triggers the reboot immediately. The app dies or wedges within seconds of docker compose up, and because the admin endpoint stays up the container is still reported healthy.
  • Production is exposed with no watcher: opcache's scheduled restart hook (frankenphp.c:1712) and any userland opcache_reset() both call rebootAllThreads(), in worker and non-worker mode alike.
  • Mode A is the nastier failure: no crash, no log after the force-kill warnings, health check green, requests hanging forever.

Workaround

Pin to 1.12.4:

FROM dunglas/frankenphp:1.12.4-php8.5

Removing pecl_http also avoids it in our tests, but that is not an option for us.

Build Type

Docker (Debian Trixie)

Worker Mode

No

Operating System

GNU/Linux

CPU Architecture

x86_64

PHP configuration

phpinfo() output
# Reproducer image: dunglas/frankenphp:1.12.6-php8.5 + `install-php-extensions http`.
# Nothing else is configured, there is no php.ini: "Loaded Configuration File => (none)".

# phpinfo() as served by FrankenPHP:
PHP Version => 8.5.8
Server API => FrankenPHP
Thread Safety => enabled
Opcode Caching => Up and Running

# php -i:
PHP Version => 8.5.8
System => Linux 6.8.0-107-generic #107-Ubuntu SMP PREEMPT_DYNAMIC x86_64
Build Date => Jul 14 2026 01:28:01
Build System => Linux - Docker
Build Provider => https://github.com/docker-library/php
Configure Command =>  './configure' '--build=x86_64-linux-gnu' '--sysconfdir=/usr/local/etc'
 '--with-config-file-path=/usr/local/etc/php' '--with-config-file-scan-dir=/usr/local/etc/php/conf.d'
 '--enable-option-checking=fatal' '--with-mhash' '--with-pic' '--enable-mbstring' '--enable-mysqlnd'
 '--with-password-argon2' '--with-sodium=shared' '--with-pdo-sqlite=/usr' '--with-sqlite3=/usr'
 '--with-curl' '--with-iconv' '--with-openssl' '--with-readline' '--with-zlib' '--enable-phpdbg'
 '--enable-phpdbg-readline' '--with-pear' '--with-libdir=lib/x86_64-linux-gnu' '--enable-embed'
 '--enable-zts' '--disable-zend-signals' 'build_alias=x86_64-linux-gnu' 'PHP_UNAME=Linux - Docker'
Configuration File (php.ini) Path => /usr/local/etc/php
Loaded Configuration File => (none)
Scan this dir for additional .ini files => /usr/local/etc/php/conf.d
Additional .ini files parsed => /usr/local/etc/php/conf.d/docker-php-ext-raphf.ini,
/usr/local/etc/php/conf.d/docker-php-ext-sodium.ini,
/usr/local/etc/php/conf.d/xx-php-ext-http.ini
PHP API => 20250925
PHP Extension => 20250925
Zend Extension => 420250925
Zend Extension Build => API420250925,TS
PHP Extension Build => API20250925,TS
PHP Integer Size => 64 bits
Debug Build => no
Thread Safety => enabled
Zend Engine v4.5.8

[PHP Modules]
Core, ctype, curl, date, dom, fileinfo, filter, hash, http, iconv, json, lexbor, libxml,
mbstring, mysqlnd, openssl, pcre, PDO, pdo_sqlite, Phar, posix, random, raphf, readline,
Reflection, session, SimpleXML, sodium, SPL, sqlite3, standard, tokenizer, uri, xml,
xmlreader, xmlwriter, Zend OPcache, zlib

[Zend Modules]
Zend OPcache

# The two extensions that make the difference:
http    => Extension Version => 4.3.1
raphf   => 2.0.2
libcurl => 8.14.1 (SSL Version => OpenSSL/3.5.6, libidn2 => 2.3.8)
OpenSSL Library Version => OpenSSL 3.5.6 7 Apr 2026

# opcache (defaults, nothing overridden):
opcache.enable => On
opcache.enable_cli => Off
opcache.memory_consumption => 128
opcache.interned_strings_buffer => 8
opcache.max_wasted_percentage => 5
opcache.validate_timestamps => On
opcache.revalidate_freq => 2
opcache.jit => disable
opcache.preload => no value

# FrankenPHP:
FrankenPHP v1.12.6 PHP 8.5.8 Caddy v2.11.4
num_threads: 41, max_threads: 249 (max_threads auto), max_requests: 0
# also reproduced with num_threads 8 / worker num 4 and no max_threads

# Host: Linux 6.8.0-107-generic x86_64, 20 cores, Docker 29.4.0, glibc 2.41 in the image.
# The full 900 line phpinfo() adds nothing: the config is entirely determined by the
# 4 line Dockerfile above.

Relevant log output

Relevant log output
</details>

### Mode A, deadlock (worker mode, 40 worker threads). Nothing is ever logged after this.
2026/07/27 13:54:24.285  INFO  frankenphp  rebooting all PHP threads
2026/07/27 13:54:30.285  WARN  frankenphp  force-killing thread on reboot timeout  {"name": "Worker PHP Thread - /app/public/index.php", "state": "rebooting (force)", "timeout": "6s"}
2026/07/27 13:54:30.285  WARN  frankenphp  force-killing thread on reboot timeout  {"name": "Worker PHP Thread - /app/public/index.php", "state": "rebooting (force)", "timeout": "6s"}
... 40 times, once per thread ...
# "thread reboot finished" never appears; the admin endpoint keeps answering 200,
# every application request hangs forever.

### Mode B, heap corruption (worker mode, admin API trigger). Process exits 139.
{"level":"info","ts":1785161269.177,"logger":"frankenphp","msg":"rebooting all PHP threads"}
double free or corruption (out)
double free or corruption (fasttop)
double free or corruption (fasttop)
... 7 to 11 threads ...

### Mode B via the watcher instead of the admin API.
{"level":"info","ts":1785163830.114,"logger":"watcher","msg":"filesystem changes detected","changes":[{"path_name":"/app/watched-trigger.php","effect_type":"create","path_type":"file"}]}
{"level":"info","ts":1785163830.121,"logger":"frankenphp","msg":"rebooting all PHP threads"}
double free or corruption (out)
double free or corruption (fasttop)

### Non-worker mode, opcache_reset() from a normal request. Process exits 139.
{"level":"info","ts":1785164089.395,"logger":"frankenphp","msg":"rebooting all PHP threads"}
malloc_consolidate(): invalid chunk size

### Non-worker mode again, this time the deadlock variant. opcache_reset() returns bool(true),
### then every request hangs. Note there is no force-kill line here: no worker threads exist,
### so nothing logs a reboot timeout at all, and the process just stops serving silently.
{"level":"info","ts":1785165351.782,"logger":"frankenphp","msg":"rebooting all PHP threads"}

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions