Fix Windows shutdown deadlock and leak with the Ruy backend

[timo9378]

Hi, thanks for CTranslate2. I ran into a Windows-only shutdown hang and tracked it down
to the Ruy backend, so here's a fix.

Creating and destroying a Translator (or Generator/Whisper) with the Ruy
backend on Windows hangs the process on shutdown. The culprit is the per-thread
ruy::Context in src/cpu/backend.cc:

ruy::Context *get_ruy_context() {
  static thread_local ruy::Context context;
  return &context;
}

It's constructed lazily on a worker thread the first time that thread runs a GEMM,
and its destructor joins Ruy's internal thread pool. Because it's thread_local, that
destructor runs while the worker thread is exiting, under the Windows loader lock, and
the join deadlocks. ThreadPool::~ThreadPool in src/thread_pool.cc then blocks
forever in worker->join().

Sequence when a model is destroyed:

1. ~ReplicaPool → ~ThreadPool closes the queue and calls worker->join()
2. the worker's run() loop returns and the thread begins to terminate
3. the thread's thread_local destructors run, so ~ruy::Context joins the Ruy threads
4. that join, on a thread that is already exiting, deadlocks

This only happens on CPU. On CUDA the GEMM never calls get_ruy_context(), so the
worker thread has nothing to join and exits cleanly. That matches the reports that GPU
builds don't hang while CPU builds do.

The fix is to destroy the context from ReplicaWorker::finalize() instead. finalize()
runs on the worker thread inside run(), before the thread exits, so the join happens
in a normal context (no loader lock) and completes. It already calls
destroy_context(device) for this kind of per-thread cleanup (freeing curand state on
CUDA today), so the Ruy cleanup goes in the same place:

• backend.cc: make the context a resettable heap pointer and add clear_ruy_context()
• backend.h: declare it
• devices.cc: call cpu::clear_ruy_context() from destroy_context() for Device::CPU

Everything stays under the existing CT2_WITH_RUY / CT2_WITH_CUDA guards, so other
backends and platforms are untouched.

Besides fixing the hang, this frees the context instead of leaking it. RSS after each
of 5 load/translate/unload cycles (Windows, MSVC 14.44, x64, int8 NLLB-200 600M, Ruy):

	cycle 1..5 (MB)
bypassing the model's destructor (workaround used downstream)	885, 1755, 2624, 3496, 4371
heap-leaking just the ruy::Context	268, 520, 772, 1024, 1276
this PR	16, 17, 18, 18, 19

The ~250 MB/cycle in the middle row is Ruy's prepacked cache, which the leak keeps
alive; finalize-clear releases it. Translation output is unchanged and identical across
repeated calls, so rebuilding the context after clearing it is fine.

Found via jkawamoto/ctranslate2-rs#64, where the hang was worked around by skipping the
model's destructor entirely (avoiding the hang but leaking the model). With this change
the destructor can run.

Happy to adjust the approach if you'd prefer the cleanup somewhere else. Thanks!

[timo9378]

Hi! Any updates on this? I'd love to get this merged when you have a chance to review. Thanks!
@jordimas

[timo9378]

Thanks, that makes sense. Scoping it to Windows keeps the risk off Linux and macOS, so I pushed an update.

One thing I ran into while doing it. Guarding just the clear_ruy_context() call in devices.cc isn't quite enough on its own. If backend.cc still hands out a heap allocated context on every platform but the cleanup only compiles on Windows, then on Linux and macOS the context gets new'd and never freed, since the raw thread_local pointer doesn't destroy anything, so it would turn today's clean RAII into a leak. So I put the _WIN32 split in backend.cc too. Non Windows keeps the original static thread_local ruy::Context, and only Windows uses the heap pointer with the finalize() cleanup. The header declares clear_ruy_context() under _WIN32 only, and devices.cc guards the call with defined(CT2_WITH_RUY) && defined(_WIN32) like you suggested. Off Windows it is a real no op versus what ships today.

For the test I added test_shutdown_does_not_deadlock in test_translator.py. It builds a CPU int8 Translator, runs a translation, and destroys it, all inside subprocess.run with a timeout so a hang fails cleanly instead of blocking the runner. It is marked skipif not win32 since the hang is Windows only.

One detail that is easy to miss and worth spelling out. The test only reproduces if the batch is large enough that Ruy actually spawns its thread pool. With a single short sentence Ruy stays single threaded, there are no threads to join at shutdown, and the process exits cleanly even without the fix, so a naive test passes whether or not the bug is present. The test uses a 512 sentence batch so the pool is really up. I checked it both ways against a from source Ruy build on Windows: unpatched it times out and fails, with this PR it passes in well under a second.

I also confirmed this is not specific to the static CRT. A shared /MD build, same config as the released wheels, hangs on shutdown too once the batch is big enough, so the wheels are affected as well. I dropped a small standalone C++ repro under tools/ruy_shutdown_repro that shows this outside the test suite: same source built once with /MT and once with /MD, both hang unpatched and both exit cleanly with this PR. Happy to move or drop it if you would rather not carry it in the tree.