[Python] Interpreter hangs at exit after `pq.read_table(python_file_object)` - thread pool teardown deadlock

[imanhabib]

Describe the bug, including details regarding any error messages, version, and platform.

Describe the bug, including details regarding any error messages, version, and platform.

Reading a parquet file by passing a Python file object to pyarrow.parquet.read_table
makes the interpreter hang forever at shutdown on my and other similar machines:

# repro.py - hangs at exit, every run
import pyarrow.parquet as pq

with open("any.parquet", "rb") as f:
    pq.read_table(f)
print("done, exiting...")  # prints - then the process never exits

$ python repro.py
done, exiting...
# hangs forever; SIGKILL required

Passing a path instead of a file object does not hang:

pq.read_table("any.parquet")   # exits cleanly, 8/8 runs

use_threads=False also avoids the hang even with a file object (8/8 clean), as does
pa.set_cpu_count(1), so this looks like a CPU thread-pool teardown deadlock that is
only triggered when the read went through a Python file handle (GIL interaction during
pool shutdown?).

How we found it

pandas.read_parquet (which internally opens the file and passes a handle to pyarrow)
was hanging intermittently (~50% of runs when reading 3 files, rarer with 1 file).
Bisecting pandas out of the equation led to the deterministic file-object reproducer
above. So this same bug likely explains "flaky CI hang" reports from pandas users.

Details

• The script body completes and atexit handlers run; the hang is after that,
  during interpreter/C-level finalization. faulthandler (SIGABRT) produces no dump.
• The file content doesn't matter - reproduces with a 5-row single-column file.
• Closing the file object (or not) before exit makes no difference.
• A hung process shows the main thread plus ~24 CPU-pool threads all in
  futex_do_wait (from /proc/<pid>/task/*/wchan), and a jemalloc_bg_thd:

python(futex_do_wait) jemalloc_bg_thd(futex_do_wait) python(futex_do_wait) x24 ...

Version matrix (same reproducer, 4 runs each; 124 = hung/timeout, 0 = clean exit)

pyarrow	result
25.0.0	124 124 124 124
24.0.0	124 124 124 124
23.0.1	124 124 124 124
22.0.0	124 0 124 124
25.0.0 + use_threads=False	0 0 0 0
25.0.0 + path instead of file object	0 0 0 0

(22.0.0 being intermittent rather than deterministic suggests a timing-dependent race
that became reliably lost at some point.)

Second interpreter build

Also reproduces on the distro CPython (Python 3.14.4, GCC 15.2.0 build,
/usr/bin/python3.14, fresh venv, pyarrow 25.0.0 wheel) - intermittent there
(3/6 runs hang) instead of deterministic, path-based reads clean 6/6. So this is
not specific to one interpreter build; build/timing differences only shift how
often the race is lost.

Environment

• pyarrow 25.0.0 (manylinux wheel from PyPI, installed via uv)
• Python 3.13.13 (python-build-standalone, Clang 22.1.3) - deterministic hang
• Python 3.14.4 (distro build, GCC 15.2.0) - intermittent hang (~50%)
• Linux 7.0.0-22-generic x86_64, glibc 2.43
• 24 CPUs
• pandas 3.0.3 (only for the original discovery path; not needed for the reproducer)
• Persists after OS upgrade + reboot (kernel 7.0.0-22 → 7.0.0-27, glibc 2.43-2ubuntu2)

Component(s)

Python

Component(s)

Python

[raulcd]

Thank you very much for creating the issue.
I am not sure I understand the Version matrix section above, can you help me understand clearly whether this:

• Is a new issue given the latest 25.0.0 release
• Has been happening since 22.0.0

Is this only happening with a specific parquet file or is it reproducible with a minimal generated parquet file?