[Python][C++] 25.0.0 aarch64 wheel: deterministic dictionary-decode failure on Neoverse V1 / AWS Graviton3 (SIGSEGV via iter_batches; ArrowInvalid single-threaded); N1/V2 unaffected; 24.0.0 OK

[eaglenebulaplayer]

Describe the bug, including details regarding any error messages, version, and platform.

Summary: the pyarrow 25.0.0 manylinux aarch64 wheel deterministically fails to decode dictionary-encoded Parquet on AWS Graviton3 (Neoverse V1, CPU part 0xd40) hosts. The same files, same container image, same code path work correctly on Graviton2 (Neoverse N1, 0xd0c) and Graviton4 (Neoverse V2, 0xd4f), and everything works on all three CPU generations with pyarrow 24.0.0.

Symptoms on Graviton3 + 25.0.0:

• ParquetFile.iter_batches() with default use_threads=True → hard SIGSEGV (faulthandler shows the crash inside native decode).
• use_threads=False → no crash, but raises ArrowInvalid: Index not in dictionary bounds — i.e. single-threaded decode is also wrong, so this is not a threading race and use_threads=False is not a mitigation.

Determinism evidence (from an internal investigation; counts are per 8 identical scan tasks on a fixed corpus per arm, tasks landing on random ARM host generations on AWS Fargate):

• pyarrow 25.0.0, defaults: 3/8 tasks failed — exactly the tasks placed on 0xd40 hosts.
• pyarrow 25.0.0, use_threads=False: 4/8 failed with ArrowInvalid — again exactly the 0xd40 placements.
• Same base image with only pyarrow==24.0.0 swapped in: 8/8 green across all host generations.
• Within a failing task: re-decoding the exact same bytes twice fails identically; re-downloading the file (sha256-verified identical) fails identically; the same file decoded on N1/V2 hosts or on x86-64 is byte-for-byte correct. 12 probe tasks reading /proc/cpuinfo: failure correlates 100% with CPU part 0xd40.
• The failing wheel was installed from PyPI (pyarrow==25.0.0, cp312 manylinux aarch64) in a python:3.12-slim (Debian, glibc 2.41) container.

Files that trigger it: zstd-compressed Parquet written by earlier pyarrow (write_table, defaults → dictionary-encoded string columns), ~500k rows per file, ~27 columns (strings, int64, date32, bool, one timestamp). Unfortunately the data is proprietary financial content and we cannot attach a triggering file. We have not yet managed to isolate a synthetic reproducer (we currently have no pinned-Graviton3 test host — Fargate placement is random), but the failure is 100% deterministic per (file, 0xd40 host, 25.0.0).

Hypothesis: a Neoverse-V1-specific code path in the 25.0.0 aarch64 build — V1 is the only one of the three affected/unaffected cores with 256-bit SVE (N1 has no SVE, V2 is 128-bit SVE2), so an SVE(-width)-dispatched kernel in the dictionary decode path is our leading suspect. Related plumbing: #47769 (dynamic dispatch on ARM64) / xsimd bump #49921.

Repro shape (on a Graviton3 host, e.g. EC2 c7g/m7g/r7g, with any sufficiently large dictionary-encoded zstd Parquet):

import pyarrow.parquet as pq
pf = pq.ParquetFile(path)
for batch in pf.iter_batches(batch_size=5000):      # SIGSEGV on 0xd40 with 25.0.0
    pass
# use_threads=False instead → ArrowInvalid: Index not in dictionary bounds

We are happy to run candidate wheels/builds or additional diagnostics (faulthandler tracebacks, ARROW_USER_SIMD_LEVEL sweeps, etc.) on Graviton3 and report back — please tell us what would be most useful.

Component(s)

Python, C++

[conzy]

Also experiencing this on Fargate