ci: Bump pyarrow version in CI to 19.0.1 (#4301)

## Changes Made

Arrow 19.0.0 is unable to read parquet files written by newer versions
of arrow with statistics.

The issue arises from an optimization where level histograms in
statistics (e.g. the level histogram used for repetition levels) is
omitted when the max level is 0.

For more details, see apache/arrow#45283

At the same time we can't bump to pyarrow 20.0.0 because this is
incompatible with ray, see
ray-project/ray#52685

This means we should use the minor release of arrow 19, i.e. arrow
19.0.1

Author: desmondcheongzx

.github/workflows/pr-test-suite.yml

@@ -22,7 +22,7 @@ jobs:
       matrix:
         python-version: ['3.9', '3.10']
         daft-runner: [py, ray, native]
-        pyarrow-version: [8.0.0, 19.0.0]
+        pyarrow-version: [8.0.0, 19.0.1]
         enable-aqe: [1, 0]
         os: [ubuntu-latest, macos-latest]
         exclude:

benchmarking/parquet/benchmark-requirements.txt

@@ -1,5 +1,5 @@
 pytest==7.4.0
 pytest-benchmark==4.0.0
 pytest-memray==1.4.1
-pyarrow==19.0.0
+pyarrow==19.0.1
 boto3==1.28.3

requirements-dev.txt

@@ -40,7 +40,10 @@ duckdb==1.1.2
 tqdm
 
 # Pyarrow
-pyarrow==19.0.0
+# Pyarrow 19.0.0 has a forwards compatibility bug so we use pyarrow 19.0.1 instead.
+# Pyarrow 20.0.0 is incompatible with Ray hence we can't use newer versions for now.
+# See: https://github.com/Eventual-Inc/Daft/pull/4301
+pyarrow==19.0.1
 # Ray
 ray[data, client]==2.34.0
 

tests/integration/io/docker-compose/retry_server/retry-server-requirements.txt

@@ -17,7 +17,7 @@ uvicorn==0.23.2
 uvloop==0.17.0
 watchfiles==0.19.0
 websockets==11.0.3
-pyarrow==19.0.0
+pyarrow==19.0.1
 slowapi==0.1.8
 
 # Pin numpy version otherwise pyarrow doesn't work