TypeError: type 'ExternalMerger' is not subscriptable

[soxofaan]

the python3.11 jenkins build of geopyspark-driver unit tests started to fail since july 14th with weird error

e.g. on tests.test_download.TestDownload.test_download_json_no_args

py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3537.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3537.0 (TID 5651) (8aae77fa03fd executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py", line 2044, in main
    process()
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py", line 2034, in process
    out_iter = func(split_index, iterator)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/jenkins/workspace/opyspark-driver-python311_master/venv311/lib64/python3.11/site-packages/pyspark/core/rdd.py", line 5290, in pipeline_func
    return func(split, prev_func(split, iterator))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/jenkins/workspace/opyspark-driver-python311_master/venv311/lib64/python3.11/site-packages/pyspark/core/rdd.py", line 5290, in pipeline_func
    return func(split, prev_func(split, iterator))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/jenkins/workspace/opyspark-driver-python311_master/venv311/lib64/python3.11/site-packages/pyspark/core/rdd.py", line 703, in func
    return f(iterator)
           ^^^^^^^^^^^
  File "/var/lib/jenkins/workspace/opyspark-driver-python311_master/venv311/lib64/python3.11/site-packages/pyspark/core/rdd.py", line 4036, in combine
    merger = ExternalMerger[K, V, list[V]](agg, memory * 0.9, serializer)
             ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
TypeError: type 'ExternalMerger' is not subscriptable

	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:581)
	at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:940)

this seems to happen since release 4.2.0 of pyspark (two days agao), which is seems to be the relevant change:

• build 751 has pyspark==4.1.2 and works
• build 752 has pyspark==4.2.0 and fails

[soxofaan]

still, I can not reproduce with pyspakr 4.2.0 on python 3.11:

❯ cat pyspark-ExternalMerger.py 
from typing import TypeVar, Hashable
from pyspark.shuffle import ExternalMerger

K = TypeVar("K", bound=Hashable)
V = TypeVar("V")
U = TypeVar("U")

x = ExternalMerger[K, V, U]

print(f"Works fine: {x}")

❯ uv run --python=3.11 --with pyspark==4.2.0 pyspark-ExternalMerger.py
Works fine: pyspark.shuffle.ExternalMerger[~K, ~V, ~U]

[soxofaan]

hypothesis: we're probably mixing two installs of pyspark in the environment:

• one baked into the docker image, not totally sure, but this suggests version 4.0.1
• one installed from jenkins pipeline, based on setup.py which is unbounded pyspark>=4.0.0; at the moment, so will resolve to 4.2.0

mixing that leads to that kind of weird inconsistency issues

[soxofaan]

45ca989: initial attempt to pin down the second pyspark to 4.0.x in this jenkins run