Skip to content

[SPARK-60139][PYTHON] Remove dead code UNSUPPORTED_COMBINATIONS - #59343

Open
bearomorphism wants to merge 2 commits into
apache:masterfrom
bearomorphism:fix-unsupported-combinations-typing
Open

bearomorphism wants to merge 2 commits into
apache:masterfrom
bearomorphism:fix-unsupported-combinations-typing

Conversation

@bearomorphism

@bearomorphism bearomorphism commented Oct 11, 2026 •

Copy link
Copy Markdown

What changes were proposed in this pull request?

Remove the unused UNSUPPORTED_COMBINATIONS constant from python/pyspark/install.py, together with its import and dead loop in test_checked_versions.

This supersedes the original annotation-only approach. The existing supported-version checks and explicit test rejecting Hive 1.2 remain unchanged.

Why are the changes needed?

History investigation shows that this constant has been unused by production code since September 2020:

  • September 23, 2020: SPARK-32017, commit 942f577b6e3 introduced the constant to reject Hive 1.2 paired with either without-hadoop or Hadoop 3.2.
  • September 24, 2020: SPARK-32982, commit 688d016c7ac removed Hive 1.2 support, emptied the list, and deleted the production membership check in checked_versions. The empty declaration and test loop were left behind.
  • September 27, 2020: SPARK-32714, commit c65b64552f9 added # type: ignore to silence mypy's missing-annotation error for the empty list.

Before this cleanup, a repository-wide search found only the declaration, the test import, and the test loop. The loop executes zero iterations because the list is empty. Removing these leftovers is preferable to adding a type annotation to dead code.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

The following focused tests passed both before and after removal (2 tests):

source .venv/bin/activate
PYTHONPATH="$PWD/python:$PWD/python/lib/py4j-0.10.9.9-src.zip" python -m unittest \
  pyspark.tests.test_install_spark.SparkInstallationTestCase.test_checked_versions \
  pyspark.tests.test_install_spark.SparkInstallationTestCase.test_package_name

Additional checks passed:

  • Ruff lint and format checks on both changed files.
  • Standalone mypy checking of the full install.py source using python/mypy.ini, without importing the parent PySpark package.
  • git diff --check.
  • Search confirmed no remaining UNSUPPORTED_COMBINATIONS references under python.

The network-dependent installation/download test was not run. An earlier package-aware mypy invocation encountered 23 errors elsewhere in PySpark; this is not a claim that the full package type check passes.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: pi 1.1.0

@bearomorphism bearomorphism changed the title [MINOR][PYTHON] Annotate UNSUPPORTED_COMBINATIONS in pyspark/install.py [SPARK-60139][PYTHON] Annotate UNSUPPORTED_COMBINATIONS in pyspark/install.py Oct 11, 2026
@bearomorphism
bearomorphism marked this pull request as ready for review October 11, 2026 03:49
@bearomorphism
bearomorphism force-pushed the fix-unsupported-combinations-typing branch from 0d1aab7 to ed00a6f Compare October 11, 2026 03:52
@bearomorphism bearomorphism changed the title [SPARK-60139][PYTHON] Annotate UNSUPPORTED_COMBINATIONS in pyspark/install.py [SPARK-60139][PYTHON] Remove unused UNSUPPORTED_COMBINATIONS Oct 11, 2026
@bearomorphism
bearomorphism marked this pull request as draft October 11, 2026 04:02
@bearomorphism bearomorphism changed the title [SPARK-60139][PYTHON] Remove unused UNSUPPORTED_COMBINATIONS [SPARK-60139][PYTHON] Remove dead code UNSUPPORTED_COMBINATIONS Oct 11, 2026
@bearomorphism
bearomorphism marked this pull request as ready for review October 11, 2026 04:04
@gaogaotiantian

Copy link
Copy Markdown
Contributor

pyspark.install should be internal only - I did not find it in our documentation. It's okay if we remove a global variable from it if it's not considered public. The variable has not been used for 6 years and we should just remove it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants