Skip to content

[WIP][PYTHON] Fix PySpark Vector dot product typing - #59349

Draft
bearomorphism wants to merge 1 commit into
apache:masterfrom
bearomorphism:bearomorphism/fix-mllib-vector-dot-typing
Draft

bearomorphism wants to merge 1 commit into
apache:masterfrom
bearomorphism:bearomorphism/fix-mllib-vector-dot-typing

Conversation

@bearomorphism

Copy link
Copy Markdown

What changes were proposed in this pull request?

  • Declare dot() on pyspark.mllib.linalg.Vector, following the base class's existing NotImplementedError pattern.
  • Add compatible overloads to Vector, DenseVector, and SparseVector: vector/list/tuple/range operands return np.float64, while NumPy/SciPy operands may return a scalar or array.
  • Remove five redundant type: ignore[attr-defined] comments from classification and regression callers.
  • Add a focused typing regression and register the MLlib typing-test directory in dev/lint-python.

The concrete dot-product implementations are unchanged. The separate .size declaration fix is not included.

Why are the changes needed?

Both concrete vector classes implement dot(), but the base Vector does not declare it. Valid calls through a Vector reference therefore require attribute-error suppressions.

The existing concrete return annotations also describe only scalar results, although matrix operands can produce arrays. The overloads expose the shared method without incorrectly treating every result as a scalar.

Does this PR introduce any user-facing change?

Static typing changes only: type checkers recognize Vector.dot() and distinguish scalar-only operand types from potentially array-valued operands. Dense and sparse dot-product runtime behavior is unchanged.

How was this patch tested?

Added one focused typing regression checking return-type inference through the base and concrete vector classes.

The following checks passed:

mypy --config-file python/mypy.ini --follow-imports=silent python/pyspark/mllib
ruff check python/pyspark/mllib/linalg/__init__.py \
  python/pyspark/mllib/classification.py python/pyspark/mllib/regression.py
ruff format --check python/pyspark/mllib/linalg/__init__.py \
  python/pyspark/mllib/classification.py python/pyspark/mllib/regression.py
bash -n dev/lint-python
git diff --check upstream/master...HEAD

Also passed 93 existing doctest examples (89 in mllib.linalg and four in LinearRegressionModelBase) using NumPy's legacy 1.13 output formatting, plus the existing VectorTests.test_dot method body without JVM fixtures. No JVM-backed test suites were run.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Pi 1.1.0

@bearomorphism bearomorphism changed the title [MINOR][MLLIB] Fix PySpark Vector dot product typing [WIP][PYTHON] Fix PySpark Vector dot product typing Oct 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant