WIP: Enable array api support in neighbor#2700
WIP: Enable array api support in neighbor#2700yuejiaointel wants to merge 150 commits intouxlfoundation:mainfrom
Conversation
a569e0c to
62c8ddd
Compare
|
As part of the PR, please add the relevant classes that will get array api support to this list now that they are documented: |
a431e04 to
8bec3dc
Compare
|
/intelci: run |
|
/intelci: run |
2 similar comments
|
/intelci: run |
|
/intelci: run |
|
/intelci: run |
4 similar comments
|
/intelci: run |
|
/intelci: run |
|
/intelci: run |
|
/intelci: run |
|
The CI error appears to come from the changes in this PR: Note that it occurs when using the SPMD class: |
|
/intelci: run |
1 similar comment
|
/intelci: run |
I think we need both standard and special because the standard (algorithem = "auto") would pick kd_tree with the test data (euclidean metric) if self._fit_method in ["auto", "ball_tree"]:
condition = (
self.n_neighbors is not None
and self.n_neighbors >= self.n_samples_fit_ // 2
)
if self.n_features_in_ > 15 or condition:
result_method = "brute"
else:
if self.effective_metric_ in ["euclidean"]:
result_method = "kd_tree"
else:
result_method = "brute"
else:
result_method = self._fit_methodand algorithm = "brute" will not be tested so we need to test it in special instance |
|
/intelci: run |
|
/intelci: run |
numpy.unique_inverse returns a plain tuple on some numpy/Python version combinations (e.g., Python 3.13), not a namedtuple with .values/.inverse_indices attributes. Use tuple unpacking which works for both tuple and namedtuple return types.
The previous check rejected all non-SYCL data, but dispatch() transfers dpctl/dpnp data to host (numpy) before the second _get_backend call. This caused dpctl/dpnp GPU tests to incorrectly fall back to sklearn. Fix: skip numpy arrays in the check (they are always valid), only reject non-numpy arrays that lack __sycl_usm_array_interface__ (e.g. torch XPU tensors).
|
/intelci: run |
|
/intelci: run ml-benchmarks |
2 similar comments
|
/intelci: run ml-benchmarks |
|
/intelci: run ml-benchmarks |
|
@yuejiaointel @ethanglaser Torch tensors with xpu device are SYCL arrays. They are supposed to execute on GPU. We used to have validation on torch back when there was a public GPU runner. To verify whether something runs on oneDAL or not, or whether it runs on CPU, you might want to enable verbose mode for both libraries: import os
os.environ["SKLEARNEX_VERBOSE"] = "INFO"
os.environ["ONEDAL_VERBOSE"] = "4"If unsure about intended behaviors for edge cases, you might also want to check what our own documentation says: It looks like SVMs are indeed broken for torch inputs, but LinearRegression on CPU and GPU for example works correctly with torch. |
@yuejiaointel This is still outputting float64 for me. Are you using the latest versions of scikit-learn and scipy? |
This is now falling back to sklearn. It should execute on GPU. |
sklearnex/neighbors/common.py
Outdated
| if non_none_data: | ||
| _, is_array_api = get_namespace(*non_none_data) | ||
| if is_array_api and not any( | ||
| hasattr(x, "__sycl_usm_array_interface__") for x in non_none_data |
There was a problem hiding this comment.
This is not part of the array API specification. It is a workaround back from a time when dpnp wasn't compliant with array API. Please use the helper tools that you'll find through the sklearnex module, such as here:
There was a problem hiding this comment.
I don't see using this anymore in common.py maybe it is old code
sklearnex/_device_offload.py
Outdated
| # Remove check for result __sycl_usm_array_interface__ on deprecation of use_raw_inputs | ||
| # For tuple/list results (e.g. kneighbors), check elements instead of the container | ||
| result_on_device = ( | ||
| all(hasattr(r, "__sycl_usm_array_interface__") for r in result) |
There was a problem hiding this comment.
Same comment here - please avoid checking or using this attribute.
| y, | ||
| dtype=[xp.float64, xp.float32], | ||
| accept_sparse="csr", | ||
| multi_output=True, |
There was a problem hiding this comment.
If multi-output is now supported, then please add it to the support tables in the documentation:
There was a problem hiding this comment.
Good catch! This was added to fix a patching test warning(treated as error):
WARNING: DataConversionWarning : A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
No warnings with multi_output=True
but multi_output=true only controls the validation part, tells validate data to accept 2D y with no warning, it does not mean onedal actually run multi output, so no documentation update needed here
|
@yuejiaointel From the discussion here: Please port the rest of the LOF code that calls scikit-learn to array API for full compatibility: I've already created a separate PR to document support for it: |
After some investigation, I see other algorithms also don't preserve the data type
pl dataframe don't have a dtype so sklearn check_array will use the first in the list passed in which is float64, an minimal example:
you can see here after check_array we have float64 even tho X is float32, for numpy we preserved the datatype but for dataframe it is just choosing the first dtype in list. One potential fix would be convert plars df to numpy before passing to check_array, but it seems to be beyond the scope of this pr |
fixed! now I can see it says when using SKLEARNEX_VERBOSE=INFO |
added! |
|
/intelci: run |
|
/intelci: run |
| usm_iface := getattr(data, "__sycl_usm_array_interface__", None) | ||
| ) and not hasattr(result, "__sycl_usm_array_interface__"): | ||
| queue = usm_iface["syclobj"] | ||
| sycl_type, _, _ = _get_sycl_namespace(data) |
There was a problem hiding this comment.
Curious here: why did it require changes for KNN if it was working for other algorithms?
Indeed, looks like something is not working correctly across several estimators. Scikit-learn would preserve the dtype in those cases: import numpy as np
rng = np.random.default_rng(seed=123)
X = rng.standard_normal(size=(1000, 20), dtype=np.float32)
y = rng.standard_normal(size=X.shape[0]).astype(np.float32)
from sklearn import config_context
from sklearn.linear_model import LinearRegression
import polars as pl
Xdf = pl.DataFrame(X)
ys = pl.Series(y)
lr = LinearRegression().fit(Xdf, ys)
print('coef_ dtype:', lr.coef_.dtype)
pred = lr.predict(Xdf)
print('predict dtype:', pred.dtype) |
Description
Follow up PR of #2284 (will rebase after this one is merged)
that refactor neighbors with array api standard
Summary of Neighbor Code Refactoring Changes
onedal layer changes:
_validate_data,_get_weights,_validate_targets,_validate_n_classes- all validation moved to sklearnex layerIntegral,_get_config, and validation functions (_check_array,_check_classification_targets,_check_X_y,_column_or_1d,_num_samples)outputs_2d_setting) moved from onedal_fit()to sklearnex layer - onedal now expects pre-processed_yandclasses_attributes(-1, 1)shape for C++ compatibility_kneighbors()method updated with full array API support: usesxpnamespace for all operations (argsort,reshape,concatenate,arange,where,all)sample_mask[:, 0][dup_gr_nbrs] = Falseto explicit operations usingxp.where()andxp.concatenate()(array API doesn't support chained indexing assignment)predict()andpredict_proba()methods removed fromKNeighborsClassifier- computation logic moved to sklearnex_predict_skl()andpredict()methods removed fromKNeighborsRegressor- dispatch logic moved to sklearnex_predict_gpu()kept in onedal for GPU backend support (called by sklearnex)sklearnex layer changes:
@enable_array_apidecorator toKNeighborsClassifierandKNeighborsRegressor(requires sklearn >= 1.5 for regressor due toy_numericparameter)common.py:_get_weights()- adapted from sklearn with array API support (handlesdpctl/dpnparrays)_compute_weighted_prediction()- regression prediction using array APItake()andstack()(array APItake()only supports 1-D indices)_compute_class_probabilities()- classification probabilities with array API support (avoids fancy indexing via sample-by-sample accumulation)_predict_skl_regression()and_predict_skl_classification()- unified prediction helpers that callkneighbors()and compute results_process_classification_targets()- handles class encoding,outputs_2d_setting, validates n_classes >= 2_process_regression_targets()- handles shape processing for regressors_kneighbors_post_processing()- handles kd_tree sorting, query_is_train (X=None) case, return_distance decision_onedal_fit()in all estimators now: validates data, sets effective metric, processes targets, then calls onedal backend_onedal_predict()in classifier uses_predict_skl_classification()helper (handles X=None LOOCV case properly)_onedal_predict_proba()computes probabilities directly in sklearnex using_compute_class_probabilities()_onedal_predict()in regressor dispatches between GPU (_predict_gpu()) and SKL (_predict_skl()) paths based on device and weights_onedal_kneighbors()simplified in all estimators: validates X, calls onedal, returns result (post-processing removed - now in onedal)_onedal_score()in classifier uses_transfer_to_host()to convert array API arrays to numpy for sklearn'saccuracy_score()_validate_n_neighbors(),_set_effective_metric(),_validate_n_classes(),_kneighbors_validation()helper methods_fit_validation()expanded with array API dtype support:dtype=[xp.float64, xp.float32]_onedal_supported()updated for array API: usesget_namespace()andxp.asarray()for type checkskneighbors_graph()uses_transfer_to_host()to ensure numpy arrays for scipycsr_matrixconstructioncityblock→manhattan,l1→manhattan,l2→euclideanfor oneDAL compatibilityTesting changes:
onedal/neighbors/tests/test_knn_classification.py) deprecated with notice pointing to sklearnex testssklearnex/neighbors/tests/test_neighbors.py:test_knn_classifier_iris(),test_knn_classifier_pickle()test_knn_classifier_single_class()with@pytest.mark.allow_sklearn_fallback(oneDAL doesn't support one-class case)Documentation changes:
KNeighborsClassifier,KNeighborsRegressor,NearestNeighbors,LocalOutlierFactortoarray_api.rstshowing array API supportArchitecture pattern:
Array API compliance:
array[:, 0][mask] = value→ explicitxp.where()+xp.concatenate()__setitem__:proba_k[all_rows, idx] += weights→ sample-by-sample accumulationtake()only supports 1-D indices: iterate over samples to gather neighbor values_transfer_to_host()before sklearn utility functions (accuracy_score, csr_matrix) that require numpyChecklist:
Completeness and readability
Testing
Performance