[enhancement] add dlpack support to to_table
#2275
Conversation
|
/intelci: run |
|
/intelci: run |
|
/intelci: run |
|
/intelci: run |
|
/intelci: run |
|
/intelci: run |
| res = convert_to_table(copy, q_obj, true); | ||
| } | ||
| else { | ||
| throw std::invalid_argument("dlpack input could not be converted into onedal table."); |
There was a problem hiding this comment.
Very minor thing here but: why not py::type_error like in the others?
| copy = space.attr("asarray")(obj, "copy"_a = true); | ||
| } | ||
| else { | ||
| throw std::runtime_error("Wrong strides"); |
There was a problem hiding this comment.
Shouldn't the error mention something about the arrai API namespace instead of strides?
There was a problem hiding this comment.
This is for conformance to raised errors in dpctl and dpnp. I'm not saying its right, but its consistent.
There was a problem hiding this comment.
If I'm getting the logic correctly, this error could potentially be raised for inputs that are from packages unrelated to dpctl / dpnp, like PyTensor.
|
@icfaust I'm again thinking that it might be better to have the dlpack header as a git submodule: It has already undergone changes since the time that you copied it here. |
https://github.com/dmlc/dlpack/blob/main/include/dlpack/dlpack.h#L57-L59 None of the changes are relevant to our current implementation, as the supported types and devices usable in scikit-learn-intelex/ oneDAL are strict and defined. I believe my previous answer showing that other more popular frameworks, including dependencies of scikit-learn-intelex (i.e. numpy), follow the currently implemented strategy. If you insist we can have a maintainer step in for an opinion and if they agree I will make the change, otherwise I would like to keep it as it is. |
|
/intelci: run |
Description
This PR introduces
__dlpack__tensor (https://github.com/dmlc/dlpack) consumption byto_tableallowing for zero-copy use of data in oneDAL. This is important for enabling array_api support and is a pre-requisite for #2096 (array api dispatching). That PR is then a pre-requisite for #2100 #2106 #2189 #2206 #2207 and #2209. Sklearn provides array_api support for some algorithms. If we wish to fully support zero copy of sycl_usm inputs, we need to be able to consume array_api inputs due to underlying sklearn dependencies (validate_data,check_array, etc.). While we support Sycl usm ndarrays (dpctl,dpnp) via the__sycl_usm_array_interface__method in the onedal folder estimators, to properly interface estimators in the sklearnex folder, we need to support the__dlpack__method of arrays/tensors. This PR does that and greatly simplifies the necessary logic in #2096 and the follow-up PRs. This PR also provides the added benefit of working with other frameworks which support SYCL gpu data which have__dlpack__interfaces (i.e. PyTorch).NOTES:
This PR continues from [enhancement] Refactor
onedal/datatypesin preparation for dlpack support #2195 to integrate dlpack support, and it takes code from DataManagement update #1568. Please use DataManagement update #1568 as reference, though it contained some mistakes.This new functionality is not yet exposed publicly and therefore must be added to the documentation with ENH: Array API dispatching #2096.
This aspect is not yet benchmarkable/ has nothing to benchmark against.
Memory leak checking using the infrastructure from ENH: Data management update to support SUA ifaces for Homogen OneDAL tables #2045 (only CPU at the moment, will be modified if/when PyTorch support is brought online)
It does numeric testing via onedal's
assert_all_finitetesting usingarray_api_strictarrays, as that has a simple interface to the backend with no checking of array aspects on the python-side beforeto_table.A special testing class is created to verify for SYCL device support and is used in
test_data.py.This does not have any work with returning
__dlpack__supported arrays as results. And must be done as a follow-up PR (probably also a good idea in order to ease reviewing). Therefore, this is only about array/tensor consumption.TODO: add a onedal function which checks a dlpack tensor for C-contiguity or F-contiguity similar to the
flagsattribute of numpy/dpctl/dpnp. This is out of the scope of this PR, but is necessary for assert_all_finite support for the next step in array_api work.PR should start as a draft, then move to ready for review state after CI is passed and all applicable checkboxes are closed.
This approach ensures that reviewers don't spend extra time asking for regular requirements.
You can remove a checkbox as not applicable only if it doesn't relate to this PR in any way.
For example, PR with docs update doesn't require checkboxes for performance while PR with any change in actual code should have checkboxes and justify how this code change is expected to affect performance (or justification should be self-evident).
Checklist to comply with before moving PR from draft:
PR completeness and readability
Testing
Performance