Replies: 2 comments 2 replies
-
Thanks Nic for opening this and Nic/Alenka for collecting these! I'll also add
I'm happy to poke through these as I have time (since many of them arose from Geo extension types) or review of others have funding to do the work! |
Beta Was this translation helpful? Give feedback.
1 reply
-
A few notes from points raised in the Arrow dev meeting:
|
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi folks,
I wanted to open a discussion around supporting compute kernels on ExtensionType Arrays or casting from ExtensionTypes to supported Arrow types. There was a recent request from an R user around this [1], and we’ve supplied them with a workaround, but after discussion with Alenka, we noticed this theme has come up repeatedly.
In the case of simple ExtensionTypes where it would make sense to operate on the storage type or be able to cast to it, it feels like this could be a great win to enhance interoperability, though I see from previous discussions there are questions to resolve around how this would work if a straightforward mapping didn’t exist [2].
There are multiple existing issues around this [3] (thanks @AlenkaF for collating these), and I recognise it could take significant dev time to implement, but if we had an idea of how this would look, perhaps we could break off smaller chunks, and potentially apply for funding to work on this? It feels like it would be a big win for interoperability.
My goals for this discussion:
Keen to hear your thoughts!
Nic
[1] R arrow cannot handle labelled data in arrow tables · Issue #45601 · apache/arrow · GitHub
[2] [C++][Python] Support ExtensionType arrays in more kernels
[3] List of previous issues on the topic:
R
User request for extension type compute support
R arrow cannot handle labelled data in arrow tables · Issue #45601 · apache/arrow · GitHub
Python
Extension types and compute functions
[C++][Python] Support ExtensionType arrays in more kernels
[C++] Provide a default implementation for ExtensionType::ExtensionEquals
Enhancement Request: Custom Operator Support for PyArrow Extension Types in Compute Functions
[Python] For extension types, compute kernels should default to storage types?
Extension types and PyArrow constructors
[Python] Error using extension types in struct in PyArrow
Extension types not fully supported in list arrays
[Python] Instantiating arrays with type ListType[ExtensionType] is not supported
[Python] Python extension types aren't usable in struct arrays, maybe the same: [Python] pyarrow.Table.cast does not handle extension types which wrap list types
[C++][Python] SEGFAULT when casting FixedSizeTensorArray to storage type then back to FixedSizeTensorArray
Beta Was this translation helpful? Give feedback.
All reactions