Fixes for pandas 2&3#976
Conversation
|
@jorisvandenbossche - I didn't mean to come back to it, but I had bot help. Now, fastparquet does support latest pandas again, so the XFAIL(strict) will start to show up as failures. I assume you only test against the released version, so you won't see it until then. |
jorisvandenbossche
left a comment
jorisvandenbossche
left a comment
There was a problem hiding this comment.
Nice to see the update! And sorry for the slow feedback (had some holiday), we are updating the pandas tests now, and noticed one issue
| for col, parts in arrow_string_columns.items(): | ||
| if not parts: | ||
| continue | ||
| final_arr = _pa.concat_arrays(parts) |
There was a problem hiding this comment.
| final_arr = _pa.concat_arrays(parts) | |
| final_arr = _pa.chunked_array(parts) |
To avoid copying the data, you might want to keep the parts as a chunked array? (concat_arrays will result in an actual contiguous pyarrow.Array)
(pandas uses a chunked array under the hood)
| # pre-allocated object array already has None for the missing | ||
| # rows; skip the arrow conversion so callers see partial data. | ||
| continue | ||
| arrow_str = pd.arrays.ArrowStringArray(_pa.chunked_array([final_arr])) |
There was a problem hiding this comment.
Unfortunately this creates the "wrong" extension dtype .. We made it a bit complicated, but the above will create one using the opt-in NA-variant of the string dtype, and not the default NaN-variant (which is the default str dtype in pandas 3), but ArrowStringArray(..) init seems to default to the non-default dtype (I assume for historical reasons).
You could do pd.arrays.ArrowStringArray(_pa.chunked_array([final_arr]), dtype=pd.StringDtype(na_value=np.nan))
But it might be better to use a slightly more general API to construct the series from arrow data:
| arrow_str = pd.arrays.ArrowStringArray(_pa.chunked_array([final_arr])) | |
| arrow_str_ser = pd.Series.from_arrow(final_arr) | |
| arrow_str_ser.index = df.index |
That should also be more robust in case pandas changes the default str dtype in the future (at least then if something is not correct, it is pandas' responsibility to ensure Series.from_arrow() works correctly).
However, that method requires pandas 3+, and I see that the _USE_ARROW_STRINGS is not sufficient condition for that (since that can also be true with pandas 2, given you are checking the option for that)
(I also only noticed this because of seeing the test assertion in a pandas test)
|
(I can do a quick PR if you want) |
|
Yes, thanks @jorisvandenbossche , that would be great. |
2 participants
Should work for both pandas 2 (object string by default) and pandas 3 (arrow strings if installed).