BUG: coerce empty-string IEEE NaN to null for pyarrow floats - #66836
BUG: coerce empty-string IEEE NaN to null for pyarrow floats#66836hebian1994 wants to merge 4 commits into
Conversation
|
Thanks for the pull request, @hebian1994! It's linked to #66834, but that issue isn't assigned to you yet. To make sure two people don't unknowingly work on the same thing, we ask contributors to claim an issue first. Just comment I've closed this PR for now to keep the queue tidy — none of your work is lost. To pick it back up: 1) comment |
github-actions
Bot
added
the
Needs Issue Assignment
|
Thanks for the pull request, @hebian1994! It's linked to #66834, but that issue isn't assigned to you yet. To make sure two people don't unknowingly work on the same thing, we ask contributors to claim an issue first. Just comment I've closed this PR for now to keep the queue tidy — none of your work is lost. To pick it back up: 1) comment |
github-actions
Bot
removed
the
Needs Issue Assignment
rhshadrach
left a comment
rhshadrach
left a comment
There was a problem hiding this comment.
Thanks for the PR! I'm seeing this fail on pd.NA
from pandas.core.arrays.arrow.array import ArrowExtensionArray
strings = np.array(["1.5", pd.NA, ""], dtype=object)
ArrowExtensionArray._from_sequence_of_strings(strings, dtype=pd.ArrowDtype(pa.float64()))
# TypeError: boolean value of NA is ambiguouswhere it should raise ValueError: could not convert string to float: ''
Can you also add a test with read_csv.
| arr = np.asarray(strings, dtype=object) | ||
| unmasked_empty = (arr == "") & ~np.asarray(mask, dtype=np.bool_) | ||
| if unmasked_empty.any(): | ||
| raise ValueError("could not convert string to float: ''") |
There was a problem hiding this comment.
I think this is also hit with e.g. int/decimal types too. Can you use pa_type instead.
| - Bug in :meth:`DataFrame.interpolate` and :meth:`Series.interpolate` with ``method`` in ``"index"``, ``"values"`` or ``"time"`` raising when the index had an :class:`ArrowDtype` timestamp or duration dtype; these now match the equivalent :class:`DatetimeIndex` or :class:`TimedeltaIndex` (:issue:`66338`) | ||
| - Bug in :meth:`DataFrame.shift`, :meth:`DataFrame.where`, :meth:`DataFrame.mask`, :meth:`Series.shift`, :meth:`Series.where`, and :meth:`Series.mask` raising an internal ``AssertionError`` for a NumPy bytes dtype instead of upcasting to ``object`` to hold a missing value; item assignment now raises the expected ``TypeError`` for the incompatible value (:issue:`52373`) | ||
| - Bug in :meth:`Series.combine_first` crashing when Series names are :class:`Timestamp` objects (:issue:`65333`) | ||
| - Bug in :class:`ArrowExtensionArray` where converting empty strings to a floating PyArrow dtype stored IEEE ``NaN`` instead of raising like numpy ``float64`` (e.g. empty CSV fields with ``dtype="double[pyarrow]"`` and ``keep_default_na=False``) (:issue:`66834`) |
There was a problem hiding this comment.
I believe the only place this code is hit is I/O, can you move to the I/O section and make it something like
- :func:`read_csv`, :func:`read_fwf`, and :func:`read_excel` with a numeric :class:`ArrowDtype` (e.g. ``dtype="double[pyarrow]"``) now raise ``ValueError`` for an empty field that is not treated as missing (e.g. with ``keep_default_na=False``), matching the behavior of numpy dtypes such as ``float64``; previously the value was silently stored as ``NaN`` but was not considered missing by :meth:`Series.isna` and :meth:`Series.fillna` (:issue:`66834`)Move the GH#66834 whatsnew note to I/O to resolve the Missing-section conflict. Empty-string conversion now skips pd.NA and raises using pa_type.
4 participants
closes BUG: coerce empty-string IEEE NaN to null for pyarrow floats #66834
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
[] Added type annotations to new arguments/methods/functions.
Added an entry in the latest
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.I have reviewed and followed all the contribution guidelines
Check exactly one of the following, per the automated contributions policy:
I did not use AI to develop this pull request.
I used AI to develop this pull request. I prompted it to follow
AGENTS.md, I have reviewed and understood every change, and I have described above how I used it and exactly which tool, model version, and effort setting — e.g.claude opus 4.8 (xhigh), not justclaude.