BUG: coerce empty-string IEEE NaN to null for pyarrow floats

[hebian1994]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

from io import StringIO

import pandas as pd

csv = """id,A,B,C
1,1.5,x,1
2,,x,2
3,2.0,x,3
"""

df = pd.read_csv(StringIO(csv), dtype={"A": "double[pyarrow]"}, keep_default_na=False)
df = df[["A", "B", "C"]]
df["A"] = df["A"].fillna(0)
print(df)
print("isna:", pd.isna(df["A"]).tolist())

Issue Description

Bug in Arrow float conversion: empty strings become IEEE NaN, so fillna does not fill them

When converting string data (e.g. CSV empty fields ,,) to a floating PyArrow dtype, empty strings are not NA (pd.isna("") is False). to_numeric turns them into IEEE NaN, and _from_sequence_of_strings builds pa.array(..., mask=isna(strings)) with that mask all-False. The NaN is stored as a value (null_count=0), not a pyarrow null.

Default pandas 3 semantics treat NaN as NA (is_nan_na() is True), but isna / fillna only look at is_null(). Result: fillna(0) is a no-op.

Expected Behavior

Empty CSV fields converted to double[pyarrow] should be pyarrow null (). isna() is True for those rows, and fillna(0) replaces them with 0.0.

Installed Versions

Details

pd.show_versions()

INSTALLED VERSIONS

commit : e68db09
python : 3.13.5
python-bits : 64
OS : Windows
OS-release : 11
Version : 10.0.26200
machine : AMD64
processor : Intel64 Family 6 Model 151 Stepping 5, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : Chinese (Simplified)_China.936

pandas : 3.0.5
numpy : 2.5.2
dateutil : 2.9.0.post0
pip : None
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : None
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
psycopg2 : None
pymysql : None
pyarrow : 25.0.0
pyiceberg : None
pyreadstat : None
pytest : None
python-calamine : None
pytz : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
qtpy : None
pyqt5 : None

[hebian1994]

/take

[github-actions]

Thanks for your interest, @hebian1994! This issue is still labeled Needs Triage, which means a maintainer needs to review it before work starts — so it isn't open to claim just yet. Please hold off on a PR until that label is removed. See the contributing guide for more.

[rhshadrach]

Cannot reproduce on Python 3.11 on main nor 3.0.x with PyArrow 25.0.1. What do you get running below?

csv = """id,A,B,C
1,1.5,x,1
2,,x,2
3,2.0,x,3
"""
df = pd.read_csv(StringIO(csv), dtype={"A": "double[pyarrow]"}, keep_default_na=False, na_values=[""])
df = df[["A", "B", "C"]]
df["A"] = df["A"].fillna(0)
print(df)
#      A  B  C
# 0  1.5  x  1
# 1  0.0  x  2
# 2  2.0  x  3

[hebian1994]

@rhshadrach can use this sample code,thanks

from io import StringIO

import pandas as pd

csv = """id,A,B,C
1,1.5,x,1
2,,x,2
3,2.0,x,3
"""

df = pd.read_csv(StringIO(csv), dtype={"A": "double[pyarrow]"}, keep_default_na=False,
                 )
df = df[["A", "B", "C"]]
df["A"] = df["A"].fillna(0)
print(df)
print("isna:", pd.isna(df["A"]).tolist())

[hebian1994]

If I write it this way, it will cause the string with length 0 to become NaN, which is not what I intended

from io import StringIO

import pandas as pd

csv = """id,A,B,C
1,1.5,x,1
2,,x,2
3,2.0,,3
"""

df = pd.read_csv(StringIO(csv), dtype={"A": "double[pyarrow]"}, keep_default_na=False,
                na_values=["",])
df = df[["A", "B", "C"]]
df["A"] = df["A"].fillna(0)
print(df)
print("isna:", pd.isna(df["A"]).tolist())

[jorisvandenbossche]

@hebian1994 it seems you removed the na_values=[""] in the top post example? And that seems the crucial part that changes behaviour. Also, is there a specific reason that you are using keep_default_na=False? (without it this works fine)

But so there seems to be some issue:

>>> csv = "id,A\n1,1.5\n2,\n3,2.0"
# by default, reads as float64 and parses empty value as missing (NaN)
>>> pd.read_csv(StringIO(csv))
   id    A
0   1  1.5
1   2  NaN
2   3  2.0
# if we explicitly ask for a pyarrow-backed float64, it still parses the empty as missing (NA in this case)
>>> pd.read_csv(StringIO(csv), dtype={"A": "double[pyarrow]"})
   id     A
0   1   1.5
1   2  <NA>
2   3   2.0
# if we turn off parsing empty as missing, the column gets parsed as strings (and empty stays en empty string)
>>> pd.read_csv(StringIO(csv), keep_default_na=False)
   id    A
0   1  1.5
1   2     
2   3  2.0
# but then asking explicitly for float, raises an error for default float64 ...
>>> pd.read_csv(StringIO(csv), dtype={"A": "float64"}, keep_default_na=False)
...
ValueError: could not convert string to float: ''
# ... but "works" for pyarrow-backed float64, only to parse the empty as NaN (instead of NA missing sentinel)
>>> pd.read_csv(StringIO(csv), dtype={"A": "double[pyarrow]"}, keep_default_na=False)
   id    A
0   1  1.5
1   2  NaN
2   3  2.0
# when explicitly marking empty as missing again (with `na_values`), everything works again as epxected
>>> pd.read_csv(StringIO(csv), keep_default_na=False, na_values=[""])
   id    A
0   1  1.5
1   2  NaN
2   3  2.0
>>> pd.read_csv(StringIO(csv), dtype={"A": "double[pyarrow]"}, keep_default_na=False, na_values=[""])
   id     A
0   1   1.5
1   2  <NA>
2   3   2.0

So I think for the case of asking for pyarrow-backed float with an empty string that is not recognized as missing, the question is: also raise (like is done for the default float64)? Or if we keep coerce to missing, at least correctly use NA instead of NaN?

[rhshadrach]

+1 I'm on raising for

pd.read_csv(StringIO(csv), dtype={"A": "double[pyarrow]"}, keep_default_na=False)

I believe the mental model for users should be that CSV files contain strings which pandas is converting to the appropriate dtype. If you disable "empty means NA" then you should expect to interpret empty as the empty string, and converting raises.