BUG: Accessing a row with loc recasts dtype and can cause values to change depending on other columns

[ceesem]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

object_ids = [864691132640188560, 864691132643754038, 864691132680223827]

df_int = pd.DataFrame(
    {
        'object_id': object_ids,
        'value_int': [1, 2, 3]
    }
)

df_float = pd.DataFrame(
    {
        'object_id': object_ids,
        'value_float': [1.1, 2.2, 3.3]
    }
)

df_mixed = pd.DataFrame(
    {
        'object_id': object_ids,
        'value_float': [1.1, 2.2, 3.3],
        'value_str': ['a', 'b', 'c'],
    }
)

print("object_id from df_int:", int(df_int.loc[0]['object_id']))
print("object_id from df_float:", int(df_float.loc[0]['object_id']))
print("object_id from df_mixed:", int(df_mixed.loc[0]['object_id']))
print("object_id from df_float with direct loc: ", int(df_float.loc[0, 'object_id']))

Issue Description

When accessing a row of a dataframe through df.loc, pandas will attempt to recast mixed dtypes even after care was taken to ensure the dataframe columns had the correct datatype using the numpy notion of safe dtype casting. Moreover, it is dependent on the data in other columns in surprising ways. For example, if all columns are ints then no recasting occurs (df_int). If columns include strings (df_mixed, no recasting occurs and the resulting Series has type object. However, if columns are a mix of ints and floats then all data is recast to floats (df_float).

This has two problems: In the example above, the numpy dtype conversions can change int64 values despite being considered "safe" by numpy. This is a known issue in numpy. Second, even with smaller safe numbers, differences can occur due to the behavior of ints versus floats. For example,

object_ids = [1, 2, 3]

df_int = pd.DataFrame(
    {
        'object_id': object_ids,
        'value_int': [1, 2, 3]
    }
)

df_float = pd.DataFrame(
    {
        'object_id': object_ids,
        'value_float': [1.1, 2.2, 3.3]
    }
)

print( str(df_int.loc[0]['object_id']), 'vs', str(df_float.loc[0]['object_id']) )
# 1 vs 1.0

yields different strings despite accessing what should be the same data.

Expected Behavior

The dtype of a Series taken using df.loc from a DataFrame with mixed column dtypes should be object and the individual column dtypes should remain as they were in the DataFrame.

Installed Versions

---

commit : d9cdd2e
python : 3.11.9.final.0
python-bits : 64
OS : Darwin
OS-release : 23.4.0
Version : Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.2
numpy : 1.23.2
pytz : 2024.1
dateutil : 2.9.0.post0
setuptools : 69.5.1
pip : 24.0
Cython : 3.0.10
pytest : 8.2.0
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.4
IPython : 8.24.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.6.0
gcsfs : None
matplotlib : 3.9.0
numba : 0.59.1
numexpr : 2.10.0
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 16.1.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.13.0
sqlalchemy : None
tables : 3.9.2
tabulate : None
xarray : 2024.5.0
xlrd : None
zstandard : 0.22.0
tzdata : 2024.1
qtpy : None
pyqt5 : None

[rhshadrach]

I think this is well-known behavior, so I've removed the Bug tag and replaced it with API Design.

I believe your expected behavior is that row selection on a non-homogeneous DataFrame results in a Series with object dtype. Is that correct?

[ceesem]

Yes, that is correct. Perhaps more specifically, I would expect that the elements of a Series returned from df.loc[idx] would match the dtypes that they had in the dataframe.

Thanks for adjusting the tag for what works for you! I could tell it was known behavior but also couldn't find an issue specifically about the df.loc recasting.