DOC: Series.diff with boolean dtype does not return a series of dtype float

[from-nowhere]

Pandas version checks

• I have checked that the issue still exists on the latest versions of the docs on main here

Location of the documentation

https://pandas.pydata.org/docs/reference/api/pandas.Series.diff.html#pandas.Series.diff
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.diff.html#pandas.DataFrame.diff

Documentation problem

The documentation for pandas.Series.diff and pandas.DataFrame.diff states that no matter the dtype of the original series/column, the output will be of dtype float64. This is not true for series/columns of dtypes bool -- the output here is of dtype object.

For example:

import pandas as pd
# pd.__version__ == '2.2.0'

s = pd.Series([True, True, False, False, True])
d = s.diff()

# d.dtype is now 'object'

Indeed, the underlying function algorithms.diff explicitly differentiates between boolean and integer dtypes.

Suggested fix for documentation

The Notes section should read something like this:

Notes
-----
For boolean dtypes, this uses :meth:`operator.xor` rather than
:meth:`operator.sub` and the result's dtype is ``object``.
Otherwise, the result is calculated according to the current dtype in {klass},
however the dtype of the result is always float64.

[from-nowhere]

I am also open to change the function behaviour to return a float64 dtype, as advertised. This would also solve another problem: in 2.2.0, downcasting object dtypes by fillna etc. is deprecated: s.diff().fillna(False) produces a warning and the best replacement I could come up with is s.diff().astype(float).fillna(0.0).astype(bool)

[rhshadrach]

Thanks for the report. Returning object is the behavior on 1.0.0; haven't tried prior versions. I'm +1 on agreeing with the docs here and returning float64 instead of object dtype.

It might also be worth adding a fill_value argument, similar to shift.

Could use some other eyes - cc @MarcoGorelli.

[mutricyl]

Looking at issue 821 in pandas-stubs it looks like Series.diff() does not always returns float64 it can return pandas.Timedelta.

Beside is look strange to me when making the difference of 2 integers to return a float. It may be due to pandas/numpy internal but mathematically speeking int - int = int. ( uint - uint = int to link with #4899)

I made a short review of current behavior:

float ➡️ float
int ➡️ float
unit ➡️ float
timestamp ➡️ timedelta
period ➡️ object
timedelta ➡️ timedelta64
interval ➡️ TypeError: IntervalArray has no 'diff' method. Convert to a suitable dtype prior to calling 'diff'.
bool ➡️ object
object ➡️ object

[mutricyl]

I made some more test to cover all possible series types:

bytes ➡️ numpy.core._exceptions._UFuncNoLoopError: ufunc 'subtract' did not contain a loop with signature matching types (dtype('S21'), dtype('S21')) -> None
dtype ➡️ TypeError: unsupported operand type(s) for -: 'type' and 'type'
datetime.date ➡️ timeDelta
datetime.time ➡️ TypeError: unsupported operand type(s) for -: 'datetime.time' and 'datetime.time'
complex ➡️ complex
Baseoffset considered already as object ➡️ object

Pretty strange that datetime.time throws and error. Would that be a bug ?