Use nullable types by default and avoid object type

[ehariri]

To make sure the Pandas APIs are Jittable, Pandas should avoid changing data types based on values and avoid the object type.

One example is that Pandas should default to nullable types directly instead of Numpy/Object types. Currently if you specify a value that Pandas will treat as NA (e.g. None) without a pd.array, you will not produce a nullable type and instead produce a Numpy array type, often an object array.

Arrays with NAs should automatically use the correct nullable type. E.g. pd.DataFrame({'A': [1, 2, None], 'B': [True, False, None]}) should have column A of type Int64 rather than float64 and column B of type boolean rather than object. Users can specify non-nullable data types directly if necessary.

For output of I/O calls like read_csv, the data type should always be a nullable type, and not determined based on values.

[ehsantn]

TODO: create a list of situation that don't use nullable types yet and track them.

[jbrockmendel]

-1. Nullable dtypes are much less performant than their numpy counterparts.

[datapythonista]

-1. Nullable dtypes are much less performant than their numpy counterparts.

I'm not sure, but I think there may be a misunderstanding here. Just to make sure we're all talking about the same, I see two different cases:

Case 1

>>> pandas.Series([1, 2, 3]).dtype
dtype('int64')  # is this issue proposing to make this Int64Dtype()?

@ehariri can you confirm if you'd want this as part of this proposal?

Case 2

>>> pandas.Series([True, False, None]).dtype
dtype('O')  # use BooleanDtype instead

@jbrockmendel I think your comment refers to Case 1, can you confirm if you're also -1 on this?

Also, @ehariri would it make sense to move the read_csv discussion to a separate issue, and expand on it? At least to me seems to be a related but different topic, and I'm not sure I fully understand the reasoning and advantages of it (as the whole schema is unknown and inferred from the csv values).

[jbrockmendel]

I think your comment refers to Case 1, can you confirm if you're also -1 on this?

Case 1 I'd definitely be -1 on changing. Case 2 I'd be more open to changing, but reticent bc None and pd.NA have different semantics.

[ehsantn]

I think it'd be the most robust to have just nullable types (e.g. to avoid unwanted type mismatch issues). The performance issue shouldn't be fundamental (as Arrow implements for example).

For JIT compilation, the most critical scenario is when output type depends on input values, not just input types. Pandas conversion of int to float to handle NAs is a clear example. In this code, the data type of merge output depends on values:

In [32]: df1 = pd.DataFrame({'A': [1, 2, 3]})

In [33]: df2 = pd.DataFrame({'A': [1, 2, 3], "B": [3, 4, 5]})

In [34]: df1.merge(df2, how="left")
Out[34]:
   A  B
0  1  3
1  2  4
2  3  5

In [35]: df2 = pd.DataFrame({'A': [1, 2], "B": [3, 4]})

In [36]: df1.merge(df2, how="left")
Out[36]:
   A    B
0  1  3.0
1  2  4.0
2  3  NaN

Here, the JIT compiler can't decide to use int or float data types (currently we default to Int64 which doesn't match Pandas):

In [37]: @bodo.jit
    ...: def f():
    ...:     df1 = pd.read_parquet("file1.pq")
    ...:     df2 = pd.read_parquet("file2.pq")
    ...:     return df1.merge(df2, how="left")