Add support for nested data types (lists and dictionaries)

[DanielAvdar]

Feature Request: Support for Nested Data Types

Description

I would like to request support for converting nested data types like lists and dictionaries when using the pandas PyArrow backend. Currently, the library handles primitive types, datetime types, and some object types well, but there's no explicit support for nested data structures.

Use Case

Working with hierarchical or structured data that contains:

• Lists of values within a DataFrame cell
• Dictionary/map structures within a DataFrame cell
• Nested combinations of the above

Expected Behavior

When calling convert_to_pyarrow() on a DataFrame with columns containing lists or dictionaries, they should be properly converted to their corresponding PyArrow types:

• Lists should convert to list[pyarrow] type
• Dictionaries should convert to struct[pyarrow] or map[pyarrow] types

Example

import pandas as pd
from pandas_pyarrow import convert_to_pyarrow

# Create a pandas DataFrame with nested data types
df = pd.DataFrame({
    'A': [1, 2, 3],
    'B': [['a', 'b'], ['c', 'd'], ['e', 'f']],  # list column
    'C': [{'x': 1}, {'y': 2}, {'z': 3}]  # dict column
})

# Convert to PyArrow dtypes
adf = convert_to_pyarrow(df)

# Expected output should properly handle the nested types
print(adf.dtypes)

Technical Considerations

• PyArrow supports list and struct types that could map to Python's lists and dictionaries
• May need to handle type inference for nested structures
• Might require recursive handling for deeply nested structures

Benefits

Adding support for these data types would make the library more versatile for real-world data analysis scenarios where hierarchical or structured data is common.

References

• PyArrow documentation on nested data types
• Pandas with PyArrow backend documentation

I appreciate your consideration of this feature request!

[DanielAvdar]

Implementation Recommendations

After examining the codebase, I can provide more specific recommendations for implementing nested data types support:

1. Update Mapper Module

In pandas_pyarrow/mappers/dtype_mapper.py, a new mapper dictionary specifically for nested types should be created:

mapper_dict_nested: Dict[str, str] = {
    "list-dtype": "list[pyarrow]",
    "dict-dtype": "struct[pyarrow]",
    # Consider map type as well
}

Then incorporate this in the create_mapper() function in pandas_pyarrow/mappers/__init__.py.

2. Type Detection

The converter needs to detect when a column contains lists or dictionaries. This likely requires:

• Type inference for columns with mixed object types
• Special handling for nested data structures that pandas normally stores as object dtype

3. Testing Strategy

Explicit Tests

Add dedicated test files:

1. tests/unit/test_nested_types.py for basic functionality:
   • Test conversion of lists to list[pyarrow]
   • Test conversion of dictionaries to struct[pyarrow]
   • Test handling of nested structures (e.g., lists of dictionaries)
   • Test edge cases like empty lists/dicts and mixed types

Property-Based Testing

Extend the property-based testing infrastructure:

1. In tests/unit/property_based/:
   • Create test_nested.py with Hypothesis strategies for:
     • Lists of various data types
     • Dictionaries with different key-value combinations
     • Nested combinations
   • Test roundtrip conversions (original → pyarrow → original)
   • Test consistency of behavior across different data shapes and sizes

Practical Integration Tests:

Ensure real-world use cases work:

• Test with pandas Series/DataFrames containing nested data from common sources
• Test interoperability with other libraries that use nested data structures
• Test performance with large datasets containing nested structures

Currently, these nested types are converted to string[pyarrow], which loses the structural information and makes it impossible to perform operations that depend on the nested nature of the data.

I believe this structured approach to testing will ensure robust implementation of nested data type support in pandas-pyarrow.