enh: Passing arguments to `pa.Table.to_pandas`

[MarcoGorelli]

Discussed in #2121

^{Originally posted by dhirschfeld February 28, 2025}
In my current code I have a very large pa.Table that I want to convert to pd.DataFrame. In the to_pandas call I specify split_blocks and self_destruct as I don't want 2 copies of the data in memory.

I'm thinking of porting this code to narwhals so was wondering if there's a way to do the same - convert to a pa.Table to different format without doubling the memory requirements?

I'm interest in converting to either pandas or polars hence why I thought narwhals might be a good fit.

[dangotbanned]

I think improved performance characteristics are a good reason to not unify things like (#2056 (comment)).

If anyone wants to pursue this - I've added a branch with the typing tools to support it main...pa-table-to-pandas

narwhals/narwhals/_arrow/typing.py

Lines 61 to 79 in 112cfd9

	class ArrowTableToPandasKwds(TypedDict, total=False):
	"""Keyword arguments for [pyarrow.Table.to_pandas](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_pandas)."""
	memory_pool: pa.MemoryPool | None
	categories: list[Any] | None
	strings_to_categorical: bool
	zero_copy_only: bool
	integer_object_nulls: bool
	date_as_object: bool
	timestamp_as_object: bool
	use_threads: bool
	deduplicate_objects: bool
	ignore_metadata: bool
	safe: bool
	split_blocks: bool
	self_destruct: bool
	maps_as_pydicts: Literal["None", "lossy", "strict"] | None
	types_mapper: Callable[[pa.DataType], PandasExtensionDtype | None] | None
	coerce_temporal_nanoseconds: bool

You'd still need to add the runtime handling of providing the wrong keywords.
Statically - they'd only appear for pa.Table:

narwhals/narwhals/dataframe.py

Lines 658 to 665 in 112cfd9

	@overload
	def to_pandas(
	self: DataFrame[pa.Table], **kwds: Unpack[ArrowTableToPandasKwds]
	) -> pd.DataFrame: ...
	@overload
	def to_pandas(self: Self, **kwds: Any) -> pd.DataFrame: ...
	def to_pandas(self: Self, **kwds: Any) -> pd.DataFrame:
	"""Convert this DataFrame to a pandas DataFrame.