fix(pandas): implement-automatic-conversion-for-pandas-null-types

[fvaleye]

Description

This PR fixes type inconsistencies in Pandas DataFrames by implementing automatic conversion of object dtype columns containing null values into their correct schema-defined types (when available).

Previously, object dtype columns with null values could remain untyped, leading to mismatches with schema-defined types and causing validation or serialization errors.

Now, these columns are automatically converted, ensuring consistency between DataFrame values and the schema while reducing type-related issues.

Related Issue(s)

• closes Python: Automatically convert Pandas null types to valid Delta Lake types in write_deltalake() #3691

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.37%. Comparing base (ef9e077) to head (dcb808d).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3695   +/-   ##
=======================================
  Coverage   75.36%   75.37%           
=======================================
  Files         144      144           
  Lines       43607    43607           
  Branches    43607    43607           
=======================================
+ Hits        32866    32868    +2     
+ Misses       9133     9131    -2     
  Partials     1608     1608           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull Request Overview

This PR implements automatic conversion of Pandas DataFrames containing null-typed columns to match their corresponding schema-defined types in Delta Lake. This fixes type inconsistencies that occur when object dtype columns with null values are converted to null types in PyArrow, which can cause validation or serialization errors when appending to existing tables.

• Modifies the _convert_arro3_schema_to_delta function to accept an optional existing schema parameter for null type conversion
• Updates the writer to pass the existing table schema when available for type conversion
• Adds comprehensive test coverage for null column conversion scenarios

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
python/deltalake/writer/writer.py	Passes existing table schema to conversion function for null type handling
python/deltalake/writer/_conversion.py	Implements null type conversion logic with recursion prevention
python/tests/test_conversion.py	Adds test cases for null column conversion and edge cases

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[FrankPortman]

Thanks for jumping on this so quickly!

[ion-elgreco]

I prefer we also check here if is not None, it's s bit more clear. Rest of code lgtm!

[fvaleye]

Do you mean:
if dtype and DataType.is_null(dtype) and existing_schema and field_name:

[ion-elgreco]

No, rather field_,name is not None and schema is not none

[fvaleye]

I see!
Thanks for your review @ion-elgreco 🙏
Let's merge this after this change!