Add Polars pydantic integration with format support and native JSON schema generation #1979
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
halicki
commented
Apr 23, 2025
halicki
commented
- Add pydantic validation for Polars DataFrames and LazyFrames
- Implement DataFrame type conversion from various formats (dict, CSV, JSON, Parquet, Feather)
- Replace pandas dependency with native Polars JSON schema generation
- Support both Pydantic v1 and v2 with appropriate validators
- Add comprehensive test suite for the integration
…chema generation - Add pydantic validation for Polars DataFrames and LazyFrames - Implement DataFrame type conversion from various formats (dict, CSV, JSON, Parquet, Feather) - Replace pandas dependency with native Polars JSON schema generation - Support both Pydantic v1 and v2 with appropriate validators - Add comprehensive test suite for the integration Signed-off-by: Arkadiusz Halicki <[email protected]>
halicki
force-pushed
the
polars-pydantic-integration
branch
from
1ccf9cc to
52fbe10
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1979 +/- ##
==========================================
- Coverage 94.28% 93.84% -0.45%
==========================================
Files 91 121 +30
Lines 7013 9742 +2729
==========================================
+ Hits 6612 9142 +2530
- Misses 401 600 +199 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
hi @halicki ! looks like there are some failing linting issues. Feel free to disable the pylint issues inline. |
Signed-off-by: Arkadiusz Halicki <[email protected]>
halicki
force-pushed
the
polars-pydantic-integration
branch
from
8f0755d to
5ef1cb4
Compare
Signed-off-by: cosmicBboy <[email protected]>
|
hey @halicki it looks like there are several areas in |
|
Hey, @cosmicBboy. Sure, but I'll probably take it next week. |
|
@halicki looks like all the unit tests are passing, now just need to make the linters happy |
- Add new test file test_polars_typing.py with complete coverage for polars typing module - Test DataFrame.from_format and to_format with various data formats - Cover both success and error paths - Add tests for Pydantic integration - Add pragmas to conditionally exclude import-time and version-specific code from coverage Achieves 100% test coverage for the module. Signed-off-by: Arkadiusz Halicki <[email protected]>
Tests now provide sufficient coverage without the need for pragma directives. Signed-off-by: Arkadiusz Halicki <[email protected]>
- Created comprehensive test suite for typing/polars.py - Added tests for DataFrame, LazyFrame, and Series classes - Added tests for format conversion methods - Added tests for Pydantic integration (v1 and v2) - Added pragma no cover to hard-to-test code paths - Achieved 100% test coverage Signed-off-by: Arkadiusz Halicki <[email protected]>
Signed-off-by: Arkadiusz Halicki <[email protected]>
Signed-off-by: Arkadiusz Halicki <[email protected]>
halicki
force-pushed
the
polars-pydantic-integration
branch
from
5ce1d7c to
a456b69
Compare
This commit adds test cases that demonstrate how to use optional columns with Polars DataFrameModels when integrating with Pydantic. The tests show: - Using Optional[Series[type]] annotation to make a column optional - Validating DataFrames with and without the optional column - Ensuring type validation still works on optional columns when present - Verifying that required columns still must be present These tests help document the supported patterns for optional columns in Pandera's Polars integration. Signed-off-by: Arkadiusz Halicki <[email protected]>
Signed-off-by: Arkadiusz Halicki <[email protected]>
Signed-off-by: Arkadiusz Halicki <[email protected]>
|
My part works. Problems are in pyspark and doc builds. |
|
@halicki thanks, will merge this and address pyspark issues separately. |
cosmicBboy
approved these changes
May 20, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.