feat(pyspark)!: forward kwargs in create_table to pyspark methods
#11120
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…k methods Forward kwargs in `create_table` method to respective pypsark methods. Also, we removed `partition_by` and `format` kwargs of `create_table` in favor of forwarding kwargs directly to pyspark methods. This is consistent with `to_delta` and `to_parquet` for pyspark backend. This is breaking for users of `partition_by` kwarg only in `create_table`, which will now need to be passed as `partitionBy`. closes ibis-project#10984
github-actions
bot
added
tests
|
ACTION NEEDED Ibis follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. See https://github.com/ibis-project/ibis/blob/main/.releaserc.js The commitlint output is: |
|
Still running into testing issues w/ spark connect. Looks like it is an issue specific to delta. I am looking into it, but lmk if anything is immediately obvious. Update: See #11123 for working solution on testing issues |
|
+1, but let's cut one more feature release (10.5) before shipping this PR in 11.0! Appreciate the work here! |
cpcloud
added a commit
that referenced
this pull request
Apr 14, 2025
…#11123) ## Description of changes 1. Upgrade `delta-spark` from v3.2.1 to [v3.3.0](https://github.com/delta-io/delta/releases/tag/v3.3.0) for pyspark backend testing in github actions 2. Upgrade `delta-spark` [maven package](https://mvnrepository.com/artifact/io.delta/delta-spark) from v.2.1.0 to v3.3.0 in spark-connect container 🐳 - Ensure consistency w/ local pyspark testing & [compatibility](https://docs.delta.io/latest/releases.html#compatibility-with-apache-spark) with pyspark v3.5.5 3. Upgrade spark-connect configuration to enable proper delta & catalog functionality ---- - **2** + **3** together resolved spark-connect testing issues from #11120 --------- Co-authored-by: Phillip Cloud <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
Follow-up on feat(pyspark): expose merge_schema option in create_table #11071
Forward kwargs in
create_tablemethod to respective pyspark methods - pyspark.sql.DataFrameWriter.saveAsTableif
objis passed or pyspark.sql.Catalog.createTable ifschemais passed.modekwarg is passed, it will take precedence overoverwrite.Changes are to improve api consistency with
to_deltaandto_parquetfor pyspark backend, while increasing flexibility ofcreate_tablemethod in correspondence with pyspark.sql.DataFrameWriter.saveAsTableBreaking: We removed
partition_byandformatkwargs ofcreate_tablein favorof forwarding kwargs directly to pyspark methods. This is breaking for users of
partition_bykwarg increate_table, which will now need to be passed aspartitionByTesting:
test_create_table_kwargsdemonstrates & tests some general usage patterns ofcreate_tablemethod. Note, in example 2, whenappendis passed with an input table of same schema as db table, this behavior will be similar to that ofinsertmethod.create_table&pyspark.sql.DataFrameWriter.saveAsTableandinsertandpyspark.sql.DataFrameWriter.insertIntofor pyspark usersIssues closed