Skip to content

Create a method to return a spark df for pyspark backend #10673

Open
@sonic-x11

Description

@sonic-x11

Is your feature request related to a problem?

No response

What is the motivation behind your request?

If we return a spark df the user can write to various formats as spark supports in my case it was hudi

Describe the solution you'd like

Create a simple to_spark or similar method which returns a spark df

currently there is hacky workaround pointed out by @gforsyth

#10671 (comment)

What version of ibis are you running?

9.5

What backend(s) are you using, if any?

Pyspark and duckdb

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureFeatures or general enhancements

    Type

    No type

    Projects

    • Status

      backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions