Skip to content

Partition overwrite strategy for DuckDB #7207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

begelundmuller
Copy link
Contributor

@begelundmuller begelundmuller commented Apr 23, 2025

Example usage:

type: model
partitions:
  glob: s3://bucket/path/**.parquet
sql: SELECT '{{.partition.path}}' as partition_id, * FROM read_parquet('{{.partition.uri}}')
output:
  incremental_strategy: partition_overwrite
  partition_by: partition_id

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

@begelundmuller begelundmuller self-assigned this Apr 23, 2025
@rilldata rilldata deleted a comment from bito-code-review bot Apr 23, 2025
@begelundmuller begelundmuller requested a review from k-anshul April 25, 2025 15:24
@begelundmuller begelundmuller marked this pull request as ready for review April 25, 2025 15:24
@begelundmuller begelundmuller changed the title WIP: Partition overwrite strategy for DuckDB Partition overwrite strategy for DuckDB Apr 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant