Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrading to DF41 #7

Merged
merged 1 commit into from
Sep 30, 2024
Merged

Upgrading to DF41 #7

merged 1 commit into from
Sep 30, 2024

Conversation

timsaucer
Copy link
Contributor

This PR is primarily an update to DataFusion 41.0.0.

Added in is a pre-commit file which was run on the files changed in this PR.

Copy link
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @timsaucer

@andygrove andygrove merged commit 78defa4 into apache:main Sep 30, 2024
1 check passed
austin362667 added a commit to austin362667/datafusion-ray that referenced this pull request Oct 2, 2024
* Initial commit

* Basic project structure

* gitignore

* Add protobuf plumbing (apache#2)

* Implement protobuf codec

* Wire up query execution (apache#4)

* query runs end to end (apache#5)

* re-organize python code (apache#6)

* Implement shuffle more fully (apache#7)

* update README (apache#8)

* Bug fix (apache#9)

* Support multiple shuffle partitions (apache#10)

* More shuffle fixes (apache#11)

* fix readme (apache#12)

* add perf chart (apache#13)

* Remove hard-coded temp dir (apache#14)

* bug fix (apache#15)

* New results (apache#16)

* Upgrade to DataFusion 17, fix a couple of bugs, add some tests (apache#18)

* Remove debug logging (apache#19)

* update README (apache#21)

* Make better use of futures (apache#23)

* Documentation & bug fixes (apache#24)

* Update README.md

* [WIP] Use Ray object store for shuffle exchange (apache#28)

* Fixes for Ray-based shuffle (apache#29)

* Small fixes for Context (apache#30)

* Make distributed execution work (apache#33)

* Make distributed execution work

* fix tips.py

* fixes; incorporate changes from apache#32

* Upgrade to DataFusion 20 (apache#31)

* Add support for DDL statements, such as `CREATE VIEW` (apache#35)

* Experimenting with supporting DDL

* update docs

* Use PyArrow for zero-copy interaction with the Ray Object Store (apache#36)

* Optimize Ray shuffle with zero-copy object store

* remove more clones

* change bytes to pyarrow.array

* revert /tmp

* remove empty_result_set

* remove empty_result_set

* Fix input partition count bug

* Add Frank as author (apache#37)

* fix hyperlink of issue 22 in docs/README.txt (apache#40)

Co-authored-by: ivanfan <[email protected]>

* delta lake and iceberg table support (apache#43)

* delta support

* imports

* Update DataFusion version to 28.0.0 (apache#41)

* Update DataFusion version

* update example

* Upgrade to DataFusion 33 (apache#45)

* Upgrade to DataFusion 33

* undo release profile change

* Add basic GitHub workflow to compile code (apache#47)

* Create rust.yml

* install protobuf

* fix

* fix

* fix

* fix

* fix

* fix

* Add ASF license header

Signed-off-by: Austin Liu <[email protected]>

* Remove ASF header for generated code

Signed-off-by: Austin Liu <[email protected]>

---------

Signed-off-by: Austin Liu <[email protected]>
Co-authored-by: Andy Grove <[email protected]>
Co-authored-by: Frank Luan <[email protected]>
Co-authored-by: Frank Luan <[email protected]>
Co-authored-by: Ivankings <[email protected]>
Co-authored-by: ivanfan <[email protected]>
Co-authored-by: raviranak <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants