Upgrading to DF41 #7
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
austin362667
added a commit
to austin362667/datafusion-ray
that referenced
this pull request
Oct 2, 2024
* Initial commit * Basic project structure * gitignore * Add protobuf plumbing (apache#2) * Implement protobuf codec * Wire up query execution (apache#4) * query runs end to end (apache#5) * re-organize python code (apache#6) * Implement shuffle more fully (apache#7) * update README (apache#8) * Bug fix (apache#9) * Support multiple shuffle partitions (apache#10) * More shuffle fixes (apache#11) * fix readme (apache#12) * add perf chart (apache#13) * Remove hard-coded temp dir (apache#14) * bug fix (apache#15) * New results (apache#16) * Upgrade to DataFusion 17, fix a couple of bugs, add some tests (apache#18) * Remove debug logging (apache#19) * update README (apache#21) * Make better use of futures (apache#23) * Documentation & bug fixes (apache#24) * Update README.md * [WIP] Use Ray object store for shuffle exchange (apache#28) * Fixes for Ray-based shuffle (apache#29) * Small fixes for Context (apache#30) * Make distributed execution work (apache#33) * Make distributed execution work * fix tips.py * fixes; incorporate changes from apache#32 * Upgrade to DataFusion 20 (apache#31) * Add support for DDL statements, such as `CREATE VIEW` (apache#35) * Experimenting with supporting DDL * update docs * Use PyArrow for zero-copy interaction with the Ray Object Store (apache#36) * Optimize Ray shuffle with zero-copy object store * remove more clones * change bytes to pyarrow.array * revert /tmp * remove empty_result_set * remove empty_result_set * Fix input partition count bug * Add Frank as author (apache#37) * fix hyperlink of issue 22 in docs/README.txt (apache#40) Co-authored-by: ivanfan <[email protected]> * delta lake and iceberg table support (apache#43) * delta support * imports * Update DataFusion version to 28.0.0 (apache#41) * Update DataFusion version * update example * Upgrade to DataFusion 33 (apache#45) * Upgrade to DataFusion 33 * undo release profile change * Add basic GitHub workflow to compile code (apache#47) * Create rust.yml * install protobuf * fix * fix * fix * fix * fix * fix * Add ASF license header Signed-off-by: Austin Liu <[email protected]> * Remove ASF header for generated code Signed-off-by: Austin Liu <[email protected]> --------- Signed-off-by: Austin Liu <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Frank Luan <[email protected]> Co-authored-by: Frank Luan <[email protected]> Co-authored-by: Ivankings <[email protected]> Co-authored-by: ivanfan <[email protected]> Co-authored-by: raviranak <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is primarily an update to DataFusion 41.0.0.
Added in is a pre-commit file which was run on the files changed in this PR.