Donate Ray SQL to the DataFusion Project (#1)

* Initial commit

* Basic project structure

* gitignore

* Add protobuf plumbing (#2)

* Implement protobuf codec

* Wire up query execution (#4)

* query runs end to end (#5)

* re-organize python code (#6)

* Implement shuffle more fully (#7)

* update README (#8)

* Bug fix (#9)

* Support multiple shuffle partitions (#10)

* More shuffle fixes (#11)

* fix readme (#12)

* add perf chart (#13)

* Remove hard-coded temp dir (#14)

* bug fix (#15)

* New results (#16)

* Upgrade to DataFusion 17, fix a couple of bugs, add some tests (#18)

* Remove debug logging (#19)

* update README (#21)

* Make better use of futures (#23)

* Documentation & bug fixes (#24)

* Update README.md

* [WIP] Use Ray object store for shuffle exchange (#28)

* Fixes for Ray-based shuffle (#29)

* Small fixes for Context (#30)

* Make distributed execution work (#33)

* Make distributed execution work

* fix tips.py

* fixes; incorporate changes from #32

* Upgrade to DataFusion 20 (#31)

* Add support for DDL statements, such as `CREATE VIEW` (#35)

* Experimenting with supporting DDL

* update docs

* Use PyArrow for zero-copy interaction with the Ray Object Store (#36)

* Optimize Ray shuffle with zero-copy object store

* remove more clones

* change bytes to pyarrow.array

* revert /tmp

* remove empty_result_set

* remove empty_result_set

* Fix input partition count bug

* Add Frank as author (#37)

* fix hyperlink of issue 22 in docs/README.txt (#40)

Co-authored-by: ivanfan <[email protected]>

* delta lake and iceberg table support (#43)

* delta support

* imports

* Update DataFusion version to 28.0.0 (#41)

* Update DataFusion version

* update example

* Upgrade to DataFusion 33 (#45)

* Upgrade to DataFusion 33

* undo release profile change

* Add basic GitHub workflow to compile code (#47)

* Create rust.yml

* install protobuf

* fix

* fix

* fix

* fix

* fix

* fix

* Add ASF license header

Signed-off-by: Austin Liu <[email protected]>

* Remove ASF header for generated code

Signed-off-by: Austin Liu <[email protected]>

---------

Signed-off-by: Austin Liu <[email protected]>
Co-authored-by: Andy Grove <[email protected]>
Co-authored-by: Frank Luan <[email protected]>
Co-authored-by: Frank Luan <[email protected]>
Co-authored-by: Ivankings <[email protected]>
Co-authored-by: ivanfan <[email protected]>
Co-authored-by: raviranak <[email protected]>

Author: 6 people

.github/workflows/rust.yml

@@ -0,0 +1,25 @@
+name: Rust
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Install protobuf compiler
+      shell: bash
+      run: sudo apt-get install protobuf-compiler
+    - name: Build Rust code
+      run: cargo build --verbose
+    - name: Run tests
+      run: cargo test --verbose

.gitignore

@@ -0,0 +1,7 @@
+.idea
+target
+__pycache__
+venv
+*.so
+*.log
+results-sf*