Skip to content

feat: Implement Python ODBC, MySQL, Postgres, Flight table providers#281

Merged
phillipleblanc merged 18 commits into
datafusion-contrib:mainfrom
crystalxyz:python-odbc
Apr 10, 2025
Merged

feat: Implement Python ODBC, MySQL, Postgres, Flight table providers#281
phillipleblanc merged 18 commits into
datafusion-contrib:mainfrom
crystalxyz:python-odbc

Conversation

@crystalxyz
Copy link
Copy Markdown
Contributor

@crystalxyz crystalxyz commented Mar 24, 2025 •
edited
Loading

Addresses part of #279

Summary

This feature implements multiple Python table providers as well as corresponding examples.

Details

  • Implemented ODBC Rust FFI and Python interface
  • Implemented MySQL Rust FFI and Python interface
  • Implemented Postgres Rust FFI and Python interface
  • Implemented Flight Rust FFI and Python interface
  • Did minor refactoring of get_tokio_runtime to share it among all Python table providers.
  • Updated documentation to set up ODBC on ARM Mac and to run examples.

Test

  • Existing Rust tests have all passed
  • Verified that all the tests in /python/examples work

Next step

Add integration testing

crystalxyz
crystalxyz commented Mar 24, 2025
View reviewed changes
Comment thread python/src/odbc.rs
Comment thread python/src/odbc.rs
Comment thread python/src/odbc.rs
Comment thread python/src/odbc.rs
@crystalxyz crystalxyz marked this pull request as ready for review March 24, 2025 19:10
@crystalxyz
Copy link
Copy Markdown
Contributor Author

crystalxyz commented Mar 24, 2025

@timsaucer If you'd like to take a look!

@crystalxyz crystalxyz changed the title feat: Implement Python ODBC table provider feat: Implement Python ODBC, MySQL, Postgres, Flight table providers Mar 30, 2025
phillipleblanc
phillipleblanc reviewed Apr 9, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@phillipleblanc phillipleblanc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @crystalxyz this is excellent. I've left a few minor comments, but I think this PR is nearly ready to go.

Comment thread core/src/flight/exec.rs Outdated
Comment thread core/src/sql/db_connection_pool/runtime.rs Outdated
Comment thread python/python/datafusion_table_providers/odbc.py Outdated
Comment thread python/src/odbc.rs
Comment thread python/src/odbc.rs
Comment thread python/src/odbc.rs
Comment thread core/src/sql/db_connection_pool/postgrespool.rs
phillipleblanc
phillipleblanc reviewed Apr 10, 2025
View reviewed changes
Comment thread core/src/sql/db_connection_pool/runtime.rs Outdated
@phillipleblanc phillipleblanc added the enhancement New feature or request label Apr 10, 2025
@phillipleblanc phillipleblanc added this to the v0.4.0 milestone Apr 10, 2025
@phillipleblanc phillipleblanc merged commit d8276cd into datafusion-contrib:main Apr 10, 2025
3 checks passed
@phillipleblanc
Copy link
Copy Markdown
Collaborator

phillipleblanc commented Apr 10, 2025

@crystalxyz Thank you for this PR! This is epic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@phillipleblanc phillipleblanc phillipleblanc approved these changes

Assignees

@crystalxyz crystalxyz

Labels

enhancement New feature or request

Projects

None yet

Milestone

v0.4.0

Development

Successfully merging this pull request may close these issues.

2 participants

@crystalxyz @phillipleblanc