[Draft] Try to add python table provider interface

[CrystalZhou0529]

Expose table providers in Python via FFI

• Forked from Initial commit for python table providers #194 and fixed python dependencies

Summary

1. Created python subcrate for python-related development
2. Reorganized Cargo dependency and share common dependencies (arrow, datafusion, duckdb) in workspace cargo.toml
3. Implemented sqlite and duckdb table provider interfaces and tested in examples
4. Updated duckdbconn.rs to start new tokio runtime when triggered from PyO3
5. Updated Github CICD testing command

Testing

• Verified that all existing tests in Rust have passed
• Verified that Python examples can execute correctly via the following script:

cd python
maturin develop
cd examples
python3 duckdb_demo.py

Follow-up works

• Improve duckdb enum interface (ongoing)

The rest will be left for future works:

• Python ODBC table provider + example
• Python Postgres table provider + example
• Python MySQL table provider + example
• Integration testing
• Finalize Python interface design, i.e. new_memory() or new(":memory:")

[timsaucer]

Overall, this looks very nice. I suggest we merge this in but do not yet publish to pypi. There should be no breaking changes to the rust code, so anyone who depends on it should be okay. I think we will need to bump the minor version number if you haven't done so (I didn't check).

@phillipleblanc Would you be comfortable if we merged in partially complete work on the python side so we can have a series of smaller PRs until we're ready to publish?

[timsaucer]

I recommend we put all of the arrow dependencies in one place - the workspace Cargo.toml. That way when someone goes to update them, they don't miss one easily.

[timsaucer]

Similar comment as above, recommend putting all datafusion dependencies in workspace Cargo.toml

[timsaucer]

One thing I'm not sure about - do we want this crate (the python crate) published to crates.io or not? @phillipleblanc do you have an opinion? I don't expect there would be a reason for someone to need it as a dependency.

[phillipleblanc]

Let's not publish it for now - if someone needs it later we can always publish it then.

[timsaucer]

Great! @CrystalZhou0529 we want a publish = false line like in this example from datafusion: https://github.com/apache/datafusion/blob/main/datafusion-examples/Cargo.toml

[CrystalZhou0529]

Sounds good!

[timsaucer]

Can you double check which version of datafusion-python first has the tokio runtime? I think it might be 44. It will be important.

[CrystalZhou0529]

If I understand correctly it is in 43? Ref: https://github.com/apache/datafusion-python/blob/b8dd97bc8eefcfecfa8dcc864c4898c654b236a9/dev/changelog/pre-43.0.0.md?plain=1#L53

[timsaucer]

This is the commit where the tokio runtimes were added on the FFI side: apache/datafusion#13937 and that is in datafusion 45. I think we need that version as the minimum.

[CrystalZhou0529]

Fixed!

[phillipleblanc]

Overall, this looks very nice. I suggest we merge this in but do not yet publish to pypi. There should be no breaking changes to the rust code, so anyone who depends on it should be okay. I think we will need to bump the minor version number if you haven't done so (I didn't check).

@phillipleblanc Would you be comfortable if we merged in partially complete work on the python side so we can have a series of smaller PRs until we're ready to publish?

Yes, that works for me - just mark this PR as ready to review and tag me when you are ready to merge. Thanks for working on this!