datafusion-remote-table

Features

1. Execute SQL queries on remote databases and stream results as datafusion table provider
2. Support inferring schema or user specified schema
3. Support pushing down filters and limit to remote databases
4. Execution plan can be serialized for distributed execution
5. Record batches can be transformed before outputting to next plan node

Usage

#[tokio::main]
pub async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let options = PostgresConnectionOptions::new("localhost", 5432, "user", "password");
    let remote_table = RemoteTable::try_new(options, "select * from supported_data_types").await?;

    let ctx = SessionContext::new();
    ctx.register_table("remote_table", Arc::new(remote_table))?;

    ctx.sql("select * from remote_table").await?.show().await?;

    Ok(())
}

Supported databases

• Postgres
  • Int2 / Int4 / Int8
  • Float4 / Float8 / Numeric
  • Char / Varchar / Text / Bpchar / Bytea
  • Date / Time / Timestamp / Timestamptz / Interval
  • Bool / Oid / Name / Json / Jsonb / Geometry(PostGIS)
  • Int2[] / Int4[] / Int8[]
  • Float4[] / Float8[]
  • Char[] / Varchar[] / Bpchar[] / Text[] / Bytea[]
• MySQL
  • TinyInt (Unsigned) / Smallint (Unsigned) / MediumInt (Unsigned) / Int (Unsigned) / Bigint (Unsigned)
  • Float / Double / Decimal
  • Date / DateTime / Time / Timestamp / Year
  • Char / Varchar / Binary / Varbinary
  • TinyText / Text / MediumText / LongText
  • TinyBlob / Blob / MediumBlob / LongBlob
  • Json / Geometry
• Oracle
  • Number / BinaryFloat / BinaryDouble / Float
  • Varchar2 / NVarchar2 / Char / NChar / Long / Clob / NClob
  • Raw / Long Raw / Blob
  • Date / Timestamp
  • Boolean
• SQLite
  • Null / Integer / Real / Text / Blob
• DM (达梦数据库)
  • TinyInt / Smallint / Int / Bigint
  • Real / Float / Double / Numeric / Decimal
  • Char / Varchar / Text
  • Binary / Varbinary / Image
  • Bit / Timestamp / Time / Date

Thanks

• datafusion-table-providers