Repository navigation

[SPARK-60101][SQL] Support native data sources implemented in Rust or C++ - #59313

Open

HyukjinKwon wants to merge 12 commits into

apache:masterfrom

HyukjinKwon:SPARK-60101

Commits on Oct 9, 2026
- [SPARK-60101][SQL] Add a columnar data source API for JVM and native (Rust, C++) data sources
  
  HyukjinKwon
  and
  Isaac
  committed
  56acc8b
- Make NativeBridge the only public API of native data sources
  
  HyukjinKwon
  and
  Isaac
  committed
  b032a0a
- Pass the options of catalog tables to native data sources
  
  HyukjinKwon
  and
  Isaac
  committed
  1fc3fc7
- Find the native data source packages installed in SPARK_HOME automatically
  
  HyukjinKwon
  and
  Isaac
  committed
  aae6a4f
- Fix the Connect client compatibility check and the documentation build
  
  HyukjinKwon
  and
  Isaac
  committed
  a5d6708
- Find installed native data source libraries automatically
  
  HyukjinKwon
  and
  Isaac
  committed
  4a2026e
- Format NativeDataSourceConnectSuite with scalafmt
  
  HyukjinKwon
  and
  Isaac
  committed
  822d33c
Commits on Oct 10, 2026
- Address review comments on native data source lookup, distribution and lifecycle
  
  HyukjinKwon
  and
  Isaac
  committed
  9dd4f88
- Address more review comments on native data source packages and stream writers
  
  HyukjinKwon
  and
  Isaac
  committed
  3814147
- Address review comments on sessions, distributed copies, schemas and batching
  
  HyukjinKwon
  and
  Isaac
  committed
  9ac5a49
- Check the Arrow data of native data sources, create writers late, push limits once
  
  HyukjinKwon
  and
  Isaac
  committed
  acdc7eb
Commits on Oct 11, 2026
- Check the lengths of the arrays that native data sources export
  
  HyukjinKwon
  and
  Isaac
  committed
  a8f178b

Navigation Menu

Repository navigation

[SPARK-60101][SQL] Support native data sources implemented in Rust or C++ - #59313

[SPARK-60101][SQL] Support native data sources implemented in Rust or C++#59313
HyukjinKwon wants to merge 12 commits into
apache:masterfrom
HyukjinKwon:SPARK-60101

Commits on Oct 9, 2026

[SPARK-60101][SQL] Add a columnar data source API for JVM and native (Rust, C++) data sources

Make NativeBridge the only public API of native data sources

Pass the options of catalog tables to native data sources

Find the native data source packages installed in SPARK_HOME automatically

Fix the Connect client compatibility check and the documentation build

Find installed native data source libraries automatically

Format NativeDataSourceConnectSuite with scalafmt

Commits on Oct 10, 2026

Address review comments on native data source lookup, distribution and lifecycle

Address more review comments on native data source packages and stream writers

Address review comments on sessions, distributed copies, schemas and batching

Check the Arrow data of native data sources, create writers late, push limits once

Commits on Oct 11, 2026

Check the lengths of the arrays that native data sources export