[Umbrella][Feature] Support spark engine #155
Description
Search before asking
- I searched in the issues and found nothing similar.
Motivation
In roadmap,we plan to support the spark engine.
for kv table spark can support micro batch read, batch read, streaming writer and batch writer.
for log table spark can support micro batch read, streaming writer and batch writer.
According to @wuchong‘s suggestion,i will create some sub-tasks.
Task list
- spark catalog [Feature] Support spark catalog(base spark 3.3.3) #228
- streaming read @Alibaba-HZY
- batch read
- union read
- streaming write
- batch write
Solution
In task "streaming read" I'll implement it and introduce some base classes
such as
Introduce flussSparkTable implements org.apache.spark.sql.connector.catalog.Table
Introduce SparkCatalog implements org.apache.spark.sql.connector.catalog.TableCatalog
Introduce SparkInternalRow and SparkTypeUtils
Anything else?
No response
Willingness to contribute
- I'm willing to submit a PR!