基于spark 数据集成工具datatunnel，支持doris

[melin]

Search before asking

• I had searched in the issues and found no similar issues.

Description

一些海外客户，在使用datatunnel。已有一个买咖啡客户，基于dataunnel 读写redshift，最近在测试doris。

https://github.com/melin/datatunnel/tree/master/examples/src/main/kotlin/com/superior/datatunnel/examples/doris

DATATUNNEL SOURCE("mysql") OPTIONS (
              username = "root",
              password = "",
              host = '172.18.6.181',
              port = 3306,
              databaseName = 'demos',
              tableName = 'account',
              columns = ["*"]
        ) 
        SINK("doris") OPTIONS (
            feEnpoints="172.18.5.44:18030",
            username = "root",
            password = "",
            databaseName = "testdb",
            tableName = "ods_account"
        )

DATATUNNEL SOURCE("kafka") OPTIONS (
            format="text",
            subscribe = "doris_demos",
            servers = "172.18.6.181:9092",
            'properties.kafka.security.protocol' = 'SASL_PLAINTEXT',
            'properties.kafka.sasl.mechanism' = 'PLAIN',
            'properties.kafka.sasl.jaas.config' = 'org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin2024";',
            includeHeaders = false,
            startingOffsets = 'earliest',
            checkpointLocation = "/Users/melin/stream_checkpoint/datatunnel/doris_demos",
            columns = ['*']
        ) 
        SINK("doris") OPTIONS (
            feEnpoints="172.18.5.44:18030",
            username = "root",
            password = "",
            databaseName = "testdb",
            tableName = "table_hash",
            passthrough = true,
            fileFormat = "json",
            columns = ["k1", "k2", "k3", "k4"]
        )

Solution

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct