[Umbrella] unified parameter class and refactor connector creation logic

[liunaijie]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I had searched in the issues and found no similar issues.

Describe the proposal

This issue has two proposal:

1. move all parameter that user can adjust to a single class, connector use XXXSourceConfig/ XXXSourceOptions.
   Update EnvCommandOptions to describe the parameters we can adjust in config env part.
   Update ServerConfigOptions to describe the parameters we can adjust about Zeta Engine config (config in SeaTunnel.yaml).

The benefits:

• We can easily know what parameter can be adjusted, that can help us to tune the performance.
• When version release, we can easily know which parameter has been changed, which can break the user version upgrade, which is added in new version, So we can record this into version upgrade document (although we don't have it yet).

2. refactor connector, implement TableSourceFactory, TableSinkFactory interface and implement createSource, createSink, optionRule method.

In optionRule method, describe which parameter is necessary, which is optional, which are the conflicts with other parameter. (need action1 done)

Move the connector creation logic from Connector to ConnectorFactory.

The benefits:

• Know the relationship between parameters. Use the optionRule check the config before create connector instance.
• This can be used in SeaTunnelWeb.

Welcome to do this with me

Task list

type		assign to	status
ZetaConfig	ServerConfigOptions	@liunaijie	Done
EnvConfig	EnvCommonOptions	@liunaijie	Done
Connector	connector-activemq	@liunaijie	Done
Connector	connector-amazondynamodb	@liunaijie	Done
Connector	connector-amazonsqs	@liunaijie	Done
Connector	connector-assert	@liunaijie	Done
Connector	connector-cassandra	@liunaijie	Done
Connector	connector-cdc-mongodb	@liunaijie	WIP
Connector	connector-cdc-mysql	@liunaijie	WIP
Connector	connector-cdc-opengauss	@liunaijie	WIP
Connector	connector-cdc-oracle	@liunaijie	WIP
Connector	connector-cdc-postgres	@liunaijie	WIP
Connector	connector-cdc-sqlserver	@liunaijie	WIP
Connector	connector-cdc-tidb	@liunaijie	WIP
Connector	connector-clickhouse	@lyne7-sc	Done
Connector	connector-console	@liunaijie	Done
Connector	connector-datahub	@liunaijie	Done
Connector	connector-dingtalk	@liunaijie	Done
Connector	connector-doris	@liunaijie	Done
Connector	connector-druid	@akulabs8	Done
Connector	connector-easysearch	@liunaijie	Done
Connector	connector-elasticsearch	@fcb-xiaobo	Done
Connector	connector-email	@xdu-chenrj	Done
Connector	connector-fake	@liunaijie	Done
Connector	connector-file-cos	@misi1987107	WIP
Connector	connector-file-ftp	@misi1987107	WIP
Connector	connector-file-hadoop	@misi1987107	WIP
Connector	connector-file-jindo-oss	@misi1987107	WIP
Connector	connector-file-local	@misi1987107	WIP
Connector	connector-file-obs	@misi1987107	WIP
Connector	connector-file-oss	@misi1987107	WIP
Connector	connector-file-s3	@litiliu	Done
Connector	connector-file-sftp	@misi1987107	WIP
Connector	connector-google-firestore	@liunaijie	Done
Connector	connector-google-sheets	@liunaijie	Done
Connector	connector-hbase	@liunaijie	Done
Connector	connector-hive	need wait File Options pr merge
Connector	connector-http-feishu	@liunaijie	Done
Connector	connector-http-github	@liunaijie	Done
Connector	connector-http-gitlab	@liunaijie	Done
Connector	connector-http-jira	@liunaijie	Done
Connector	connector-http-klaviyo	@liunaijie	Done
Connector	connector-http-lemlist	@liunaijie	Done
Connector	connector-http-myhours	@liunaijie	Done
Connector	connector-http-notion	@liunaijie	Done
Connector	connector-http-onesignal	@liunaijie	Done
Connector	connector-http-persistiq	@liunaijie	Done
Connector	connector-http-wechat	@liunaijie	Done
Connector	connector-hudi	@liunaijie	Done
Connector	connector-iceberg	@liunaijie	Done
Connector	connector-influxdb	@liunaijie	Done
Connector	connector-iotdb	@liunaijie	Done
Connector	connector-jdbc	@liunaijie	WIP
Connector	connector-kafka	@wattt3	Done
Connector	connector-kudu
Connector	connector-maxcompute
Connector	connector-milvus
Connector	connector-mongodb	@stephenZkang	WIP
Connector	connector-neo4j
Connector	connector-openmldb
Connector	connector-paimon
Connector	connector-prometheus	@liunaijie	Done
Connector	connector-pulsar
Connector	connector-qdrant
Connector	connector-rabbitmq	@liunaijie	Done
Connector	connector-redis	@akulabs8	Done
Connector	connector-rocketmq
Connector	connector-s3-redshift
Connector	connector-selectdb-cloud
Connector	connector-sentry
Connector	connector-slack	@akulabs8	Done
Connector	connector-sls
Connector	connector-socket
Connector	connector-starrocks	@fcb-xiaobo	Done
Connector	connector-tablestore
Connector	connector-tdengine
Connector	connector-typesense
Connector	connector-web3j	@akulabs8	Done

What should a Pull Request accomplish?

• List and describe the connector parameters in the Factory.optionRule() method.
  While doing so, we can move all parameters into the XXXXOptions class to unify the code style and centralize the parameters into a single file.

• Move the connector creation logic into Factory.createSource/createSink.
  Implement the createSource/createSink methods in Factory and instantiate the corresponding connector.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[Hisoka-X]

+1 from proposal1.
About proposal2, could you share more details about Move the connector creation logic from Connector to ConnectorFactory.? What it would be look like?

[liunaijie]

+1 from proposal1. About proposal2, could you share more details about Move the connector creation logic from Connector to ConnectorFactory.? What it would be look like?

About proposal2
now some connector implement the TableSourceFactory, but they not implement createSource method. so we still has 2 way to create the Connector instance.

• one is use tableSourceFactory.createSource/Sink() it return the connector instance.
• one is use ServiceLoader.load to load all implement and use pluginIdentifier to find the connector instance.

So I suggest we unified the behavior, implement the createSource/Sink method. use Factory to create the connector instance.

    private static boolean isFallback(Optional<Factory> factory) {
        if (!factory.isPresent()) {
            return true;
        }
        try {
            ((TableSourceFactory) factory.get()).createSource(null);
        } catch (Exception e) {
            if (e instanceof UnsupportedOperationException
                    && "The Factory has not been implemented and the deprecated Plugin will be used."
                            .equals(e.getMessage())) {
                return true;
            }
        }
        return false;
    }

[Hisoka-X]

Oh I got it. That's right.

[liunaijie]

example pr : #8580

[akulabs8]

Hey , I would like to contribute

[liunaijie]

Hey , I would like to contribute

Hi, thank you want to do this with me. And Just comment which connector you want to update, I will assign it to you.

[akulabs8]

Hey @liunaijie ,
I would like to start with Redis one

P.S. I looked into kafka but I couldn't find the Client sub directory there, any reasons why so?

[akulabs8]

Hey @liunaijie , I have come with a draft PR: #8594

Please take a look and let me know

[liunaijie]

Hey @liunaijie , I would like to start with Redis one

P.S. I looked into kafka but I couldn't find the Client sub directory there, any reasons why so?

Hi, the kafka connector is here https://github.com/apache/seatunnel/tree/dev/seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka