[Feature][Config] Support nested references to external files in configuration files #8984

thirsd · 2025-03-16T03:08:04Z

thanks to @hailin0 https://github.com/hailin0.

reference the design:
#8945 (comment)

Does this PR introduce any user-facing change?

yes
user can use ref config file to reuse config in job config file.

#8941

How was this patch tested?

i have tested unset ref file .
i have tested set ref file and use in mysql to console and sqlite to console.

Check list

[x ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[x ] If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
[ x] Update the release-note.

…set path of ref_config, then you can use __st_config_ref_key__ in source\transform\sink to merge ref config.

liunaijie · 2025-03-17T01:12:51Z

docs/zh/concept/config.md

+source {
+  Jdbc {
+    __st_config_ref_key__ = "mysql_prod"
+    query="select * from department"


From the code, looks if the mysql_prod ref config also contains query parameter, it will overwrite the existed query parameter.

pluginConfig.withoutPath("__st_config_ref_key__").withFallback(refConfig);

Please add the priority for replacements when there are duplicate parameters.

got it

We can just describe it in the document to let use know this thing. Replace behavior is good to me.

已经补充了配置优先级。

…nces and add descriptions of ref configuration and plugin configuration priorities.

liunaijie · 2025-03-20T00:59:49Z

Please add test case

… ref_config

wuchunfu · 2025-03-27T03:19:23Z

docs/en/concept/config.md

+```hocon
+# 定义 MySQL 连接配置
+mysql_prod {


@thirsd Please describe in English here, thanks.

wuchunfu · 2025-03-27T03:20:07Z

docs/en/concept/config.md

+
+# 定义 Kafka 连接配置
+kafka_test {


litiliu · 2025-03-27T08:48:56Z

...nel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java

+            Map<String, Map<String, Object>> refMap = new HashMap<>();
+            // get map element in ref
+            for (String key : configMap.keySet()) {
+                Object ref_config = configMap.get(key);


we should name the variables in the camelcase style.

...nel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java

litiliu · 2025-03-27T09:02:25Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/options/EnvCommonOptions.java

                    .withDescription("Define the worker where the job runs by tag");
+
+    public static Option<String> REF_PATH =
+            Options.key("__st_config_ref_path__")


__st_config_ref_path__ looks weird, how about naming it to config_ref_path?

Not surprising, he cannot conflict with normal attributes

Copilot

Pull Request Overview

This PR introduces configuration reference support, allowing users to reuse shared configuration files in their job configurations. Key changes include adding a new environment option for reference config paths, merging plugin and reference configurations in the job parser, and updating tests and documentation to verify and explain the new behavior.

Reviewed Changes

Copilot reviewed 8 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java	Introduces refMaps field and merging logic for reference configs
seatunnel-engine-client/src/test/java/org/apache/seatunnel/engine/client/MultipleTableJobConfigParserTest.java	Adds tests for parsing job configs using the ref config feature
seatunnel-core/seatunnel-core-starter/src/test/java/org/apache/seatunnel/core/starter/utils/ConfigShadeTest.java	Extends tests to cover parsing and merging of reference configs
seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java	Adds processing branch to handle configuration references
seatunnel-api/src/main/java/org/apache/seatunnel/api/options/EnvCommonOptions.java	Adds a new environment option for specifying the ref config file path
docs/*	Updates both Chinese and English documentation to explain configuration reference functionality

Files not reviewed (4)

seatunnel-core/seatunnel-core-starter/src/test/resources/mysql_to_console_by_ref.conf: Language not supported
seatunnel-core/seatunnel-core-starter/src/test/resources/ref.conf: Language not supported
seatunnel-engine/seatunnel-engine-client/src/test/resources/fake_to_console_by_ref.conf: Language not supported
seatunnel-engine/seatunnel-engine-client/src/test/resources/ref.conf: Language not supported

Comments suppressed due to low confidence (1)

seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java:164

[nitpick] The variable name 'RefPath' does not follow the Java camelCase naming convention. Consider renaming it to 'refPath'.

String RefPath = envOptions.getOptional(EnvCommonOptions.REF_PATH).orElse(null);

...-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java

…e/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java Co-authored-by: Copilot <[email protected]>

Copilot

Pull Request Overview

This PR adds support for nested external file references in configuration files so that common configurations can be reused across job config files. Key changes include:

Introducing a new configuration field (refMaps) and merging logic based on a dedicated reference key.
Adding tests to verify the new reference-merging behavior.
Updating documentation (both English and Chinese) and the API to include the new ref config options.

Reviewed Changes

Copilot reviewed 8 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java	Added refMaps field, updated constructors to load external config files, and implemented merge logic.
seatunnel-engine-client/src/test/java/org/apache/seatunnel/engine/client/MultipleTableJobConfigParserTest.java	Introduced new test case for reference-based job configuration parsing.
seatunnel-core-starter/src/test/java/org/apache/seatunnel/core/starter/utils/ConfigShadeTest.java	Added tests to validate reference configuration parsing.
seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java	Updated config processing logic to handle ref configurations.
seatunnel-api/src/main/java/org/apache/seatunnel/api/options/EnvCommonOptions.java	Added a new option for the external reference config file path.
docs/zh/concept/config.md & docs/en/concept/config.md	Updated documents to explain and illustrate configuration reference usage.

Files not reviewed (4)

seatunnel-core/seatunnel-core-starter/src/test/resources/mysql_to_console_by_ref.conf: Language not supported
seatunnel-core/seatunnel-core-starter/src/test/resources/ref.conf: Language not supported
seatunnel-engine/seatunnel-engine-client/src/test/resources/fake_to_console_by_ref.conf: Language not supported
seatunnel-engine/seatunnel-engine-client/src/test/resources/ref.conf: Language not supported

Comments suppressed due to low confidence (4)

seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java:164

[nitpick] The variable name 'RefPath' starts with an uppercase letter, which is inconsistent with Java naming conventions. Consider renaming it to 'refPath'.

String RefPath = envOptions.getOptional(EnvCommonOptions.REF_PATH).orElse(null);

seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java:198

[nitpick] Again, the variable 'RefPath' should be renamed to 'refPath' for consistency with standard Java naming conventions.

String RefPath = envOptions.getOptional(EnvCommonOptions.REF_PATH).orElse(null);

seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java:213

[nitpick] The variable 'ref_dit' appears to be a typo; consider renaming it to 'refDict' for clarity and consistency.

Map<String, Object> ref_dit = new HashMap<>(RefConfig.size());

seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java:200

In the second constructor, the variables parameter is passed as null to ConfigBuilder.of; verify if this is intended or if the required variables should be provided.

this.refMaps = ConfigBuilder.of(Paths.get(RefPath), null);

Copilot · 2025-04-18T02:04:25Z

...nnel-core-starter/src/test/java/org/apache/seatunnel/core/starter/utils/ConfigShadeTest.java

+        Config jobConfig = ConfigBuilder.of(Paths.get(jobConfigUrl.toURI()));
+
+        Config mysqlPluginConfig = jobConfig.getConfigList("source").get(0);
+        String refId = mysqlPluginConfig.getString("__st_config_ref_key__ ");


There is an extra whitespace in the key string "st_config_ref_key ". This may lead to lookup failures; please remove the trailing space.

Suggested change

String refId = mysqlPluginConfig.getString("__st_config_ref_key__ ");

String refId = mysqlPluginConfig.getString("__st_config_ref_key__");

github-actions · 2025-08-17T00:39:31Z

This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.

github-actions · 2025-08-25T00:36:03Z

This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.

thirsd added 4 commits March 15, 2025 20:20

Merge remote-tracking branch 'upstream/dev' into dev

9d0a299

support ref_config. in env option, support __st_config_ref_path__ to …

8294c05

…set path of ref_config, then you can use __st_config_ref_key__ in source\transform\sink to merge ref config.

Code Style Format

ed29abf

add doc

8aefcf6

github-actions bot added document core SeaTunnel core module Zeta api labels Mar 16, 2025

thirsd mentioned this pull request Mar 16, 2025

[Feature] Support connection config, and use conn_id to replace conn info in job config #8945

Closed

4 tasks

liunaijie reviewed Mar 17, 2025

View reviewed changes

thirsd added 4 commits March 18, 2025 22:50

update config.md，增加ref配置和plugin配置优先级的描述。

9c0fd04

update docs/en/concept/config.md，Add examples of configuration refere…

73b4e0a

…nces and add descriptions of ref configuration and plugin configuration priorities.

Merge remote-tracking branch 'upstream/dev' into ref_config

5ebd068

Merge branch 'apache:dev' into ref_config

55972ae

thirsd added 5 commits March 23, 2025 00:47

add test examples

580ecd7

Merge remote-tracking branch 'upstream/dev' into ref_config

e4b41f7

Merge branch 'apache:dev' into ref_config

ccc1d22

Merge branch 'ref_config' of https://github.com/thirsd/seatunnel into…

6b1ad30

… ref_config

add license header

7ab6ec7

wuchunfu reviewed Mar 27, 2025

View reviewed changes

docs/en/concept/config.md

Comment on lines +357 to +359

# 定义 Kafka 连接配置

kafka_test {

Copy link

Member

wuchunfu Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

davidzollo added the First-time contributor First-time contributor label Mar 27, 2025

litiliu reviewed Mar 27, 2025

View reviewed changes

...nel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java Show resolved Hide resolved

litiliu reviewed Mar 27, 2025

View reviewed changes

hailin0 requested a review from Copilot March 31, 2025 05:17

Copilot AI reviewed Mar 31, 2025

View reviewed changes

...-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java Outdated Show resolved Hide resolved

Update seatunnel-engine/seatunnel-engine-core/src/main/java/org/apach…

2973521

…e/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java Co-authored-by: Copilot <[email protected]>

hailin0 changed the title ~~Ref config~~ [Feature][Config] Support nested references to external files in configuration files Apr 2, 2025

This was referenced Apr 2, 2025

SeaTunnel community meeting topic collect #9100

Closed

[Feature][seatunnel-config] connection configuration uniform and reus… #9116

Closed

nielifeng requested a review from Copilot April 18, 2025 02:03

Copilot AI reviewed Apr 18, 2025

View reviewed changes

github-actions bot added the stale label Aug 17, 2025

github-actions bot closed this Aug 25, 2025

	String refId = mysqlPluginConfig.getString("__st_config_ref_key__ ");
	String refId = mysqlPluginConfig.getString("__st_config_ref_key__");

[Feature][Config] Support nested references to external files in configuration files #8984

[Feature][Config] Support nested references to external files in configuration files #8984

Uh oh!

Conversation

thirsd commented Mar 16, 2025

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liunaijie commented Mar 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 17, 2025

Uh oh!

github-actions bot commented Aug 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants