[FLINK-38839][runtime] Support semicolon delimiter for table-options

docs/content.zh/docs/core-concept/transform.md

@@ -31,16 +31,17 @@ under the License.
 # 参数
 为了定义一个 transform 规则，可以使用以下参数：
 
-| 参数                        | 含义                                                     | 是否必填     |
-|---------------------------|--------------------------------------------------------|----------|
-| source-table              | 源表 ID，支持正则表达式                                         | 必填       |
-| projection                | 投影规则，支持类似 SQL 中 SELECT 子句的语法                           | 可选       |
-| filter                    | 过滤规则，支持类似 SQL 中 WHERE 子句的语法                            | 可选       |
-| primary-keys              | 目标表主键，以逗号分隔                                            | 可选       |
-| partition-keys            | 目标表分区键，以逗号分隔                                           | 可选       |
-| table-options             | 用于配置自动建表时的建表语句                                         | 可选       |
-| converter-after-transform | 用于在 transform 处理后添加转换器来修改 DataChangeEvent                | 可选       |
-| description               | Transform 规则描述                                         | 可选       |
+| 参数                        | 含义                                        | 是否必填 |
+|---------------------------|-------------------------------------------|------|
+| source-table              | 源表 ID，支持正则表达式                             | 必填   |
+| projection                | 投影规则，支持类似 SQL 中 SELECT 子句的语法              | 可选   |
+| filter                    | 过滤规则，支持类似 SQL 中 WHERE 子句的语法               | 可选   |
+| primary-keys              | 目标表主键，以逗号分隔                               | 可选   |
+| partition-keys            | 目标表分区键，以逗号分隔                              | 可选   |
+| table-options             | 用于配置自动建表时的建表语句                            | 可选   |
+| table-options.delimiter   | 多个表属性的分隔符, 默认值为 `,`       | 可选   |
+| converter-after-transform | 用于在 transform 处理后添加转换器来修改 DataChangeEvent | 可选   |
+| description               | Transform 规则描述                            | 可选   |
 
 多个 transform 规则可以声明在一个 pipeline YAML 文件中。
 
@@ -335,7 +336,13 @@ transform:
     table-options: comment=web order
     description: auto creating table options example
 ```
-小技巧：table-options 的格式是 `key1=value1,key2=value2`。
+小技巧：table-options 的格式是 `key1=value1,key2=value2`；如果 value 中包含逗号或其他特殊字符，可以使用 `table-options.delimiter` 指定自定义分隔符（如 `;`、`|`、`$` 等）：
+```yaml
+transform:
+  - source-table: mydb.web_order
+    table-options: sequence.field=gxsj,jjsj;file-index.bloom-filter.columns=jjdbh
+    table-options.delimiter: ";"
+```
 
 ## 分类映射
 在一张表同时被多个转换规则命中时，
@@ -467,8 +474,8 @@ transform:
 
 ## Embedding AI 模型
 
-Embedding AI 模型可以在 transform 规则中使用。
-为了使用 Embedding AI 模型，你需要下载内置模型的 jar，然后在 `flink-cdc.sh` 命令中添加 `--jar {$BUILT_IN_MODEL_PATH}`。
+内置 AI 模型可以在 transform 规则中使用。
+为了使用内置 AI 模型，你需要下载内置模型的 jar ，然后在 `flink-cdc.sh` 命令中添加 `--jar {$BUILT_IN_MODEL_PATH}`。
 
 如何定义一个 Embedding AI 模型：
 

docs/content/docs/core-concept/transform.md

@@ -39,6 +39,7 @@ To describe a transform rule, the following parameters can be used:
 | primary-keys              | Sink table primary keys, separated by commas                                      | optional          |
 | partition-keys            | Sink table partition keys, separated by commas                                    | optional          |
 | table-options             | used to the configure table creation statement when automatically creating tables | optional          |
+| table-options.delimiter   | delimiter for table-options key-value pairs, default is `,`                       | optional          |
 | converter-after-transform | used to add a converter to change DataChangeEvent after transform                 | optional          |
 | description               | Transform rule description                                                        | optional          |
 
@@ -339,6 +340,13 @@ transform:
     description: auto creating table options example
 ```
 Tips: The format of table-options is `key1=value1,key2=value2`.
+If option values contain commas or other special characters, you can specify a custom delimiter using `table-options.delimiter` (such as `;`, `|`, `$`, etc.):
+```yaml
+transform:
+  - source-table: mydb.web_order
+    table-options: sequence.field=gxsj,jjsj;file-index.bloom-filter.columns=jjdbh
+    table-options.delimiter: ";"
+```
 
 ## Classification mapping
 If a table hits ultiple transform rules, only the first matched transform rule will apply.
@@ -538,4 +546,4 @@ The following built-in models are provided:
 |---------------|--------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | openai.model  | STRING | required          | Name of model to be called, for example: "text-embedding-3-small", Available options are "text-embedding-3-small", "text-embedding-3-large", "text-embedding-ada-002". |
 | openai.host   | STRING | required          | Host of the Model server to be connected, for example: `http://langchain4j.dev/demo/openai/v1`.                                                                        |
-| openai.apikey | STRING | required          | Api Key for verification of the Model server, for example, "demo".                                                                                                     |
+| openai.apikey | STRING | required          | Api Key for verification of the Model server, for example, "demo".                                                                                                     |

flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParser.java

@@ -105,6 +105,8 @@ public class YamlPipelineDefinitionParser implements PipelineDefinitionParser {
 
     public static final String TRANSFORM_TABLE_OPTION_KEY = "table-options";
 
+    public static final String TRANSFORM_TABLE_OPTION_DELIMITER_KEY = "table-options.delimiter";
+
     private final ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
 
     /** Parse the specified pipeline definition file. */
@@ -333,6 +335,7 @@ private TransformDef toTransformDef(JsonNode transformNode) {
                         TRANSFORM_PRIMARY_KEY_KEY,
                         TRANSFORM_PARTITION_KEY_KEY,
                         TRANSFORM_TABLE_OPTION_KEY,
+                        TRANSFORM_TABLE_OPTION_DELIMITER_KEY,
                         TRANSFORM_DESCRIPTION_KEY,
                         TRANSFORM_CONVERTER_AFTER_TRANSFORM_KEY));
 
@@ -366,6 +369,10 @@ private TransformDef toTransformDef(JsonNode transformNode) {
                 Optional.ofNullable(transformNode.get(TRANSFORM_TABLE_OPTION_KEY))
                         .map(JsonNode::asText)
                         .orElse(null);
+        String tableOptionsDelimiter =
+                Optional.ofNullable(transformNode.get(TRANSFORM_TABLE_OPTION_DELIMITER_KEY))
+                        .map(JsonNode::asText)
+                        .orElse(",");
         String description =
                 Optional.ofNullable(transformNode.get(TRANSFORM_DESCRIPTION_KEY))
                         .map(JsonNode::asText)
@@ -382,6 +389,7 @@ private TransformDef toTransformDef(JsonNode transformNode) {
                 primaryKeys,
                 partitionKeys,
                 tableOptions,
+                tableOptionsDelimiter,
                 description,
                 postTransformConverter);
     }

flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/definition/TransformDef.java

@@ -30,12 +30,14 @@
  *       definition.
  *   <li>filter: a string for filtering the row of matched table as output. Optional for the
  *       definition.
- *   <li>primaryKeys: a string for primary key columns for matching input table IDs, seperated by
+ *   <li>primaryKeys: a string for primary key columns for matching input table IDs, separated by
  *       `,`. Optional for the definition.
- *   <li>partitionKeys: a string for partition key columns for matching input table IDs, seperated
+ *   <li>partitionKeys: a string for partition key columns for matching input table IDs, separated
  *       by `,`. Optional for the definition.
  *   <li>tableOptions: a string for table options for matching input table IDs, options are
- *       seperated by `,`, key and value are seperated by `=`. Optional for the definition.
+ *       separated by `,`, key and value are separated by `=`. Optional for the definition.
+ *   <li>tableOptionsDelimiter: a string for delimiter of table options, default is `,`. Optional
+ *       for the definition.
  *   <li>description: description for the transformation. Optional for the definition.
  * </ul>
  */
@@ -47,6 +49,7 @@ public class TransformDef {
     private final String primaryKeys;
     private final String partitionKeys;
     private final String tableOptions;
+    private final String tableOptionsDelimiter;
     private final String postTransformConverter;
 
     public TransformDef(
@@ -56,6 +59,7 @@ public TransformDef(
             String primaryKeys,
             String partitionKeys,
             String tableOptions,
+            String tableOptionsDelimiter,
             String description,
             String postTransformConverter) {
         this.sourceTable = sourceTable;
@@ -64,10 +68,32 @@ public TransformDef(
         this.primaryKeys = primaryKeys;
         this.partitionKeys = partitionKeys;
         this.tableOptions = tableOptions;
+        this.tableOptionsDelimiter = tableOptionsDelimiter;
         this.description = description;
         this.postTransformConverter = postTransformConverter;
     }
 
+    public TransformDef(
+            String sourceTable,
+            String projection,
+            String filter,
+            String primaryKeys,
+            String partitionKeys,
+            String tableOptions,
+            String description,
+            String postTransformConverter) {
+        this(
+                sourceTable,
+                projection,
+                filter,
+                primaryKeys,
+                partitionKeys,
+                tableOptions,
+                ",",
+                description,
+                postTransformConverter);
+    }
+
     public String getSourceTable() {
         return sourceTable;
     }
@@ -96,6 +122,10 @@ public String getTableOptions() {
         return tableOptions;
     }
 
+    public String getTableOptionsDelimiter() {
+        return tableOptionsDelimiter;
+    }
+
     public String getPostTransformConverter() {
         return postTransformConverter;
     }
@@ -137,6 +167,7 @@ public boolean equals(Object o) {
                 && Objects.equals(primaryKeys, that.primaryKeys)
                 && Objects.equals(partitionKeys, that.partitionKeys)
                 && Objects.equals(tableOptions, that.tableOptions)
+                && Objects.equals(tableOptionsDelimiter, that.tableOptionsDelimiter)
                 && Objects.equals(postTransformConverter, that.postTransformConverter);
     }
 
@@ -150,6 +181,7 @@ public int hashCode() {
                 primaryKeys,
                 partitionKeys,
                 tableOptions,
+                tableOptionsDelimiter,
                 postTransformConverter);
     }
 }

flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/TransformTranslator.java

@@ -74,6 +74,7 @@ private PreTransformOperator generatePreTransform(
                     transform.getPrimaryKeys(),
                     transform.getPartitionKeys(),
                     transform.getTableOptions(),
+                    transform.getTableOptionsDelimiter(),
                     transform.getPostTransformConverter(),
                     supportedMetadataColumns);
         }
@@ -111,6 +112,7 @@ public DataStream<Event> translatePostTransform(
                     transform.getPrimaryKeys(),
                     transform.getPartitionKeys(),
                     transform.getTableOptions(),
+                    transform.getTableOptionsDelimiter(),
                     transform.getPostTransformConverter(),
                     supportedMetadataColumns);
         }

flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/unexpected.yaml

@@ -48,7 +48,7 @@ steps:
               sink: uuid
     error: |
       Unexpected key `mapping` in YAML transform block.
-      Allowed keys in this context are: [source-table, projection, filter, primary-keys, partition-keys, table-options, description, converter-after-transform]
+      Allowed keys in this context are: [source-table, projection, filter, primary-keys, partition-keys, table-options, table-options.delimiter, description, converter-after-transform]
       Note: option mapping: [{"source":"userUuid","sink":"uuid"}] is unexpected. It was silently ignored in previous versions, and probably should be removed.
   # Unexpected route block keys
   - type: submit

flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperatorBuilder.java

@@ -44,6 +44,28 @@ public PostTransformOperatorBuilder addTransform(
             String tableOptions,
             String postTransformConverter,
             SupportedMetadataColumn[] supportedMetadataColumns) {
+        return addTransform(
+                tableInclusions,
+                projection,
+                filter,
+                primaryKey,
+                partitionKey,
+                tableOptions,
+                ",",
+                postTransformConverter,
+                supportedMetadataColumns);
+    }
+
+    public PostTransformOperatorBuilder addTransform(
+            String tableInclusions,
+            @Nullable String projection,
+            @Nullable String filter,
+            String primaryKey,
+            String partitionKey,
+            String tableOptions,
+            String tableOptionsDelimiter,
+            String postTransformConverter,
+            SupportedMetadataColumn[] supportedMetadataColumns) {
         transformRules.add(
                 new TransformRule(
                         tableInclusions,
@@ -52,6 +74,7 @@ public PostTransformOperatorBuilder addTransform(
                         primaryKey,
                         partitionKey,
                         tableOptions,
+                        tableOptionsDelimiter,
                         postTransformConverter,
                         supportedMetadataColumns));
         return this;
@@ -67,6 +90,7 @@ public PostTransformOperatorBuilder addTransform(
                         "",
                         "",
                         "",
+                        ",",
                         null,
                         new SupportedMetadataColumn[0]));
         return this;

flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java

@@ -110,6 +110,7 @@ public void setup(
             String primaryKeys = transformRule.getPrimaryKey();
             String partitionKeys = transformRule.getPartitionKey();
             String tableOptions = transformRule.getTableOption();
+            String tableOptionsDelimiter = transformRule.getTableOptionsDelimiter();
             Selectors selectors =
                     new Selectors.SelectorsBuilder().includeTables(tableInclusions).build();
             transforms.add(
@@ -120,7 +121,11 @@ public void setup(
             schemaMetadataTransformers.add(
                     new Tuple2<>(
                             selectors,
-                            new SchemaMetadataTransform(primaryKeys, partitionKeys, tableOptions)));
+                            new SchemaMetadataTransform(
+                                    primaryKeys,
+                                    partitionKeys,
+                                    tableOptions,
+                                    tableOptionsDelimiter)));
         }
         this.preTransformProcessorMap = new ConcurrentHashMap<>();
         this.hasAsteriskMap = new ConcurrentHashMap<>();

flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperatorBuilder.java

@@ -43,6 +43,7 @@ public PreTransformOperatorBuilder addTransform(
                         "",
                         "",
                         "",
+                        ",",
                         null,
                         new SupportedMetadataColumn[0]));
         return this;
@@ -57,6 +58,28 @@ public PreTransformOperatorBuilder addTransform(
             String tableOption,
             @Nullable String postTransformConverter,
             SupportedMetadataColumn[] supportedMetadataColumns) {
+        return addTransform(
+                tableInclusions,
+                projection,
+                filter,
+                primaryKey,
+                partitionKey,
+                tableOption,
+                ",",
+                postTransformConverter,
+                supportedMetadataColumns);
+    }
+
+    public PreTransformOperatorBuilder addTransform(
+            String tableInclusions,
+            @Nullable String projection,
+            @Nullable String filter,
+            String primaryKey,
+            String partitionKey,
+            String tableOption,
+            String tableOptionsDelimiter,
+            @Nullable String postTransformConverter,
+            SupportedMetadataColumn[] supportedMetadataColumns) {
         transformRules.add(
                 new TransformRule(
                         tableInclusions,
@@ -65,6 +88,7 @@ public PreTransformOperatorBuilder addTransform(
                         primaryKey,
                         partitionKey,
                         tableOption,
+                        tableOptionsDelimiter,
                         postTransformConverter,
                         supportedMetadataColumns));
         return this;