Skip to content

Commit 78b23c0

Browse files
Hisoka-XHisoka-X
authored
[Feature][Connector-V2] Add filename_extension parameter for read/write file (#8769)
1 parent 2dc8296 commit 78b23c0

File tree

46 files changed

+432
-71
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+432
-71
lines changed

Diff for: ‎docs/en/connector-v2/sink/CosFile.md

+1
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ By default, we use 2PC commit to ensure `exactly-once`
4646
| file_name_expression | string | no | "${transactionId}" | Only used when custom_filename is true |
4747
| filename_time_format | string | no | "yyyy.MM.dd" | Only used when custom_filename is true |
4848
| file_format_type | string | no | "csv" | |
49+
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
4950
| field_delimiter | string | no | '\001' | Only used when file_format is text |
5051
| row_delimiter | string | no | "\n" | Only used when file_format is text |
5152
| have_partition | boolean | no | false | Whether you need processing partitions. |

Diff for: ‎docs/en/connector-v2/sink/FtpFile.md

+1
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ By default, we use 2PC commit to ensure `exactly-once`
4545
| file_name_expression | string | no | "${transactionId}" | Only used when custom_filename is true |
4646
| filename_time_format | string | no | "yyyy.MM.dd" | Only used when custom_filename is true |
4747
| file_format_type | string | no | "csv" | |
48+
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
4849
| field_delimiter | string | no | '\001' | Only used when file_format_type is text |
4950
| row_delimiter | string | no | "\n" | Only used when file_format_type is text |
5051
| have_partition | boolean | no | false | Whether you need processing partitions. |

Diff for: ‎docs/en/connector-v2/sink/HdfsFile.md

+1
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ Output data to hdfs file
4848
| file_name_expression | string | no | "${transactionId}" | Only used when `custom_filename` is `true`.`file_name_expression` describes the file expression which will be created into the `path`. We can add the variable `${now}` or `${uuid}` in the `file_name_expression`, like `test_${uuid}_${now}`,`${now}` represents the current time, and its format can be defined by specifying the option `filename_time_format`.Please note that, If `is_enable_transaction` is `true`, we will auto add `${transactionId}_` in the head of the file. |
4949
| filename_time_format | string | no | "yyyy.MM.dd" | Only used when `custom_filename` is `true`.When the format in the `file_name_expression` parameter is `xxxx-${now}` , `filename_time_format` can specify the time format of the path, and the default value is `yyyy.MM.dd` . The commonly used time formats are listed as follows:[y:Year,M:Month,d:Day of month,H:Hour in day (0-23),m:Minute in hour,s:Second in minute] |
5050
| file_format_type | string | no | "csv" | We supported as the following file types:`text` `csv` `parquet` `orc` `json` `excel` `xml` `binary`.Please note that, The final file name will end with the file_format's suffix, the suffix of the text file is `txt`. |
51+
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
5152
| field_delimiter | string | no | '\001' | Only used when file_format is text,The separator between columns in a row of data. Only needed by `text` file format. |
5253
| row_delimiter | string | no | "\n" | Only used when file_format is text,The separator between rows in a file. Only needed by `text` file format. |
5354
| have_partition | boolean | no | false | Whether you need processing partitions. |

Diff for: ‎docs/en/connector-v2/sink/LocalFile.md

+13-12
Original file line numberDiff line numberDiff line change
@@ -33,18 +33,19 @@ By default, we use 2PC commit to ensure `exactly-once`
3333

3434
## Options
3535

36-
| Name | Type | Required | Default | Description |
37-
|---------------------------------------|---------|----------|------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
38-
| path | string | yes | - | |
39-
| tmp_path | string | no | /tmp/seatunnel | The result file will write to a tmp path first and then use `mv` to submit tmp dir to target dir. |
40-
| custom_filename | boolean | no | false | Whether you need custom the filename |
41-
| file_name_expression | string | no | "${transactionId}" | Only used when custom_filename is true |
42-
| filename_time_format | string | no | "yyyy.MM.dd" | Only used when custom_filename is true |
43-
| file_format_type | string | no | "csv" | |
44-
| field_delimiter | string | no | '\001' | Only used when file_format_type is text |
45-
| row_delimiter | string | no | "\n" | Only used when file_format_type is text |
46-
| have_partition | boolean | no | false | Whether you need processing partitions. |
47-
| partition_by | array | no | - | Only used then have_partition is true |
36+
| Name | Type | Required | Default | Description |
37+
|---------------------------------------|---------|----------|--------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
38+
| path | string | yes | - | |
39+
| tmp_path | string | no | /tmp/seatunnel | The result file will write to a tmp path first and then use `mv` to submit tmp dir to target dir. |
40+
| custom_filename | boolean | no | false | Whether you need custom the filename |
41+
| file_name_expression | string | no | "${transactionId}" | Only used when custom_filename is true |
42+
| filename_time_format | string | no | "yyyy.MM.dd" | Only used when custom_filename is true |
43+
| file_format_type | string | no | "csv" | |
44+
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
45+
| field_delimiter | string | no | '\001' | Only used when file_format_type is text |
46+
| row_delimiter | string | no | "\n" | Only used when file_format_type is text |
47+
| have_partition | boolean | no | false | Whether you need processing partitions. |
48+
| partition_by | array | no | - | Only used then have_partition is true |
4849
| partition_dir_expression | string | no | "${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/" | Only used then have_partition is true |
4950
| is_partition_field_write_in_file | boolean | no | false | Only used then have_partition is true |
5051
| sink_columns | array | no | | When this parameter is empty, all fields are sink columns |

Diff for: ‎docs/en/connector-v2/sink/OssFile.md

+1
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ If write to `csv`, `text` file type, All column will be string.
9898
| file_name_expression | string | no | "${transactionId}" | Only used when custom_filename is true |
9999
| filename_time_format | string | no | "yyyy.MM.dd" | Only used when custom_filename is true |
100100
| file_format_type | string | no | "csv" | |
101+
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
101102
| field_delimiter | string | no | '\001' | Only used when file_format_type is text |
102103
| row_delimiter | string | no | "\n" | Only used when file_format_type is text |
103104
| have_partition | boolean | no | false | Whether you need processing partitions. |

Diff for: ‎docs/en/connector-v2/sink/OssJindoFile.md

+1
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ By default, we use 2PC commit to ensure `exactly-once`
5050
| file_name_expression | string | no | "${transactionId}" | Only used when custom_filename is true |
5151
| filename_time_format | string | no | "yyyy.MM.dd" | Only used when custom_filename is true |
5252
| file_format_type | string | no | "csv" | |
53+
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
5354
| field_delimiter | string | no | '\001' | Only used when file_format_type is text |
5455
| row_delimiter | string | no | "\n" | Only used when file_format_type is text |
5556
| have_partition | boolean | no | false | Whether you need processing partitions. |

0 commit comments

Comments
 (0)