Skip to content

Commit 92aa855

Browse files
hailin0hailin0
authored
[Improve][File] Add row_delimiter options into text file sink (#9017)
1 parent 25b5abb commit 92aa855

File tree

28 files changed

+217
-199
lines changed

28 files changed

+217
-199
lines changed

Diff for: ‎docs/en/connector-v2/sink/CosFile.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ By default, we use 2PC commit to ensure `exactly-once`
5050
| file_format_type | string | no | "csv" | |
5151
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
5252
| field_delimiter | string | no | '\001' | Only used when file_format is text |
53-
| row_delimiter | string | no | "\n" | Only used when file_format is text |
53+
| row_delimiter | string | no | "\n" | Only used when file_format is `text`, `csv` and `json` |
5454
| have_partition | boolean | no | false | Whether you need processing partitions. |
5555
| partition_by | array | no | - | Only used then have_partition is true |
5656
| partition_dir_expression | string | no | "${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/" | Only used then have_partition is true |
@@ -134,7 +134,7 @@ The separator between columns in a row of data. Only needed by `text` file forma
134134

135135
### row_delimiter [string]
136136

137-
The separator between rows in a file. Only needed by `text` file format.
137+
The separator between rows in a file. Only needed by `text`, `csv` and `json` file format.
138138

139139
### have_partition [boolean]
140140

Diff for: ‎docs/en/connector-v2/sink/FtpFile.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ By default, we use 2PC commit to ensure `exactly-once`
4949
| file_format_type | string | no | "csv" | |
5050
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
5151
| field_delimiter | string | no | '\001' | Only used when file_format_type is text |
52-
| row_delimiter | string | no | "\n" | Only used when file_format_type is text |
52+
| row_delimiter | string | no | "\n" | Only used when file_format_type is `text`, `csv` and `json` |
5353
| have_partition | boolean | no | false | Whether you need processing partitions. |
5454
| partition_by | array | no | - | Only used then have_partition is true |
5555
| partition_dir_expression | string | no | "${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/" | Only used then have_partition is true |
@@ -142,7 +142,7 @@ The separator between columns in a row of data. Only needed by `text` file forma
142142

143143
### row_delimiter [string]
144144

145-
The separator between rows in a file. Only needed by `text` file format.
145+
The separator between rows in a file. Only needed by `text`, `csv` and `json` file format.
146146

147147
### have_partition [boolean]
148148

Diff for: ‎docs/en/connector-v2/sink/HdfsFile.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ Output data to hdfs file
5252
| file_format_type | string | no | "csv" | We supported as the following file types:`text` `csv` `parquet` `orc` `json` `excel` `xml` `binary`.Please note that, The final file name will end with the file_format's suffix, the suffix of the text file is `txt`. |
5353
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
5454
| field_delimiter | string | no | '\001' | Only used when file_format is text,The separator between columns in a row of data. Only needed by `text` file format. |
55-
| row_delimiter | string | no | "\n" | Only used when file_format is text,The separator between rows in a file. Only needed by `text` file format. |
55+
| row_delimiter | string | no | "\n" | Only used when file_format is text,The separator between rows in a file. Only needed by `text`, `csv` and `json` file format. |
5656
| have_partition | boolean | no | false | Whether you need processing partitions. |
5757
| partition_by | array | no | - | Only used then have_partition is true,Partition data based on selected fields. |
5858
| partition_dir_expression | string | no | "${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/" | Only used then have_partition is true,If the `partition_by` is specified, we will generate the corresponding partition directory based on the partition information, and the final file will be placed in the partition directory. Default `partition_dir_expression` is `${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/`. `k0` is the first partition field and `v0` is the value of the first partition field. |

Diff for: ‎docs/en/connector-v2/sink/LocalFile.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ By default, we use 2PC commit to ensure `exactly-once`
4545
| file_format_type | string | no | "csv" | |
4646
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
4747
| field_delimiter | string | no | '\001' | Only used when file_format_type is text |
48-
| row_delimiter | string | no | "\n" | Only used when file_format_type is text |
48+
| row_delimiter | string | no | "\n" | Only used when file_format_type is `text`, `csv` and `json` |
4949
| have_partition | boolean | no | false | Whether you need processing partitions. |
5050
| partition_by | array | no | - | Only used then have_partition is true |
5151
| partition_dir_expression | string | no | "${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/" | Only used then have_partition is true |
@@ -116,7 +116,7 @@ The separator between columns in a row of data. Only needed by `text` file forma
116116

117117
### row_delimiter [string]
118118

119-
The separator between rows in a file. Only needed by `text` file format.
119+
The separator between rows in a file. Only needed by `text`, `json` and `json` file format.
120120

121121
### have_partition [boolean]
122122

Diff for: ‎docs/en/connector-v2/sink/ObsFile.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ It only supports hadoop version **2.9.X+**.
6464
| filename_time_format | string | no | "yyyy.MM.dd" | Specify the time format of the `path`. Only used when custom_filename is true. [Tips](#filename_time_format) |
6565
| file_format_type | string | no | "csv" | Supported file types. [Tips](#file_format_type) |
6666
| field_delimiter | string | no | '\001' | The separator between columns in a row of data.Only used when file_format is text. |
67-
| row_delimiter | string | no | "\n" | The separator between rows in a file. Only needed by `text` file format. |
67+
| row_delimiter | string | no | "\n" | The separator between rows in a file. Only needed by `text`, `csv` and `json` file format. |
6868
| have_partition | boolean | no | false | Whether you need processing partitions. |
6969
| partition_by | array | no | - | Partition data based on selected fields. Only used then have_partition is true. |
7070
| partition_dir_expression | string | no | "${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/" | Only used then have_partition is true.[Tips](#partition_dir_expression) |

Diff for: ‎docs/en/connector-v2/sink/OssFile.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ By default, we use 2PC commit to ensure `exactly-once`
4040

4141
## Data Type Mapping
4242

43-
If write to `csv`, `text` file type, All column will be string.
43+
If write to `csv`, `text`, `json` file type, All column will be string.
4444

4545
### Orc File Type
4646

@@ -102,7 +102,7 @@ If write to `csv`, `text` file type, All column will be string.
102102
| file_format_type | string | no | "csv" | |
103103
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
104104
| field_delimiter | string | no | '\001' | Only used when file_format_type is text |
105-
| row_delimiter | string | no | "\n" | Only used when file_format_type is text |
105+
| row_delimiter | string | no | "\n" | Only used when file_format_type is `text`, `csv` and `json` |
106106
| have_partition | boolean | no | false | Whether you need processing partitions. |
107107
| partition_by | array | no | - | Only used then have_partition is true |
108108
| partition_dir_expression | string | no | "${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/" | Only used then have_partition is true |
@@ -187,7 +187,7 @@ The separator between columns in a row of data. Only needed by `text` file forma
187187

188188
### row_delimiter [string]
189189

190-
The separator between rows in a file. Only needed by `text` file format.
190+
The separator between rows in a file. Only needed by `text`, `csv` and `json` file format.
191191

192192
### have_partition [boolean]
193193

Diff for: ‎docs/en/connector-v2/sink/OssJindoFile.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ By default, we use 2PC commit to ensure `exactly-once`
5454
| file_format_type | string | no | "csv" | |
5555
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
5656
| field_delimiter | string | no | '\001' | Only used when file_format_type is text |
57-
| row_delimiter | string | no | "\n" | Only used when file_format_type is text |
57+
| row_delimiter | string | no | "\n" | Only used when file_format_type is `text`, `csv` and `json` |
5858
| have_partition | boolean | no | false | Whether you need processing partitions. |
5959
| partition_by | array | no | - | Only used then have_partition is true |
6060
| partition_dir_expression | string | no | "${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/" | Only used then have_partition is true |
@@ -138,7 +138,7 @@ The separator between columns in a row of data. Only needed by `text` file forma
138138

139139
### row_delimiter [string]
140140

141-
The separator between rows in a file. Only needed by `text` file format.
141+
The separator between rows in a file. Only needed by `text`, `csv` and `json` file format.
142142

143143
### have_partition [boolean]
144144

0 commit comments

Comments
 (0)