Skip to content

Commit 227a11f

Browse files
corgy-wcorgy-w
authored
[Fix][Connector-V2] User selects csv string pattern (#8572)
1 parent 0bf0693 commit 227a11f

File tree

16 files changed

+281
-78
lines changed

16 files changed

+281
-78
lines changed

Diff for: ‎docs/en/connector-v2/sink/CosFile.md

+10-1
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ By default, we use 2PC commit to ensure `exactly-once`
5959
| common-options | object | no | - | |
6060
| max_rows_in_memory | int | no | - | Only used when file_format is excel. |
6161
| sheet_name | string | no | Sheet${Random number} | Only used when file_format is excel. |
62+
| csv_string_quote_mode | enum | no | MINIMAL | Only used when file_format is csv. |
6263
| xml_root_tag | string | no | RECORDS | Only used when file_format is xml. |
6364
| xml_row_tag | string | no | RECORD | Only used when file_format is xml. |
6465
| xml_use_attr_format | boolean | no | - | Only used when file_format is xml. |
@@ -107,7 +108,7 @@ Only used when `custom_filename` is `true`
107108

108109
When the format in the `file_name_expression` parameter is `xxxx-${now}` , `filename_time_format` can specify the time format of the path, and the default value is `yyyy.MM.dd` . The commonly used time formats are listed as follows:
109110

110-
| Symbol | Description |
111+
| Symbol | Description |
111112
|--------|--------------------|
112113
| y | Year |
113114
| M | Month |
@@ -199,6 +200,14 @@ When File Format is Excel,The maximum number of data items that can be cached in
199200

200201
Writer the sheet of the workbook
201202

203+
### csv_string_quote_mode [string]
204+
205+
When File Format is CSV,The string quote mode of CSV.
206+
207+
- ALL: All String fields will be quoted.
208+
- MINIMAL: Quotes fields which contain special characters such as a the field delimiter, quote character or any of the characters in the line separator string.
209+
- NONE: Never quotes fields. When the delimiter occurs in data, the printer prefixes it with the escape character. If the escape character is not set, format validation throws an exception.
210+
202211
### xml_root_tag [string]
203212

204213
Specifies the tag name of the root element within the XML file.

Diff for: ‎docs/en/connector-v2/sink/FtpFile.md

+14
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ By default, we use 2PC commit to ensure `exactly-once`
5858
| common-options | object | no | - | |
5959
| max_rows_in_memory | int | no | - | Only used when file_format_type is excel. |
6060
| sheet_name | string | no | Sheet${Random number} | Only used when file_format_type is excel. |
61+
| csv_string_quote_mode | enum | no | MINIMAL | Only used when file_format is csv. |
6162
| xml_root_tag | string | no | RECORDS | Only used when file_format is xml. |
6263
| xml_row_tag | string | no | RECORD | Only used when file_format is xml. |
6364
| xml_use_attr_format | boolean | no | - | Only used when file_format is xml. |
@@ -207,6 +208,14 @@ When File Format is Excel,The maximum number of data items that can be cached in
207208

208209
Writer the sheet of the workbook
209210

211+
### csv_string_quote_mode [string]
212+
213+
When File Format is CSV,The string quote mode of CSV.
214+
215+
- ALL: All String fields will be quoted.
216+
- MINIMAL: Quotes fields which contain special characters such as a the field delimiter, quote character or any of the characters in the line separator string.
217+
- NONE: Never quotes fields. When the delimiter occurs in data, the printer prefixes it with the escape character. If the escape character is not set, format validation throws an exception.
218+
210219
### xml_root_tag [string]
211220

212221
Specifies the tag name of the root element within the XML file.
@@ -237,17 +246,22 @@ Only used when file_format_type is json,text,csv,xml.
237246
The encoding of the file to write. This param will be parsed by `Charset.forName(encoding)`.
238247

239248
### schema_save_mode [string]
249+
240250
Existing dir processing method.
251+
241252
- RECREATE_SCHEMA: will create when the dir does not exist, delete and recreate when the dir is exist
242253
- CREATE_SCHEMA_WHEN_NOT_EXIST: will create when the dir does not exist, skipped when the dir is exist
243254
- ERROR_WHEN_SCHEMA_NOT_EXIST: error will be reported when the dir does not exist
244255
- IGNORE ：Ignore the treatment of the table
245256

246257
### data_save_mode [string]
258+
247259
Existing data processing method.
260+
248261
- DROP_DATA: preserve dir and delete data files
249262
- APPEND_DATA: preserve dir, preserve data files
250263
- ERROR_WHEN_DATA_EXISTS: when there is data files, an error is reported
264+
251265
## Example
252266

253267
For text file format simple config

Diff for: ‎docs/en/connector-v2/sink/HdfsFile.md

+9
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@ Output data to hdfs file
6565
| common-options | object | no | - | Sink plugin common parameters, please refer to [Sink Common Options](../sink-common-options.md) for details |
6666
| max_rows_in_memory | int | no | - | Only used when file_format is excel.When File Format is Excel,The maximum number of data items that can be cached in the memory. |
6767
| sheet_name | string | no | Sheet${Random number} | Only used when file_format is excel.Writer the sheet of the workbook |
68+
| csv_string_quote_mode | enum | no | MINIMAL | Only used when file_format is csv. |
6869
| xml_root_tag | string | no | RECORDS | Only used when file_format is xml, specifies the tag name of the root element within the XML file. |
6970
| xml_row_tag | string | no | RECORD | Only used when file_format is xml, specifies the tag name of the data rows within the XML file |
7071
| xml_use_attr_format | boolean | no | - | Only used when file_format is xml, specifies Whether to process data using the tag attribute format. |
@@ -203,6 +204,14 @@ HdfsFile {
203204

204205
Only used when file_format_type is text,csv.false:don't write header,true:write header.
205206

207+
### csv_string_quote_mode [string]
208+
209+
When File Format is CSV,The string quote mode of CSV.
210+
211+
- ALL: All String fields will be quoted.
212+
- MINIMAL: Quotes fields which contain special characters such as a the field delimiter, quote character or any of the characters in the line separator string.
213+
- NONE: Never quotes fields. When the delimiter occurs in data, the printer prefixes it with the escape character. If the escape character is not set, format validation throws an exception.
214+
206215
### For compress simple config
207216

208217
```

0 commit comments

Comments
 (0)