You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+37-4
Original file line number
Diff line number
Diff line change
@@ -46,7 +46,40 @@ When a document is received and written by the connector, you'll see logging lik
46
46
47
47
## Configuring the connector
48
48
49
-
Connector-specific properties are defined in config/marklogic-sink.properties. Please see that file for a list
50
-
of all of the properties along with a description of each one.
51
-
52
-
[Descriptions of the available properties](Properties.html)
49
+
#### Connector-specific properties are defined in config/marklogic-connect-standalone.properties
50
+
| Property | Default Value | Description |
51
+
|:-------- |:--------------|:------------|
52
+
| bootstrap.servers | 9092 | This points to the Kafka server and port |
53
+
| key.converter | org.apache.kafka.connect.storage.StringConverter | This controls the format of the data that will be written to Kafka for source connectors or read from Kafka for sink connectors. |
54
+
| value.converter | org.apache.kafka.connect.storage.StringConverter | This controls the format of the data that will be written to Kafka for source connectors or read from Kafka for sink connectors. |
55
+
| key.converter.schemas.enable | false | Control the use of schemas for keys |
56
+
| value.converter.schemas.enable | false | Control the use of schemas for values |
57
+
| offset.storage.file.filename | /tmp/connect.offsets | The file to store connector offsets in. By storing offsets on disk, a standalone process can be stopped and started on a single node and resume where it previously left off. |
58
+
| offset.flush.interval.ms | 10000 | Interval at which to try committing offsets for tasks. |
59
+
60
+
#### MarkLogic-specific properties are defined in config/marklogic-sink.properties
61
+
| Property | Default Value | Description |
62
+
|:-------- |:--------------|:------------|
63
+
| name | marklogic-sink | The name of the connector |
64
+
| connector.class | com.marklogic.kafka.connect.sink.MarkLogicSinkConnector | The FQ name of the connector class |
65
+
| tasks.max | 1 | The maximum number of concurrent tasks |
66
+
| topics | marklogic | The name of the topic(s) to subscribe to |
67
+
| ml.connection.host | localhost | A MarkLogic host to connect to. The connector uses the Data Movement SDK, and thus it will connect to each of the hosts in a cluster. |
68
+
| ml.connection.port | 8000 | The port of a REST API server to connect to. |
69
+
| ml.connection.database | Documents | Optional - the name of a database to connect to. If your REST API server has a content database matching that of the one that you want to write documents to, you do not need to set this. |
70
+
| ml.connection.type | (empty) | Optional - set to "gateway" when using a load balancer, else leave blank. See https://docs.marklogic.com/guide/java/data-movement#id_26583 for more information. |
71
+
| ml.connection.securityContextType | DIGEST | Either DIGEST, BASIC, CERTIFICATE, KERBEROS, or NONE |
| ml.connection.certFile | (empty) | Certificate file for Certificate based authentication |
75
+
| ml.connection.certPassword | (empty) | Certificate password for Certificate based authentication |
76
+
| ml.connection.externalName | (empty) | The external name to use to connect to MarkLogic |
77
+
| ml.connection.simpleSsl | false | Set to "true" for a "simple" SSL strategy that uses the JVM's default SslContext and X509TrustManager and a "trust everything" HostnameVerifier. Further customization of an SSL connection via properties is not supported. If you need to do so, consider using the source code for this connector as a starting point. |
78
+
| ml.dmsdk.batchSize | 100 | Sets the number of documents to be written in a batch to MarkLogic. This may not have any impact depending on the connector receives data from Kafka, as the connector calls flushAsync on the DMSDK WriteBatcher after processing every collection of records. Thus, if the connector never receives at one time more than the value of this property, then the value of this property will have no impact. |
79
+
| ml.dmsdk.threadCount | 8 | Sets the number of threads used by the Data Movement SDK for parallelizing writes to MarkLogic. Similar to the batch size property above, this may never come into play depending on how many records the connector receives at once. |
80
+
| ml.document.collections | kafka-data | Optional - a comma-separated list of collections that each document should be written to |
81
+
| ml.document.format | JSON | Optional - specify the format of each document; either JSON, XML, BINARY, TEXT, or UNKNOWN |
82
+
| ml.document.mimeType | (empty) | Optional - specify a mime type for each document; typically the format property above will be used instead of this |
83
+
| ml.document.permissions | rest-reader,read,rest-writer,update | Optional - a comma-separated list of roles and capabilities that define the permissions for each document written to MarkLogic |
84
+
| ml.document.uriPrefix | /kafka-data/ | Optional - a prefix to prepend to each URI; the URI itself is a UUID |
85
+
| ml.document.uriSuffix | .json | Optional - a suffix to append to each URI |
0 commit comments