Skip to content

Commit 3c688b1

Browse files
elyse-mdbcarriecwk
authored and
your proper name
committed
(DOCSP-43962) [Kafka] Kafka connector nested components (#167)
* (DOCSP-43962) Unnested 1 example in tip. * (DOCSP-43962) Removing the subjunctive should * (DOCSP-42962) Unnested 1 important admonition in table. * (DOCSP-42962) Typo fix. * (DOCSP-42962) Unnested 1 important admonition in table. * (DOCSP-42962) Unnested 1 tip in table. * (DOCSP-42962) Fixing indentation error and unnesting 1 important admonition in table. * (DOCSP-42962) Spacing. * (DOCSP-42962) Build errors. * (DOCSP-42962) Unnested 1 example and 1 note in table. * (DOCSP-42962) Unnested 2 examples and 1 note in include. * (DOCSP-42962) Replace quotation marks with monospace. * (DOCSP-42962) Unnested 2 notes in table. * (DOCSP-42962) consistency. * (DOCSP-42962) Edited for consistency and unnested 2 tips. * (DOCSP-42962) Removing extra line breaks. * (DOCSP-42962) Unnested 2 important admonitions in table. * (DOCSP-42962) Fix rendering error. * (DOCSP-42962) Fixing rendering issues. * (DOCSP-42962) Unnested 2 notes and 1 example in table. * (DOCSP-42962) Unnested 3 important admonitions * (DOCSP-42962) Spacing fix. * (DOCSP-42962) Unnested 1 example and 3 tips. * (DOCSP-42962) Unnested 1 note, 2 tips, 1 important admonition. * (DOCSP-42962) Spacing. * (DOCSP-42962) Spacing. * (DOCSP-42962) Normalizing spacing and adding reference to the Data Formats page. * (DOCSP-42962) Unnested 3 tips, 1 example and 1 important admonition in table. * (DOCSP-42962) Regularizing usage of pipeline operator line breaks. * (DOCSP-42962) Spacing. * (DOCSP-42962) Spacing. * (DOCSP-42962) Normalizing spacing across pages. * Update source/source-connector/configuration-properties/kafka-topic.txt Co-authored-by: carriecwk <[email protected]> * Update source/source-connector/configuration-properties/kafka-topic.txt Co-authored-by: carriecwk <[email protected]> --------- Co-authored-by: Elyse Foreman <[email protected]> Co-authored-by: carriecwk <[email protected]>
1 parent 007cc76 commit 3c688b1

20 files changed

+258
-276
lines changed

‎source/includes/copy-existing-admonition.rst

+5-7
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
1-
.. note:: Data Copy Can Produce Duplicate Events
2-
3-
If any system changes the data in the database while the source connector
4-
converts existing data from it, MongoDB may produce duplicate change
5-
stream events to reflect the latest changes. Since the change stream
6-
events on which the data copy relies are idempotent, the copied data is
7-
eventually consistent.
1+
If any system changes the data in the database while the source connector
2+
converts existing data from it, MongoDB may produce duplicate change
3+
stream events to reflect the latest changes. Since the change stream
4+
events on which the data copy relies are idempotent, the copied data is
5+
eventually consistent.

‎source/includes/externalize-secrets.rst

+4-6
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
1-
.. important:: Avoid Exposing Your Authentication Credentials
2-
3-
To avoid exposing your authentication credentials in your
4-
``connection.uri`` setting, use a
5-
`ConfigProvider <https://docs.confluent.io/current/connect/security.html#externalizing-secrets>`__
6-
and set the appropriate configuration parameters.
1+
:gold:`IMPORTANT:` To avoid exposing your authentication credentials in your
2+
``connection.uri`` setting, use a
3+
`ConfigProvider <https://docs.confluent.io/current/connect/security.html#externalizing-secrets>`__
4+
and set the appropriate configuration parameters.
75

‎source/introduction/data-formats.txt

+2
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
.. _kafka-data-formats:
2+
13
============
24
Data Formats
35
============

‎source/sink-connector/configuration-properties/connector-message.txt

+10-11
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,9 @@ Settings
7575
When set to ``true``, the default value, the
7676
connector writes a batch of records as an ordered bulk write
7777
operation.
78-
| To learn more about bulk write operations, see the :ref:`Write
79-
Model Strategies page <sink-connector-bulk-write-ops>`.
78+
|
79+
| To learn more about bulk write operations, see
80+
:ref:`Bulk Write Operations <sink-connector-bulk-write-ops>`.
8081
|
8182
| **Default**: ``true``
8283
| **Accepted Values**: ``true`` or ``false``
@@ -111,15 +112,13 @@ Settings
111112
| The maximum number of tasks to create for this connector. The
112113
connector may create fewer than the maximum tasks specified if it
113114
cannot handle the level of parallelism you specify.
114-
115-
.. important:: Multiple Tasks May Process Messages Out of Order
116-
117-
If you specify a value greater than ``1``, the connector enables
118-
parallel processing of the tasks. If your topic has multiple
119-
partition logs, which enables the connector to read from the
120-
topic in parallel, the tasks may process the messages out of
121-
order.
122-
115+
|
116+
| :gold:`IMPORTANT:` If you specify a value greater than ``1``,
117+
the connector enables parallel processing of the tasks.
118+
If your topic has multiple partition logs, which enables
119+
the connector to read from the topic in parallel,
120+
the tasks may process the messages out of order.
121+
|
123122
| **Default**: ``1``
124123
| **Accepted Values**: An integer
125124

‎source/sink-connector/configuration-properties/error-handling.txt

+21-17
Original file line numberDiff line numberDiff line change
@@ -48,19 +48,21 @@ Settings
4848
| Whether to continue processing messages if the connector encounters
4949
an error. Allows the connector to override the ``errors.tolerance``
5050
Kafka cluster setting.
51+
|
5152
| When set to ``none``, the connector reports any error and
5253
blocks further processing of the rest of the messages.
54+
|
5355
| When set to ``all``, the connector ignores any problematic messages.
56+
|
5457
| When set to ``data``, the connector tolerates only data errors and
5558
fails on all other errors.
59+
|
5660
| To learn more about error handling strategies, see the
5761
:ref:`<kafka-sink-handle-errors>` page.
58-
59-
.. note::
60-
61-
This property overrides the `errors.tolerance <https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html#errors-tolerance>`__
62-
property of the Connect Framework.
63-
62+
|
63+
| This property overrides the `errors.tolerance <https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html#errors-tolerance>`__
64+
| property of the Connect Framework.
65+
|
6466
| **Default:** Inherits the value from the ``errors.tolerance``
6567
setting.
6668
| **Accepted Values**: ``"none"`` or ``"all"``
@@ -73,16 +75,14 @@ Settings
7375
failed operations to the log file. The connector classifies
7476
errors as "tolerated" or "not tolerated" using the
7577
``errors.tolerance`` or ``mongo.errors.tolerance`` settings.
76-
78+
|
7779
| When set to ``true``, the connector logs both "tolerated" and
7880
"not tolerated" errors.
7981
| When set to ``false``, the connector logs "not tolerated" errors.
80-
81-
.. note::
82-
83-
This property overrides the `errors.log.enable <https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html#errors-log-enable>`__
84-
property of the Connect Framework.
85-
82+
|
83+
| This property overrides the `errors.log.enable <https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html#errors-log-enable>`__
84+
| property of the Connect Framework.
85+
|
8686
| **Default:** ``false``
8787
| **Accepted Values**: ``true`` or ``false``
8888

@@ -104,7 +104,8 @@ Settings
104104
| Name of topic to use as the dead letter queue. If blank, the
105105
connector does not send any invalid messages to the dead letter
106106
queue.
107-
| For more information about the dead letter queue, see the
107+
|
108+
| To learn more about the dead letter queue, see the
108109
:ref:`Dead Letter Queue Configuration Example <sink-dead-letter-queue-configuration-example>`.
109110
|
110111
| **Default:** ``""``
@@ -116,11 +117,13 @@ Settings
116117
| **Description:**
117118
| Whether the connector should include context headers when it
118119
writes messages to the dead letter queue.
120+
|
119121
| To learn more about the dead letter queue, see the
120122
:ref:`Dead Letter Queue Configuration Example <sink-dead-letter-queue-configuration-example>`.
123+
|
121124
| To learn about the exceptions the connector defines and
122-
reports through context headers, see the
123-
:ref:`<sink-configuration-error-handling-dlq-errors>` section.
125+
reports through context headers, see
126+
:ref:`<sink-configuration-error-handling-dlq-errors>`.
124127
|
125128
| **Default:** ``false``
126129
| **Accepted Values**: ``true`` or ``false``
@@ -132,7 +135,8 @@ Settings
132135
| The number of nodes on which to replicate the dead letter queue
133136
topic. If you are running a single-node Kafka cluster, you must
134137
set this to ``1``.
135-
| For more information about the dead letter queue, see the
138+
|
139+
| To learn more about the dead letter queue, see the
136140
:ref:`Dead Letter Queue Configuration Example <sink-dead-letter-queue-configuration-example>`.
137141
|
138142
| **Default:** ``3``

‎source/sink-connector/configuration-properties/id-strategy.txt

+3-2
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,9 @@ Settings
7676
|
7777
| **Description:**
7878
| Whether the connector should delete documents when the key value
79-
matches a document in MongoDB and the value field is null. This
80-
setting applies when you specify an id generation strategy that
79+
matches a document in MongoDB and the value field is null.
80+
|
81+
| This setting applies when you specify an id generation strategy that
8182
operates on the key document such as ``FullKeyStrategy``,
8283
``PartialKeyStrategy``, and ``ProvidedInKeyStrategy``.
8384
|

‎source/sink-connector/configuration-properties/kafka-topic.txt

+9-15
Original file line numberDiff line numberDiff line change
@@ -42,10 +42,8 @@ Settings
4242
| **Description:**
4343
| A list of Kafka topics that the sink connector watches.
4444

45-
.. note::
46-
47-
You can define either the ``topics`` or the ``topics.regex``
48-
setting, but not both.
45+
You can define either the ``topics`` or the ``topics.regex``
46+
setting, but not both.
4947

5048
| **Accepted Values**: A comma-separated list of valid Kafka topics
5149

@@ -58,20 +56,16 @@ Settings
5856
| A regular expression that matches the Kafka topics that the sink
5957
connector watches.
6058

61-
.. example::
62-
63-
.. code-block:: properties
64-
65-
topics.regex=activity\\.\\w+\\.clicks$
59+
For example, the following regex matches topic names such as
60+
"activity.landing.clicks" and "activity.support.clicks".
61+
It does not match the topic names "activity.landing.views" and "activity.clicks".
6662

67-
This regex matches topic names such as "activity.landing.clicks"
68-
and "activity.support.clicks". It does not match the topic names
69-
"activity.landing.views" and "activity.clicks".
63+
.. code-block:: properties
7064

71-
.. note::
65+
topics.regex=activity\\.\\w+\\.clicks$
7266

73-
You can define either the ``topics`` or the ``topics.regex``
74-
setting, but not both.
67+
You can define either the ``topics`` or the ``topics.regex``
68+
setting, but not both.
7569

7670
| **Accepted Values**: A valid regular expression pattern using ``java.util.regex.Pattern``.
7771

‎source/sink-connector/configuration-properties/mongodb-connection.txt

+2
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ Settings
6868
| When set to ``true``, if the connector calls a command on your
6969
MongoDB instance that's deprecated in the declared {+stable-api+}
7070
version, it raises an exception.
71+
|
7172
| You can set the API version with the ``server.api.version``
7273
configuration option. For more information on the {+stable-api+}, see
7374
the MongoDB manual entry on the
@@ -83,6 +84,7 @@ Settings
8384
| When set to ``true``, if the connector calls a command on your
8485
MongoDB instance that's not covered in the declared {+stable-api+}
8586
version, it raises an exception.
87+
|
8688
| You can set the API version with the ``server.api.version``
8789
configuration option. For more information on the {+stable-api+}, see
8890
the MongoDB manual entry on the

‎source/sink-connector/configuration-properties/mongodb-namespace.txt

+8-8
Original file line numberDiff line numberDiff line change
@@ -43,14 +43,12 @@ Settings
4343
database or collection in which to sink the data. The default
4444
``DefaultNamespaceMapper`` uses values specified in the
4545
``database`` and ``collection`` properties.
46-
47-
.. seealso::
48-
49-
The connector includes an alternative class for specifying the
50-
database and collection called ``FieldPathNamespaceMapper``. See
51-
the :ref:`FieldPathNamespaceMapper settings <fieldpathnamespacemapper-settings>`
52-
for more information.
53-
46+
|
47+
| The connector includes an alternative class for specifying the
48+
| database and collection called ``FieldPathNamespaceMapper``. See
49+
| :ref:`FieldPathNamespaceMapper Settings <fieldpathnamespacemapper-settings>`
50+
| for more information.
51+
|
5452
| **Default**:
5553

5654
.. code-block:: none
@@ -149,10 +147,12 @@ You can use the following settings to customize the behavior of the
149147
| **Description**:
150148
| Whether to throw an exception when either the document is missing the
151149
mapped field or it has an invalid BSON type.
150+
|
152151
| When set to ``true``, the connector does not process documents
153152
missing the mapped field or that contain an invalid BSON type.
154153
The connector may halt or skip processing depending on the related
155154
error-handling configuration settings.
155+
|
156156
| When set to ``false``, if a document is missing the mapped field or
157157
if it has an invalid BSON type, the connector defaults to
158158
writing to the specified ``database`` and ``collection`` settings.

‎source/sink-connector/configuration-properties/post-processors.txt

+7-13
Original file line numberDiff line numberDiff line change
@@ -40,13 +40,10 @@ Settings
4040
| **Description:**
4141
| A list of post-processor classes the connector should apply to
4242
process the data before saving it to MongoDB.
43-
44-
.. seealso::
45-
46-
For more information on post-processors and examples of
47-
their usage, see the section on
48-
:doc:`Post-processors </sink-connector/fundamentals/post-processors>`.
49-
43+
|
44+
| To learn more about post-processors and see examples of
45+
| their usage, see
46+
| :doc:`Sink Connector Post Processors </sink-connector/fundamentals/post-processors>`.
5047
|
5148
| **Default**:
5249

@@ -129,12 +126,9 @@ Settings
129126
| **Description:**
130127
| The class that specifies the ``WriteModelStrategy`` the connector should
131128
use for :manual:`Bulk Writes </core/bulk-write-operations/index.html>`.
132-
133-
.. seealso::
134-
135-
For information on how to create your own strategy, see
136-
:ref:`<kafka-sink-write-model-create-custom-strategy>`.
137-
129+
|
130+
| To learn more about how to create your own strategy, see
131+
| :ref:`<kafka-sink-write-model-create-custom-strategy>`.
138132
|
139133
| **Default**:
140134

‎source/sink-connector/configuration-properties/time-series.txt

+18-14
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,11 @@ Settings
5757
| The date format pattern the connector should use to convert the
5858
source data contained in the field specified by the
5959
``timeseries.timefield`` setting.
60-
The connector passes the date format pattern to the Java
60+
|
61+
| The connector passes the date format pattern to the Java
6162
`DateTimeFormatter.ofPattern(pattern, locale) <https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#ofPattern-java.lang.String-java.util.Locale->`__
6263
method to perform date and time conversions on the time field.
64+
|
6365
| If the date value from the source data only contains date information,
6466
the connector sets the time information to the start of the specified
6567
day. If the date value does not contain the timezone offset, the
@@ -79,6 +81,7 @@ Settings
7981
| **Description:**
8082
| Whether to convert the data in the field into the BSON ``Date``
8183
format.
84+
|
8285
| When set to ``true``, the connector uses the milliseconds
8386
after epoch and discards fractional parts if the value is
8487
a number. If the value is a string, the connector uses the
@@ -99,8 +102,9 @@ Settings
99102
|
100103
| **Description:**
101104
| Which ``DateTimeFormatter`` locale language tag to use with the date
102-
format pattern (e.g. ``"en-US"``). For more information on
103-
locales, see the Java SE documentation of `Locale <https://docs.oracle.com/javase/8/docs/api/java/util/Locale.html>`__.
105+
format pattern (e.g. ``"en-US"``).
106+
|
107+
| To learn more about locales, see the Java SE documentation of `Locale <https://docs.oracle.com/javase/8/docs/api/java/util/Locale.html>`__.
104108
|
105109
| **Default**: ``ROOT``
106110
| **Accepted Values**: A valid ``Locale`` language tag format
@@ -111,12 +115,10 @@ Settings
111115
| **Description:**
112116
| Which top-level field to read from the source data to describe
113117
a group of related time series documents.
114-
115-
.. important::
116-
117-
This field must not be the ``_id`` field nor the field you specified
118-
in the ``timeseries.timefield`` setting.
119-
118+
|
119+
| :gold:`IMPORTANT:` This field must not be the ``_id`` field nor the field you specified
120+
in the ``timeseries.timefield`` setting.
121+
|
120122
| **Default**: ``""``
121123
| **Accepted Values**: An empty string or the name of a field
122124
that contains any BSON type except ``BsonArray``.
@@ -128,8 +130,9 @@ Settings
128130
| The number of seconds MongoDB should wait before automatically
129131
removing the time series collection data. The connector disables
130132
timed expiry when the setting value is less than ``1``.
131-
For more information on this collection setting, see the MongoDB
132-
Server Manual page on :manual:`Automatic Removal for Time Series Collections </core/timeseries/timeseries-automatic-removal/>`.
133+
|
134+
| To learn more, see :manual:`Set up Automatic Removal for Time Series Collections </core/timeseries/timeseries-automatic-removal/>`
135+
in the MongoDB manual.
133136
|
134137
| **Default**: ``0``
135138
| **Accepted Values**: An integer
@@ -140,9 +143,10 @@ Settings
140143
|
141144
| **Description:**
142145
| The expected interval between subsequent measurements of your
143-
source data. For more information on this setting, see the
144-
MongoDB server manual page on :manual:`Granularity for Time
145-
Series Data </core/timeseries/timeseries-granularity/>`.
146+
source data.
147+
|
148+
| To learn more, see :manual:`Set Granularity for Time Series Data
149+
</core/timeseries/timeseries-granularity/>` in the MongoDB manual.
146150
|
147151
| *Optional*
148152
| **Default**: ``""``

‎source/sink-connector/configuration-properties/topic-override.txt

+9-13
Original file line numberDiff line numberDiff line change
@@ -40,19 +40,15 @@ Settings
4040
| **Description:**
4141
| Specify a topic and property name to override the corresponding
4242
global or default property setting.
43-
44-
.. example::
45-
46-
The ``topic.override.foo.collection=bar`` setting instructs the
47-
sink connector to store data from the ``foo`` topic in the ``bar``
48-
collection.
49-
50-
.. note::
51-
52-
You can specify any valid configuration setting in the
53-
``<propertyName>`` segment on a per-topic basis except
54-
``connection.uri`` and ``topics``.
55-
43+
|
44+
| For example, the ``topic.override.foo.collection=bar`` setting instructs
45+
| the sink connector to store data from the ``foo`` topic in the ``bar``
46+
| collection.
47+
|
48+
| You can specify any valid configuration setting in the
49+
| ``<propertyName>`` segment on a per-topic basis except
50+
| ``connection.uri`` and ``topics``.
51+
|
5652
| **Default**: ``""``
5753
| **Accepted Values**: Accepted values specific to the overridden property
5854

0 commit comments

Comments
 (0)