Skip to content

Commit 26e6c98

Browse files
elyse-mdbcarriecwk
authored and
your proper name
committed
(DOCSP-43962) [Kafka] Kafka connector nested components (#167)
* (DOCSP-43962) Unnested 1 example in tip. * (DOCSP-43962) Removing the subjunctive should * (DOCSP-42962) Unnested 1 important admonition in table. * (DOCSP-42962) Typo fix. * (DOCSP-42962) Unnested 1 important admonition in table. * (DOCSP-42962) Unnested 1 tip in table. * (DOCSP-42962) Fixing indentation error and unnesting 1 important admonition in table. * (DOCSP-42962) Spacing. * (DOCSP-42962) Build errors. * (DOCSP-42962) Unnested 1 example and 1 note in table. * (DOCSP-42962) Unnested 2 examples and 1 note in include. * (DOCSP-42962) Replace quotation marks with monospace. * (DOCSP-42962) Unnested 2 notes in table. * (DOCSP-42962) consistency. * (DOCSP-42962) Edited for consistency and unnested 2 tips. * (DOCSP-42962) Removing extra line breaks. * (DOCSP-42962) Unnested 2 important admonitions in table. * (DOCSP-42962) Fix rendering error. * (DOCSP-42962) Fixing rendering issues. * (DOCSP-42962) Unnested 2 notes and 1 example in table. * (DOCSP-42962) Unnested 3 important admonitions * (DOCSP-42962) Spacing fix. * (DOCSP-42962) Unnested 1 example and 3 tips. * (DOCSP-42962) Unnested 1 note, 2 tips, 1 important admonition. * (DOCSP-42962) Spacing. * (DOCSP-42962) Spacing. * (DOCSP-42962) Normalizing spacing and adding reference to the Data Formats page. * (DOCSP-42962) Unnested 3 tips, 1 example and 1 important admonition in table. * (DOCSP-42962) Regularizing usage of pipeline operator line breaks. * (DOCSP-42962) Spacing. * (DOCSP-42962) Spacing. * (DOCSP-42962) Normalizing spacing across pages. * Update source/source-connector/configuration-properties/kafka-topic.txt Co-authored-by: carriecwk <[email protected]> * Update source/source-connector/configuration-properties/kafka-topic.txt Co-authored-by: carriecwk <[email protected]> --------- Co-authored-by: Elyse Foreman <[email protected]> Co-authored-by: carriecwk <[email protected]>
1 parent f0e9977 commit 26e6c98

20 files changed

+278
-311
lines changed

‎source/includes/copy-existing-admonition.rst

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
1-
.. note:: Data Copy Can Produce Duplicate Events
2-
3-
If any system changes the data in the database while the source connector
4-
converts existing data from it, MongoDB may produce duplicate change
5-
stream events to reflect the latest changes. Since the change stream
6-
events on which the data copy relies are idempotent, the copied data is
7-
eventually consistent.
1+
If any system changes the data in the database while the source connector
2+
converts existing data from it, MongoDB may produce duplicate change
3+
stream events to reflect the latest changes. Since the change stream
4+
events on which the data copy relies are idempotent, the copied data is
5+
eventually consistent.

‎source/includes/externalize-secrets.rst

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
1-
.. important:: Avoid Exposing Your Authentication Credentials
2-
3-
To avoid exposing your authentication credentials in your
4-
``connection.uri`` setting, use a
5-
`ConfigProvider <https://docs.confluent.io/current/connect/security.html#externalizing-secrets>`__
6-
and set the appropriate configuration parameters.
1+
:gold:`IMPORTANT:` To avoid exposing your authentication credentials in your
2+
``connection.uri`` setting, use a
3+
`ConfigProvider <https://docs.confluent.io/current/connect/security.html#externalizing-secrets>`__
4+
and set the appropriate configuration parameters.
75

‎source/introduction/data-formats.txt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
.. _kafka-data-formats:
2+
13
============
24
Data Formats
35
============

‎source/sink-connector/configuration-properties/connector-message.txt

Lines changed: 10 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,9 @@ Settings
7575
When set to ``true``, the default value, the
7676
connector writes a batch of records as an ordered bulk write
7777
operation.
78-
| To learn more about bulk write operations, see the :ref:`Write
79-
Model Strategies page <sink-connector-bulk-write-ops>`.
78+
|
79+
| To learn more about bulk write operations, see
80+
:ref:`Bulk Write Operations <sink-connector-bulk-write-ops>`.
8081
|
8182
| **Default**: ``true``
8283
| **Accepted Values**: ``true`` or ``false``
@@ -111,15 +112,13 @@ Settings
111112
| The maximum number of tasks to create for this connector. The
112113
connector may create fewer than the maximum tasks specified if it
113114
cannot handle the level of parallelism you specify.
114-
115-
.. important:: Multiple Tasks May Process Messages Out of Order
116-
117-
If you specify a value greater than ``1``, the connector enables
118-
parallel processing of the tasks. If your topic has multiple
119-
partition logs, which enables the connector to read from the
120-
topic in parallel, the tasks may process the messages out of
121-
order.
122-
115+
|
116+
| :gold:`IMPORTANT:` If you specify a value greater than ``1``,
117+
the connector enables parallel processing of the tasks.
118+
If your topic has multiple partition logs, which enables
119+
the connector to read from the topic in parallel,
120+
the tasks may process the messages out of order.
121+
|
123122
| **Default**: ``1``
124123
| **Accepted Values**: An integer
125124

‎source/sink-connector/configuration-properties/error-handling.txt

Lines changed: 23 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -41,17 +41,21 @@ Settings
4141
| Whether to continue processing messages if the connector encounters
4242
an error. Allows the connector to override the ``errors.tolerance``
4343
Kafka cluster setting.
44+
|
4445
| When set to ``none``, the connector reports any error and
4546
blocks further processing of the rest of the messages.
47+
|
4648
| When set to ``all``, the connector ignores any problematic messages.
49+
|
50+
| When set to ``data``, the connector tolerates only data errors and
51+
fails on all other errors.
52+
|
4753
| To learn more about error handling strategies, see the
4854
:ref:`<kafka-sink-handle-errors>` page.
49-
50-
.. note::
51-
52-
This property overrides the `errors.tolerance <https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html#errors-tolerance>`__
53-
property of the Connect Framework.
54-
55+
|
56+
| This property overrides the `errors.tolerance <https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html#errors-tolerance>`__
57+
| property of the Connect Framework.
58+
|
5559
| **Default:** Inherits the value from the ``errors.tolerance``
5660
setting.
5761
| **Accepted Values**: ``"none"`` or ``"all"``
@@ -64,16 +68,14 @@ Settings
6468
failed operations to the log file. The connector classifies
6569
errors as "tolerated" or "not tolerated" using the
6670
``errors.tolerance`` or ``mongo.errors.tolerance`` settings.
67-
71+
|
6872
| When set to ``true``, the connector logs both "tolerated" and
6973
"not tolerated" errors.
7074
| When set to ``false``, the connector logs "not tolerated" errors.
71-
72-
.. note::
73-
74-
This property overrides the `errors.log.enable <https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html#errors-log-enable>`__
75-
property of the Connect Framework.
76-
75+
|
76+
| This property overrides the `errors.log.enable <https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html#errors-log-enable>`__
77+
| property of the Connect Framework.
78+
|
7779
| **Default:** ``false``
7880
| **Accepted Values**: ``true`` or ``false``
7981

@@ -95,7 +97,8 @@ Settings
9597
| Name of topic to use as the dead letter queue. If blank, the
9698
connector does not send any invalid messages to the dead letter
9799
queue.
98-
| For more information about the dead letter queue, see the
100+
|
101+
| To learn more about the dead letter queue, see the
99102
:ref:`Dead Letter Queue Configuration Example <sink-dead-letter-queue-configuration-example>`.
100103
|
101104
| **Default:** ``""``
@@ -107,11 +110,13 @@ Settings
107110
| **Description:**
108111
| Whether the connector should include context headers when it
109112
writes messages to the dead letter queue.
113+
|
110114
| To learn more about the dead letter queue, see the
111115
:ref:`Dead Letter Queue Configuration Example <sink-dead-letter-queue-configuration-example>`.
116+
|
112117
| To learn about the exceptions the connector defines and
113-
reports through context headers, see the
114-
:ref:`<sink-configuration-error-handling-dlq-errors>` section.
118+
reports through context headers, see
119+
:ref:`<sink-configuration-error-handling-dlq-errors>`.
115120
|
116121
| **Default:** ``false``
117122
| **Accepted Values**: ``true`` or ``false``
@@ -123,7 +128,8 @@ Settings
123128
| The number of nodes on which to replicate the dead letter queue
124129
topic. If you are running a single-node Kafka cluster, you must
125130
set this to ``1``.
126-
| For more information about the dead letter queue, see the
131+
|
132+
| To learn more about the dead letter queue, see the
127133
:ref:`Dead Letter Queue Configuration Example <sink-dead-letter-queue-configuration-example>`.
128134
|
129135
| **Default:** ``3``

‎source/sink-connector/configuration-properties/id-strategy.txt

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,9 @@ Settings
7676
|
7777
| **Description:**
7878
| Whether the connector should delete documents when the key value
79-
matches a document in MongoDB and the value field is null. This
80-
setting applies when you specify an id generation strategy that
79+
matches a document in MongoDB and the value field is null.
80+
|
81+
| This setting applies when you specify an id generation strategy that
8182
operates on the key document such as ``FullKeyStrategy``,
8283
``PartialKeyStrategy``, and ``ProvidedInKeyStrategy``.
8384
|

‎source/sink-connector/configuration-properties/kafka-topic.txt

Lines changed: 9 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -42,10 +42,8 @@ Settings
4242
| **Description:**
4343
| A list of Kafka topics that the sink connector watches.
4444

45-
.. note::
46-
47-
You can define either the ``topics`` or the ``topics.regex``
48-
setting, but not both.
45+
You can define either the ``topics`` or the ``topics.regex``
46+
setting, but not both.
4947

5048
| **Accepted Values**: A comma-separated list of valid Kafka topics
5149

@@ -58,20 +56,16 @@ Settings
5856
| A regular expression that matches the Kafka topics that the sink
5957
connector watches.
6058

61-
.. example::
62-
63-
.. code-block:: properties
64-
65-
topics.regex=activity\\.\\w+\\.clicks$
59+
For example, the following regex matches topic names such as
60+
"activity.landing.clicks" and "activity.support.clicks".
61+
It does not match the topic names "activity.landing.views" and "activity.clicks".
6662

67-
This regex matches topic names such as "activity.landing.clicks"
68-
and "activity.support.clicks". It does not match the topic names
69-
"activity.landing.views" and "activity.clicks".
63+
.. code-block:: properties
7064

71-
.. note::
65+
topics.regex=activity\\.\\w+\\.clicks$
7266

73-
You can define either the ``topics`` or the ``topics.regex``
74-
setting, but not both.
67+
You can define either the ``topics`` or the ``topics.regex``
68+
setting, but not both.
7569

7670
| **Accepted Values**: A valid regular expression pattern using ``java.util.regex.Pattern``.
7771

‎source/sink-connector/configuration-properties/mongodb-connection.txt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ Settings
6868
| When set to ``true``, if the connector calls a command on your
6969
MongoDB instance that's deprecated in the declared {+stable-api+}
7070
version, it raises an exception.
71+
|
7172
| You can set the API version with the ``server.api.version``
7273
configuration option. For more information on the {+stable-api+}, see
7374
the MongoDB manual entry on the
@@ -83,6 +84,7 @@ Settings
8384
| When set to ``true``, if the connector calls a command on your
8485
MongoDB instance that's not covered in the declared {+stable-api+}
8586
version, it raises an exception.
87+
|
8688
| You can set the API version with the ``server.api.version``
8789
configuration option. For more information on the {+stable-api+}, see
8890
the MongoDB manual entry on the

‎source/sink-connector/configuration-properties/mongodb-namespace.txt

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -43,14 +43,12 @@ Settings
4343
database or collection in which to sink the data. The default
4444
``DefaultNamespaceMapper`` uses values specified in the
4545
``database`` and ``collection`` properties.
46-
47-
.. seealso::
48-
49-
The connector includes an alternative class for specifying the
50-
database and collection called ``FieldPathNamespaceMapper``. See
51-
the :ref:`FieldPathNamespaceMapper settings <fieldpathnamespacemapper-settings>`
52-
for more information.
53-
46+
|
47+
| The connector includes an alternative class for specifying the
48+
| database and collection called ``FieldPathNamespaceMapper``. See
49+
| :ref:`FieldPathNamespaceMapper Settings <fieldpathnamespacemapper-settings>`
50+
| for more information.
51+
|
5452
| **Default**:
5553

5654
.. code-block:: none
@@ -149,10 +147,12 @@ You can use the following settings to customize the behavior of the
149147
| **Description**:
150148
| Whether to throw an exception when either the document is missing the
151149
mapped field or it has an invalid BSON type.
150+
|
152151
| When set to ``true``, the connector does not process documents
153152
missing the mapped field or that contain an invalid BSON type.
154153
The connector may halt or skip processing depending on the related
155154
error-handling configuration settings.
155+
|
156156
| When set to ``false``, if a document is missing the mapped field or
157157
if it has an invalid BSON type, the connector defaults to
158158
writing to the specified ``database`` and ``collection`` settings.

‎source/sink-connector/configuration-properties/post-processors.txt

Lines changed: 7 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -40,13 +40,10 @@ Settings
4040
| **Description:**
4141
| A list of post-processor classes the connector should apply to
4242
process the data before saving it to MongoDB.
43-
44-
.. seealso::
45-
46-
For more information on post-processors and examples of
47-
their usage, see the section on
48-
:doc:`Post-processors </sink-connector/fundamentals/post-processors>`.
49-
43+
|
44+
| To learn more about post-processors and see examples of
45+
| their usage, see
46+
| :doc:`Sink Connector Post Processors </sink-connector/fundamentals/post-processors>`.
5047
|
5148
| **Default**:
5249

@@ -129,12 +126,9 @@ Settings
129126
| **Description:**
130127
| The class that specifies the ``WriteModelStrategy`` the connector should
131128
use for :manual:`Bulk Writes </core/bulk-write-operations/index.html>`.
132-
133-
.. seealso::
134-
135-
For information on how to create your own strategy, see
136-
:ref:`<kafka-sink-write-model-create-custom-strategy>`.
137-
129+
|
130+
| To learn more about how to create your own strategy, see
131+
| :ref:`<kafka-sink-write-model-create-custom-strategy>`.
138132
|
139133
| **Default**:
140134

‎source/sink-connector/configuration-properties/time-series.txt

Lines changed: 18 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -53,9 +53,11 @@ Settings
5353
| The date format pattern the connector should use to convert the
5454
source data contained in the field specified by the
5555
``timeseries.timefield`` setting.
56-
The connector passes the date format pattern to the Java
56+
|
57+
| The connector passes the date format pattern to the Java
5758
`DateTimeFormatter.ofPattern(pattern, locale) <https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#ofPattern-java.lang.String-java.util.Locale->`__
5859
method to perform date and time conversions on the time field.
60+
|
5961
| If the date value from the source data only contains date information,
6062
the connector sets the time information to the start of the specified
6163
day. If the date value does not contain the timezone offset, the
@@ -75,6 +77,7 @@ Settings
7577
| **Description:**
7678
| Whether to convert the data in the field into the BSON ``Date``
7779
format.
80+
|
7881
| When set to ``true``, the connector uses the milliseconds
7982
after epoch and discards fractional parts if the value is
8083
a number. If the value is a string, the connector uses the
@@ -95,8 +98,9 @@ Settings
9598
|
9699
| **Description:**
97100
| Which ``DateTimeFormatter`` locale language tag to use with the date
98-
format pattern (e.g. ``"en-US"``). For more information on
99-
locales, see the Java SE documentation of `Locale <https://docs.oracle.com/javase/8/docs/api/java/util/Locale.html>`__.
101+
format pattern (e.g. ``"en-US"``).
102+
|
103+
| To learn more about locales, see the Java SE documentation of `Locale <https://docs.oracle.com/javase/8/docs/api/java/util/Locale.html>`__.
100104
|
101105
| **Default**: ``ROOT``
102106
| **Accepted Values**: A valid ``Locale`` language tag format
@@ -107,12 +111,10 @@ Settings
107111
| **Description:**
108112
| Which top-level field to read from the source data to describe
109113
a group of related time series documents.
110-
111-
.. important::
112-
113-
This field must not be the ``_id`` field nor the field you specified
114-
in the ``timeseries.timefield`` setting.
115-
114+
|
115+
| :gold:`IMPORTANT:` This field must not be the ``_id`` field nor the field you specified
116+
in the ``timeseries.timefield`` setting.
117+
|
116118
| **Default**: ``""``
117119
| **Accepted Values**: An empty string or the name of a field
118120
that contains any BSON type except ``BsonArray``.
@@ -124,8 +126,9 @@ Settings
124126
| The number of seconds MongoDB should wait before automatically
125127
removing the time series collection data. The connector disables
126128
timed expiry when the setting value is less than ``1``.
127-
For more information on this collection setting, see the MongoDB
128-
Server Manual page on :manual:`Automatic Removal for Time Series Collections </core/timeseries/timeseries-automatic-removal/>`.
129+
|
130+
| To learn more, see :manual:`Set up Automatic Removal for Time Series Collections </core/timeseries/timeseries-automatic-removal/>`
131+
in the MongoDB manual.
129132
|
130133
| **Default**: ``0``
131134
| **Accepted Values**: An integer
@@ -136,9 +139,10 @@ Settings
136139
|
137140
| **Description:**
138141
| The expected interval between subsequent measurements of your
139-
source data. For more information on this setting, see the
140-
MongoDB Server Manual page on :manual:`Granularity for Time
141-
Series Data </core/timeseries/timeseries-granularity/>`.
142+
source data.
143+
|
144+
| To learn more, see :manual:`Set Granularity for Time Series Data
145+
</core/timeseries/timeseries-granularity/>` in the MongoDB manual.
142146
|
143147
| *Optional*
144148
| **Default**: ``""``

‎source/sink-connector/configuration-properties/topic-override.txt

Lines changed: 9 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -40,19 +40,15 @@ Settings
4040
| **Description:**
4141
| Specify a topic and property name to override the corresponding
4242
global or default property setting.
43-
44-
.. example::
45-
46-
The ``topic.override.foo.collection=bar`` setting instructs the
47-
sink connector to store data from the ``foo`` topic in the ``bar``
48-
collection.
49-
50-
.. note::
51-
52-
You can specify any valid configuration setting in the
53-
``<propertyName>`` segment on a per-topic basis except
54-
``connection.uri`` and ``topics``.
55-
43+
|
44+
| For example, the ``topic.override.foo.collection=bar`` setting instructs
45+
| the sink connector to store data from the ``foo`` topic in the ``bar``
46+
| collection.
47+
|
48+
| You can specify any valid configuration setting in the
49+
| ``<propertyName>`` segment on a per-topic basis except
50+
| ``connection.uri`` and ``topics``.
51+
|
5652
| **Default**: ``""``
5753
| **Accepted Values**: Accepted values specific to the overridden property
5854

0 commit comments

Comments
 (0)