Skip to content

Commit 25fcdaf

Browse files
authored
docs(review): minor updates to mm2 config (#12273)
Signed-off-by: prmellor <pmellor@redhat.com>
1 parent b7ff07b commit 25fcdaf

8 files changed

+98
-55
lines changed

documentation/modules/configuring/con-config-kafka-connect-multiple-instances.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ spec:
2828
<3> Kafka topic that stores connector offsets.
2929
<4> Kafka topic that stores connector and task status updates.
3030

31-
NOTE: Values for the three topics must be the same for all instances with the same `group.id`.
31+
This requirement applies across all Kafka Connect–based deployments, including Kafka Connect and MirrorMaker 2 instances that target the same Kafka cluster.
3232

3333
Unless you use different settings for each instance, all instances form a cluster and use the same internal topics.
3434
Multiple instances attempting to use the same internal topics will cause unexpected errors, so you must change the values of these properties for each instance.

documentation/modules/configuring/con-config-mirrormaker2-connect-workers.adoc

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,11 @@
88
= Tuning Kafka Connect worker settings
99

1010
[role="_abstract"]
11-
Adjust Kafka Connect worker behavior for MirrorMaker 2 by adding configuration to your target cluster's `config` section.
11+
Adjust Kafka Connect worker runtime behavior for MirrorMaker 2 by configuring worker properties on the Kafka Connect cluster that runs the MirrorMaker 2 connectors.
12+
13+
Kafka Connect workers running MirrorMaker 2 connectors are configured to store their internal state in a single target Kafka cluster.
14+
The worker configuration specifies Kafka Connect runtime settings, such as task rebalance behavior, shutdown handling, and how frequently offsets are written to internal topics.
15+
To preserve records without modification, configure byte array converters for record keys and values.
1216

1317
.Example Kafka Connect configuration
1418
[source,yaml,subs="+quotes,attributes"]
@@ -28,12 +32,19 @@ spec:
2832
offsetStorageTopic: my-mirror-maker2-offset
2933
statusStorageTopic: my-mirror-maker2-status
3034
config: # <2>
35+
# Key and value converters
36+
key.converter: org.apache.kafka.connect.converters.ByteArrayConverter
37+
value.converter: org.apache.kafka.connect.converters.ByteArrayConverter
38+
# Worker shutdown and rebalance behavior
3139
task.shutdown.graceful.timeout.ms: 30000
3240
scheduled.rebalance.max.delay.ms: 300000
41+
# Offset flush interval
3342
offset.flush.interval.ms: 10000
34-
config.storage.replication.factor: 1
35-
offset.storage.replication.factor: 1
36-
status.storage.replication.factor: 1
43+
# Kafka Connect internal topic replication
44+
config.storage.replication.factor: -1
45+
offset.storage.replication.factor: -1
46+
status.storage.replication.factor: -1
3747
----
3848
<1> The target cluster configuration. The target cluster is always used to store Kafka Connect's internal topics.
39-
<2> These properties configure the Kafka Connect workers in this cluster.
49+
<2> These properties configure the Kafka Connect workers created by this `KafkaMirrorMaker2` resource.
50+
The settings apply to all MirrorMaker 2 connectors managed by the resource.

documentation/modules/configuring/con-config-mirrormaker2-connectors.adoc

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
= Configuring MirrorMaker 2 connectors
99

1010
[role="_abstract"]
11-
MirrorMaker 2 uses internal connectors to synchronize data between Kafka clusters.
11+
MirrorMaker 2 comprises a set of connectors to synchronize data between Kafka clusters.
1212
You configure these connectors in the `KafkaMirrorMaker2` custom resource.
1313

1414
MirrorMaker 2 consists of the following connectors:
@@ -20,11 +20,14 @@ MirrorMaker 2 consists of the following connectors:
2020
`MirrorSourceConnector` and `MirrorCheckpointConnector` are configured in the
2121
`spec.mirrors` section of the `KafkaMirrorMaker2` custom resource using the `sourceConnector` and `checkpointConnector` properties.
2222

23+
These connectors run in the Kafka Connect cluster associated with the target Kafka cluster.
24+
2325
`MirrorHeartbeatConnector` is not configurable through the `KafkaMirrorMaker2` custom resource.
2426
To run a `MirrorHeartbeatConnector`, use a separate `KafkaConnect` and `KafkaConnector` custom resource.
27+
Because it produces heartbeat records to the source Kafka cluster, the `MirrorHeartbeatConnector` typically runs in a Kafka Connect cluster that stores its state in the source cluster.
2528

2629
For a full list of supported MirrorMaker 2 connector properties, including common configuration and connector-specific options, see the {kafkaDoc}.
2730

28-
WARNING: Configure the following properties with the *same values in both the source and checkpoint connectors*:
31+
WARNING: Configure the following properties with the same values in both the `MirrorSourceConnector` and `MirrorCheckpointConnector`:
2932
`replication.policy.class`, `replication.policy.separator`, `offset-syncs.topic.location`, and `topic.filter.class`.
3033
If these properties do not match, replication or offset synchronization can fail.

documentation/modules/configuring/con-config-mirrormaker2-producers-consumers.adoc

Lines changed: 20 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -9,50 +9,57 @@
99

1010
[role="_abstract"]
1111
MirrorMaker 2 connectors use internal producers and consumers.
12-
If needed, you can configure these producers and consumers to override the default settings.
12+
If required, you can configure these producers and consumers to override the default settings.
1313

1414
For example, you can increase the `batch.size` for the source producer that sends topics to the target Kafka cluster to better accommodate large volumes of messages.
1515

1616
IMPORTANT: Producer and consumer configuration options depend on the MirrorMaker 2 implementation, and may be subject to change.
1717

1818
The following tables describe the producers and consumers for each of the connectors and where you can add configuration.
1919

20+
Configuration can be applied as follows:
21+
22+
* Overriding the configuration used by the internal producer or consumer by using the `producer.override.` or `consumer.override.` properties.
23+
* Configuring the internal producer or consumer directly in the connector by using the `producer.` or `consumer.` properties.
24+
2025
.Source connector producers and consumers
21-
[cols="1,1a,3m",options="header"]
26+
[cols="1,2a,2m",options="header"]
2227
|===
2328

2429
|Type
2530
|Description
2631
|Configuration
2732

2833
|Producer
29-
|Sends topic messages to the target Kafka cluster. Consider tuning the configuration of this producer when it is handling large volumes of data.
30-
|mirrors.sourceConnector.config: producer.override.*
34+
|Sends replicated topic messages to the target Kafka cluster.
35+
Tune this producer when handling large volumes of data.
36+
|`producer.override.*`
3137

3238
|Producer
33-
|Writes to the `offset-syncs` topic, which maps the source and target offsets for replicated topic partitions.
34-
|mirrors.sourceConnector.config: producer.*
39+
|Writes to the `offset-syncs` topic, which maps source and target offsets.
40+
|`producer.*`
3541

3642
|Consumer
37-
|Retrieves topic messages from the source Kafka cluster.
38-
|mirrors.sourceConnector.config: consumer.*
43+
|Consumes topic messages from the source Kafka cluster.
44+
|`consumer.*`
45+
3946
|===
4047

4148
.Checkpoint connector producers and consumers
42-
[cols="1,1a,3m",options="header"]
49+
[cols="1,2a,2m",options="header"]
4350
|===
4451

4552
|Type
4653
|Description
4754
|Configuration
4855

4956
|Producer
50-
|Emits consumer offset checkpoints.
51-
|mirrors.checkpointConnector.config: producer.override.*
57+
|Emits consumer group offset checkpoints.
58+
|`producer.override.*`
5259

5360
|Consumer
54-
|Loads the `offset-syncs` topic.
55-
|mirrors.checkpointConnector.config: consumer.*
61+
|Consumes data from the `offset-syncs` topic.
62+
|`consumer.*`
5663

5764
|===
5865

documentation/modules/configuring/con-config-mirrormaker2-sync-acls.adoc

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -11,15 +11,20 @@
1111
MirrorMaker 2 can synchronize topic ACL rules from source to target clusters.
1212
Enable this feature by configuring the `MirrorSourceConnector`.
1313

14-
With `simple` authorization:
14+
When using simple ACL-based authorization:
1515

16-
* ACL rules apply to both source and target topics
17-
* Users with source topic access automatically get equivalent target topic access
16+
* Topic ACL rules can be synchronized between source and target topics
17+
* MirrorMaker 2 copies topic-related ACL rules from the source cluster to the target cluster
18+
* The `ALLOW ALL` topic ACL is not copied
1819

19-
IMPORTANT: This feature is not compatible with the User Operator, so disable by setting `sync.topic.acls.enabled` to `false`.
20+
Copying ACL rules does not guarantee equivalent access for all operations on the target cluster.
21+
Additional ACL rules may still be required to allow producers or consumers to operate as expected on the target cluster, for example, ACLs for other resource types such as consumer groups or transactional IDs.
22+
23+
IMPORTANT: This feature is not compatible with the User Operator.
24+
If you use the User Operator, set `sync.topic.acls.enabled` to `false`.
2025
When using OAuth 2.0 authorization, the setting has no effect.
2126

22-
.Example configuration to enable offset synchronization
27+
.Example configuration to enable ACL synchronization
2328
[source,yaml,subs="+quotes,attributes"]
2429
----
2530
apiVersion: {KafkaMirrorMaker2ApiVersion}
@@ -34,4 +39,4 @@ metadata:
3439
config:
3540
sync.topic.acls.enabled: "true" # <1>
3641
----
37-
<1> Replicates all source topic ACLs to the target cluster.
42+
<1> Copies topic-related ACL rules from the source cluster to the target cluster.

documentation/modules/configuring/con-config-mirrormaker2-sync.adoc

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -24,15 +24,17 @@ Configuring the `MirrorCheckpointConnector` to emit periodic offset checkpoints
2424
* Failover recovery as consumers switch to the target cluster at the correct position
2525

2626
Offset synchronization occurs at regular intervals.
27-
The `MirrorCheckpointConnector` emits checkpoints for all consumer groups.
27+
The `MirrorCheckpointConnector` emits checkpoints for all consumer groups enabled by the group configuration.
2828
However, applying those checkpoints in the target cluster requires the target group to be inactive.
29-
If consumers switch to the target cluster between checkpoints, they might reprocess some messages because the last synchronized offset lags behind the source.
29+
If consumers switch to the target cluster between checkpoints, they might reprocess some messages because offset synchronization is asynchronous and the last synchronized offset lags behind the source.
3030
This duplication is expected.
3131

3232
NOTE: The connector continues to emit checkpoints to the checkpoints topic even if the target consumer group is active.
33-
However, the offsets cannot be committed while the group has active members.
33+
However, the offsets cannot be committed while the consumer group has active members.
3434
In this case, the connector logs a warning and the offsets are not synchronized.
3535

36+
Each connector is configured separately.
37+
3638
.Example configuration to enable offset synchronization
3739
[source,yaml,subs="+quotes,attributes"]
3840
----
@@ -67,15 +69,15 @@ A value of -1 uses the broker’s default replication factor, which is typically
6769
<7> Replication factor for the internal checkpoints topic that stores the last committed offsets for each consumer group.
6870
A value of -1 uses the broker’s default replication factor, which is typically set to provide resilience (for example, 3).
6971

70-
TIP: If you have an application written in Java, you can use the `RemoteClusterUtils.java` utility to fetch offsets through the application.
71-
The utility fetches remote offsets for a consumer group from the `checkpoints` topic.
72+
TIP: Java applications can use the `RemoteClusterUtils` utility to fetch remote offsets for a consumer group from the checkpoints topic.
73+
74+
== Changing the location of the offset synchronization topic
7275

73-
== Changing the location of the consumer group offsets topic
76+
By default, the `offset-syncs` topic used by the `MirrorSourceConnector` is created in the source cluster.
77+
You can change this behavior by setting the `offset-syncs.topic.location` connector configuration to `target`.
7478

75-
The location of the `offset-syncs` topic is the `source` cluster by default.
76-
You can use the `offset-syncs.topic.location` connector configuration to change this to the `target` cluster.
77-
You need read/write access to the cluster that contains the topic.
78-
Using the target cluster as the location of the `offset-syncs` topic allows you to use MirrorMaker 2 even if you have only read access to the source cluster.
79+
Placing the `offset-syncs` topic on the target cluster allows the `MirrorSourceConnector` to run when it has read-only access to the source cluster.
80+
The connector must still have read and write access to the `offset-syncs` topic on the cluster where the topic is located.
7981

8082
== Listing the offsets of connectors
8183

@@ -170,5 +172,5 @@ data:
170172
To provide a custom owner reference, create the `ConfigMap` in advance and set the owner reference.
171173
<2> The `.json` property uses the connector name. Since `&#45;&#62;` characters are not allowed in `ConfigMap` keys, `&#45;&#62;` is changed to `--` in the connector name.
172174

173-
NOTE: It is possible to use configuration to xref:proc-altering-connector-offsets-{context}[alter] or xref:proc-resetting-connector-offsets-{context}[reset] connector offsets, though this is rarely necessary.
174-
175+
NOTE: It is possible to use configuration to xref:proc-altering-connector-offsets-{context}[alter] or xref:proc-resetting-connector-offsets-{context}[reset] connector offsets.
176+
This should be done with caution.

documentation/modules/configuring/con-config-mm2-multiple-instances.adoc

Lines changed: 25 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,25 @@
88
= Configuring MirrorMaker 2 using a shared target Kafka cluster
99

1010
[role="_abstract"]
11-
The group ID and names of the internal topics used by the Kafka Connect framework that MirrorMaker 2 runs on are configured in the `.spec.target` section.
12-
When running multiple instances of MirrorMaker 2, and they share the same target cluster, you must use unique values to make sure the MirrorMaker 2 clusters operate independently.
11+
MirrorMaker 2 runs on the Kafka Connect framework and uses internal Kafka topics to store configuration, offsets, and status information.
12+
13+
You can run multiple MirrorMaker 2 connectors in a single Kafka Connect cluster, for example to replicate data from multiple source Kafka clusters to the same target Kafka cluster.
14+
15+
When running multiple Kafka Connect clusters that replicate data to the same target Kafka cluster, you must configure unique Kafka Connect worker settings so that each cluster operates independently.
16+
17+
This requirement applies across all Kafka Connect–based deployments, including Kafka Connect and MirrorMaker 2 instances that target the same Kafka cluster.
18+
19+
Kafka Connect identifies a worker cluster by its group ID and by the names of the internal topics it uses.
20+
If multiple Kafka Connect clusters use the same values, they form a single Kafka Connect cluster and share internal state.
21+
22+
When deploying multiple Kafka Connect clusters against the same target Kafka cluster, configure unique values for the following Kafka Connect worker properties in each deployment:
23+
24+
* `group.id`
25+
* `config.storage.topic`
26+
* `offset.storage.topic`
27+
* `status.storage.topic`
28+
29+
These settings are configured in the `.spec.target` section.
1330

1431
[source,yaml,subs="attributes+"]
1532
----
@@ -33,13 +50,11 @@ spec:
3350
<3> Kafka topic that stores connector and task status configurations.
3451
<4> Kafka topic that stores connector and task status updates.
3552

36-
NOTE: Values for the three topics must be the same for all instances with the same `group.id`.
37-
38-
The `target` cluster specifies the target Kafka cluster used by Kafka Connect for its internal topics.
39-
As a result, modifications to the `target`, group ID, and internal topic naming configuration are specific to the target Kafka cluster.
40-
You don't need to make changes if two MirrorMaker 2 instances are using the same source Kafka cluster or in an active-active mode where each MirrorMaker 2 instance has a different target cluster.
53+
If multiple Kafka Connect clusters are deployed with the same worker group ID and internal topic names, they form a single Kafka Connect cluster and share internal state.
54+
This can result in unexpected behavior and replication errors.
4155

42-
However, if multiple MirrorMaker 2 instances share the same `target`, each instance connecting to the same target Kafka cluster is deployed with the same values.
43-
In practice, this means all instances form a cluster and use the same internal topics.
56+
You do not need to change these settings when:
4457

45-
Multiple instances attempting to use the same internal topics will cause unexpected errors, so you must change the values of these properties for each instance.
58+
* Running multiple MirrorMaker 2 connectors in a single Kafka Connect cluster
59+
* Replicating data from the same source Kafka cluster to different target Kafka clusters by using separate Kafka Connect clusters
60+
* Deploying active-active replication, where each Kafka Connect cluster targets a different Kafka cluster

documentation/modules/configuring/con-mm2-recovery.adoc

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ To support this, the Kafka cluster should also be monitored for health and perfo
1313

1414
If failover occurs, which can be automated, operations switch from the active cluster to the passive cluster when the active cluster becomes unavailable.
1515
The original active cluster is typically considered permanently lost.
16-
The passive cluster is promoted to active status, taking over as the source for all application traffic.
16+
The passive cluster is promoted to active status, taking over as the source and target for all application traffic.
1717
In this state, MirrorMaker 2 no longer replicates data from the original active cluster while it remains unavailable.
1818

1919
Failback, or restoring operations to the original active cluster, requires careful planning.
@@ -25,20 +25,20 @@ For a simpler and more reliable failback process, rebuild the original active cl
2525

2626
Follow these best practices for disaster recovery in the event of failure of the active cluster in an active/passive configuration:
2727

28-
. Promote the passive recovery cluster to an active role. +
28+
. Promote the passive recovery cluster to an active role.
2929
Designate the passive cluster as the active cluster for all client connections.
3030
This minimizes downtime and ensures operations can continue.
31-
. Redirect applications to the new active recovery cluster. +
31+
. Redirect applications to the new active recovery cluster.
3232
MirrorMaker 2 synchronizes committed offsets to passive clusters, allowing consumer applications to resume from the last transferred offset when switching to the recovery cluster.
3333
However, because of the time lag in offset synchronization, switching consumers may result in some message duplication.
3434
To minimize duplication, switch all members of a consumer group together as soon as possible.
3535
Keeping the group intact minimizes the chance of a consumer processing duplicate messages.
36-
. Remove the MirrorMaker 2 configuration for replication from the original active cluster to the passive cluster. +
36+
. Remove the MirrorMaker 2 configuration for replication from the original active cluster to the passive cluster.
3737
After failover, the original configuration is no longer needed and should be removed to avoid conflicts.
3838
. Re-create the failed cluster in a clean state, adhering to the original configuration.
39-
. Deploy a new MirrorMaker 2 instance to replicate data from the active recovery cluster to the rebuilt cluster. +
39+
. Deploy a new MirrorMaker 2 instance to replicate data from the active recovery cluster to the rebuilt cluster.
4040
Treat the rebuilt cluster as the passive cluster during this replication process.
41-
To prevent automatic renaming of topics, configure MirrorMaker 2 to use the `IdentityReplicationPolicy` by setting the `replication.policy.class` property in the MirrorMaker 2 configuration.
41+
To prevent automatic renaming of topics, configure MirrorMaker 2 to use the `IdentityReplicationPolicy` by setting the `replication.policy.class` property in the connector configuration.
4242
With this configuration applied, topics retain their original names in the target cluster.
4343
. Ensure the rebuilt cluster mirrors all data from the now-active recovery cluster.
4444
. (Optional) Promote the rebuilt cluster back to active status by redirecting applications to the rebuilt cluster, after ensuring it is fully synchronized with the active cluster.

0 commit comments

Comments
 (0)