Skip to content

Commit 9274494

Browse files
mayankshah1607mayankshah1607
authored
[blogpost] Cross-site replication with PS operator (#1210)
* add blog post for ps-operator cross-site DR Signed-off-by: Mayank Shah <mayank.shah@percona.com> * add yaml reference Signed-off-by: Mayank Shah <mayank.shah@percona.com> * remove newlines Signed-off-by: Mayank Shah <mayank.shah@percona.com> * markdown improvements Signed-off-by: Mayank Shah <mayank.shah@percona.com> * markdown improvements Signed-off-by: Mayank Shah <mayank.shah@percona.com> * Update 2026-06-25-ps-operator-cross-site-dr.md * title update Signed-off-by: Mayank Shah <mayank.shah@percona.com> * typos and update docs link Signed-off-by: Mayank Shah <mayank.shah@percona.com> * img update Signed-off-by: Mayank Shah <mayank.shah@percona.com> * cleanup Signed-off-by: Mayank Shah <mayank.shah@percona.com> * finalize Signed-off-by: Mayank Shah <mayank.shah@percona.com> * added header Signed-off-by: Mayank Shah <mayank.shah@percona.com> * update title Signed-off-by: Mayank Shah <mayank.shah@percona.com> * updated cover image Signed-off-by: Mayank Shah <mayank.shah@percona.com> --------- Signed-off-by: Mayank Shah <mayank.shah@percona.com>
1 parent 5fe2c84 commit 9274494

3 files changed

Lines changed: 270 additions & 0 deletions

File tree

‎assets/blog/2026/06/mayank-cross-cluster-ps-cover.jpg‎

255 KB
Loading

‎assets/blog/2026/06/ps-clusterset-architecture.svg‎

Lines changed: 4 additions & 0 deletions
Loading

‎content/blog/2026/2026-06-25-ps-operator-cross-site-dr.md‎

Lines changed: 266 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,266 @@
1+
---
2+
title: "Cross-site Disaster Recovery with Percona Operator for MySQL"
3+
date: "2026-07-06T10:00:00+00:00"
4+
tags: ["Percona", "Kubernetes", "Cloud", "Community", "Open Source", "MySQL"]
5+
categories: ['MySQL', 'Cloud']
6+
authors:
7+
- mayank_shah
8+
images:
9+
- blog/2026/06/mayank-cross-cluster-ps-cover.jpg
10+
---
11+
12+
A MySQL InnoDB Cluster provides high availability for a single database cluster using Group Replication. This works well for node failures inside the cluster, but disaster recovery usually requires another cluster in a separate location: another Kubernetes cluster, region, data center, or cloud.
13+
14+
This replica cluster needs to stay in sync with the primary, remain protected from accidental writes, and be ready to take over when you need to move traffic, either as a planned operation or during an outage.
15+
16+
[InnoDB ClusterSet](https://dev.mysql.com/doc/mysql-shell/8.0/en/innodb-clusterset.html) addresses this by linking multiple MySQL clusters into a single disaster-recovery topology. One cluster handles writes, while the others stay synchronized as read-only replicas.
17+
18+
Starting from v1.2.0, the Percona Operator for MySQL adds a new custom resource, `PerconaServerMySQLClusterSet`, which allows managing InnoDB ClusterSets. Creating the ClusterSet, adding replicas, switching the primary, and performing a forced failover are all handled declaratively by updating the Kubernetes spec and letting the operator reconcile the desired state.
19+
20+
This post explains how ClusterSet works, how to set it up with the Percona Operator, and how planned switchovers and emergency failovers work in practice.
21+
22+
## Understanding InnoDB ClusterSet
23+
24+
Any disaster recovery design usually comes down to two important numbers:
25+
26+
- **Recovery Point Objective**, or RPO, is how much data you can afford to lose. For example, an RPO of five seconds means the business can tolerate losing up to five seconds of writes.
27+
- **Recovery Time Objective**, or RTO, is how long the system can be unavailable before service must be restored.
28+
29+
The way you design and operate a ClusterSet directly affects both. To understand why, it helps to first look at the architecture.
30+
31+
An InnoDB ClusterSet is built from two or more InnoDB Clusters. Each InnoDB Cluster is a Group Replication group. In other words, it is the same kind of highly available MySQL cluster that the [Percona Operator for MySQL](https://docs.percona.com/percona-operator-for-mysql/latest/index.html) can already deploy and manage.
32+
33+
A ClusterSet adds another layer on top of those clusters. One cluster is the primary cluster and accepts writes, while the others are replica clusters and remain read-only. The primary sends its changes to each replica using asynchronous replication over a dedicated replication channel.
34+
35+
This gives us two layers of replication, each solving a different problem.
36+
37+
Inside each cluster, Group Replication protects against the loss of individual MySQL nodes. Members are expected to be closer together, usually within the same region or availability zone group. Writes are coordinated by the group, which helps keep the local cluster consistent and highly available.
38+
39+
Between clusters, asynchronous replication protects against the loss of an entire site. Replica clusters can be located in another region, another Kubernetes cluster, or another cloud provider. Because this replication is asynchronous, long-distance network latency does not slow down writes on the primary cluster.
40+
41+
But the tradeoff here is that a replica cluster may be slightly behind the primary. The amount of lag depends on write volume, network latency, and the health of the replication channel. If the primary site is lost, any writes that had not yet reached the replica are lost. That lag is the practical data-loss window during an emergency failover. Any transactions that had not replicated before failover could be lost.
42+
43+
Before building a ClusterSet with the operator, there are a few important requirements to keep in mind:
44+
45+
- Every cluster in the ClusterSet must use the Group Replication topology. The operator also supports asynchronous replication with Orchestrator for standalone clusters, but that topology cannot be part of an InnoDB ClusterSet.
46+
- You need MySQL 8.0.27 or later
47+
- Clusters are linked by network address, not by Kubernetes references. A replica cluster only needs to be reachable and managed by an operator. It does not need to live in the same Kubernetes cluster as the primary.
48+
49+
With the model in place, let’s build a simple cross-site disaster recovery setup.
50+
51+
52+
53+
![Figure 1. InnoDB ClusterSet overview](../../../assets/blog/2026/06/ps-clusterset-architecture.svg)
54+
55+
## Setting up ClusterSet
56+
57+
We’ll create the simplest useful ClusterSet: two Group Replication clusters named `dc1` and `dc2`.
58+
59+
In this example:
60+
`dc1` is the primary cluster.
61+
`dc2` is the read-only replica cluster.
62+
63+
In a real deployment, these would usually run in separate Kubernetes clusters, regions, or cloud environments. The steps are mostly the same. The main requirement is that the endpoints listed in the ClusterSet spec must be routable between sites.
64+
65+
### Creating a primary cluster
66+
67+
The primary cluster `dc1` is a regular Group Replication cluster. There is nothing ClusterSet-specific about it at this stage.
68+
69+
```yaml
70+
apiVersion: ps.percona.com/v1
71+
kind: PerconaServerMySQL
72+
metadata:
73+
name: dc1
74+
spec:
75+
mysql:
76+
clusterType: group-replication
77+
# ... the rest of a normal cluster spec
78+
```
79+
80+
You can find a complete YAML [here](https://github.com/percona/percona-server-mysql-operator/blob/main/deploy/cr.yaml). Apply it and wait for it to come up the way you normally would, just as you would for any normal Percona Operator-managed MySQL cluster.
81+
82+
### Creating the replica cluster
83+
84+
The replica cluster `dc2` is also a Group Replication cluster, but with one important difference:
85+
86+
```yaml
87+
apiVersion: ps.percona.com/v1
88+
kind: PerconaServerMySQL
89+
metadata:
90+
name: dc2
91+
spec:
92+
mysql:
93+
clusterType: group-replication
94+
bootstrap:
95+
mode: manual # <- set this!
96+
```
97+
98+
Normally, when the operator creates a Group Replication cluster, the first MySQL pod bootstraps the group as soon as it starts. Subsequent pods then join that group.
99+
100+
For a ClusterSet replica, that is not what we want. We do not want `dc2` to form an independent empty cluster. Instead, we want it to receive data from the primary cluster and then join the ClusterSet as a replica.
101+
102+
With `bootstrap.mode: manual`, the first pod starts but does not bootstrap its own Group Replication group. It waits until the ClusterSet process adopts it, clones data from the primary, and then forms the replica cluster. During this stage, the first `dc2` pod may remain in a `NotReady` state until it is a part of the ClusterSet.
103+
104+
### Sharing cluster credentials
105+
106+
The operator automatically creates a `clusterset` MySQL user in every cluster and stores its password in the cluster secret.
107+
108+
The operator uses this user to orchestrate ClusterSet operations, so the password must be the same across all clusters in the ClusterSet. When your clusters are deployed separately, copy the `clusterset` value from the primary cluster secret into the replica cluster secret before linking them.
109+
110+
For example, if `dc1` is the primary, copy the `clusterset` password from the `dc1` secret into the corresponding secret for `dc2`.
111+
112+
### Linking the clusters
113+
114+
Once both clusters are applied, create a `PerconaServerMySQLClusterSet` custom resource.
115+
116+
```yaml
117+
apiVersion: ps.percona.com/v1
118+
kind: PerconaServerMySQLClusterSet
119+
metadata:
120+
name: my-cluster-set
121+
finalizers:
122+
- percona.com/clusterset-dissolve
123+
spec:
124+
primaryCluster: dc1
125+
credentialsSecret:
126+
name: dc1-secrets
127+
key: clusterset
128+
sslMode: AUTO
129+
createReplicaClusterOptions:
130+
recoveryMethod: clone
131+
clusters:
132+
- innodbClusterName: dc1
133+
endpoints:
134+
- host: dc1-mysql-primary.default.svc.cluster.local
135+
- innodbClusterName: dc2
136+
endpoints:
137+
- host: dc2-mysql-0.dc2-mysql.default.svc.cluster.local
138+
mysqlshellRunner:
139+
image: perconalab/percona-server-mysql-operator:main-psmysql8.4
140+
```
141+
142+
The most important fields are:
143+
144+
- `primaryCluster` defines which cluster currently accepts writes. The value must match one of the entries under clusters.
145+
- `clusters` lists every member of the ClusterSet and the endpoint the operator should use to reach it. These endpoints are plain network addresses, which is what allows members to run in different Kubernetes clusters or regions.
146+
- `credentialsSecret` points to the secret that contains the clusterset user password.
147+
- `recoveryMethod: clone` tells the replica cluster to take a full copy of the primary data when it joins the ClusterSet. The alternative is an incremental recovery method, which uses existing binary logs instead of cloning the full dataset.
148+
- `mysqlshellRunner` defines the helper pod image used by the operator to run MySQL Shell operations.
149+
150+
After you apply this resource, the operator starts a MySQL Shell runner pod and creates the ClusterSet on `dc1`. It then joins `dc2`, which clones the data, starts replication, and brings up the remaining pods in the replica cluster.
151+
152+
At this point, `dc1` serves reads and writes, while `dc2` acts as a live read-only copy.
153+
154+
> **Seeding large replica clusters**
155+
>
156+
> In this example, the replica cluster is created with `recoveryMethod: clone`, so MySQL Shell provisions the first replica member by copying a physical snapshot from an existing ClusterSet member. That is convenient for medium/small datasets, but it can be fragile across WAN links or very large databases.
157+
>
158+
> A full clone can take hours, consume significant bandwidth, add load to the donor, run into network interruptions, and become expensive to retry if the operation fails partway through. It can also not be the best fit when the primary is busy or when cross-region egress cost is a concern.
159+
>
160+
> The operator makes it possible to seed the replica cluster from an existing backup of the primary cluster instead. Create a `PerconaServerMySQLBackup` on the primary, restore that backup into the replica cluster with `PerconaServerMySQLRestore`, and then add the replica to the ClusterSet using `recoveryMethod: incremental`. You can find the exact restore procedure in the [documentation](https://docs.percona.com/percona-operator-for-mysql/latest/backups-restore-to-new-cluster.html).
161+
>
162+
> At that point, the replica already has the primary’s data and GTID history, so ClusterSet only needs to catch it up from the primary’s binary logs instead of transferring the full dataset again.
163+
164+
### Verifying it worked
165+
166+
The simplest way to confirm that the ClusterSet is working is to write data to the primary cluster and read it from the replica.
167+
168+
For example:
169+
- Connect to `dc1`.
170+
- Create a test table or insert a row.
171+
- Connect to `dc2`.
172+
- Confirm that the same data appears there.
173+
174+
If the row appears on `dc2`, the asynchronous replication channel is running and the replica cluster is receiving changes from the primary.
175+
176+
### Planned Switchover
177+
178+
A planned switchover is used when both clusters are healthy and you intentionally want to move writes from one site to another. This is useful for regional maintenance, Kubernetes cluster upgrades, cloud migrations, or controlled DR testing.
179+
180+
To move the primary role from `dc1` to `dc2`, update the primaryCluster field:
181+
182+
```shell
183+
kubectl patch ps-clusterset my-cluster-set --type=merge \
184+
-p '{"spec":{"primaryCluster":"dc2"}}'
185+
```
186+
187+
The operator notices that the desired primary cluster no longer matches the current primary. It then uses MySQL Shell to perform a clean switchover.
188+
189+
Because both clusters are available, the operator can make sure the replica has caught up before changing roles. After the switchover completes, `dc2` becomes the writable primary and `dc1` becomes a read-only replica.
190+
191+
### Emergency Failover
192+
193+
An emergency failover can be used when the primary cluster is unreachable and a clean handover is no longer possible.
194+
195+
This is the disaster recovery case: the Kubernetes cluster, region, or network path to the primary may be down, and you need to promote a surviving replica so the application can resume writes.
196+
197+
To fail over to `dc2`, update primaryCluster and explicitly set the forced failover flag:
198+
199+
```shell
200+
kubectl patch ps-clusterset my-cluster-set --type=merge \
201+
-p '{"spec":{"primaryCluster":"dc2","unsafeFlags":{"forcedFailover":true}}}'
202+
```
203+
204+
The operator only follows this path when it can confirm that the current primary cluster is unreachable. It then promotes `dc2`, allowing it to accept writes.
205+
206+
The explicit flag is important because failover can cause data loss. Replication between clusters is asynchronous, so any writes that reached the old primary but had not yet replicated to `dc2` are not present on the new primary. Once `dc2` is promoted, those missing writes become unrecoverable through normal ClusterSet recovery.
207+
208+
The risk of data loss is why the field is named `unsafeFlags.forcedFailover`.
209+
210+
Another important point is that when the old primary comes back, it does not automatically resume as primary. After a forced failover, the recovered cluster must be explicitly reintroduced into the ClusterSet as a replica.
211+
212+
### Adding and removing clusters
213+
214+
Adding or removing clusters follows the same declarative pattern: update the custom resource spec and let the operator reconcile the difference.
215+
216+
To add another replica cluster, add a new entry under clusters:
217+
218+
```yaml
219+
apiVersion: ps.percona.com/v1
220+
kind: PerconaServerMySQLClusterSet
221+
metadata:
222+
name: my-cluster-set
223+
spec:
224+
# .. existing spec
225+
clusters:
226+
# .. existing clusters
227+
- innodbClusterName: dc3
228+
endpoints:
229+
- host: dc3-mysql-primary.default.svc.cluster.local
230+
```
231+
232+
The operator joins the new cluster in the same way it joined `dc2`: it clones data from the primary, configures replication, and brings the cluster into the ClusterSet as a read-only replica.
233+
234+
To remove a cluster, delete its entry from the clusters list. You can update your manifest and reapply it, or use a JSON patch:
235+
236+
```shell
237+
kubectl patch ps-clusterset my-cluster-set --type=json \
238+
-p '[{"op":"remove","path":"/spec/clusters/1"}]'
239+
```
240+
241+
If the cluster is healthy, the operator detaches it cleanly and it becomes a normal standalone cluster again.
242+
243+
If the cluster being removed is unreachable, you can force its removal:
244+
245+
```shell
246+
kubectl patch ps-clusterset my-cluster-set --type=json -p '[
247+
{"op":"remove","path":"/spec/clusters/1"},
248+
{"op":"add","path":"/spec/unsafeFlags/forcedClusterRemoval","value":true}
249+
]'
250+
```
251+
252+
Like forced failover, forced removal is gated behind an unsafe flag because the operator should not make this decision silently. Removing an unreachable cluster from a ClusterSet is an operational decision with consequences, and it should be made explicitly.
253+
254+
### Wrapping up
255+
256+
The Percona Operator for MySQL allows extending Group Replication beyond a single site by managing InnoDB ClusterSet through a custom resource `PerconaServerMySQLClusterSet`. A primary cluster handles writes, replica clusters stay synchronized, and the operator manages switchovers, failovers, and membership changes declaratively.
257+
258+
For planned maintenance, switchover moves the primary role safely with no data loss. For outages, forced failover promotes a surviving replica, with the expected risk of losing any writes that had not yet replicated. That replication lag is the practical RPO, so it should be monitored and tested as part of the DR plan.
259+
260+
With the Percona Operator for MySQL, disaster recovery becomes repeatable, Kubernetes-native, and easier to operate across regions or clusters.
261+
262+
### Further reading
263+
- [InnoDB ClusterSet docs](https://dev.mysql.com/doc/mysql-shell/8.0/en/innodb-clusterset.html)
264+
- [Cross-site replication in Percona Operator for MySQL](https://docs.percona.com/percona-operator-for-mysql/latest/replication.html)
265+
266+

0 commit comments

Comments
 (0)