Skip to content

Commit b2aed5d

Browse files
nastena1606nastena1606
committed
K8SPS-508 Documented the cross-site replication
 new file: docs/clusterset-cr.md new file: docs/cr-statuses.md modified: docs/features.md modified: docs/operator.md new file: docs/replication-setup.md new file: docs/replication.md modified: docs/users.md modified: mkdocs-base.yml
1 parent d81ed77 commit b2aed5d

11 files changed

Lines changed: 1266 additions & 2 deletions

‎docs/assets/diagrams/cross-site-replication-clusterset.mmd‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
%% Cross-site replication / InnoDB ClusterSet architecture
2+
%% Source for the diagram in docs/replication.md
3+
%% Percona docs palette: fill #D4EDFB, stroke #789CD6
4+
5+
flowchart TB
6+
classDef clusterBox fill:#D4EDFB,stroke:#789CD6,stroke-width:2px,color:#1a1a1a
7+
classDef innerBox fill:#ffffff,stroke:#789CD6,stroke-width:1px,color:#1a1a1a
8+
classDef mgmtBox fill:#E8F4FC,stroke:#326690,stroke-width:2px,color:#1a1a1a
9+
classDef appBox fill:#ffffff,stroke:#326690,stroke-width:2px,color:#1a1a1a
10+
classDef dashedBox fill:#ffffff,stroke:#789CD6,stroke-width:1px,stroke-dasharray:5 5,color:#1a1a1a
11+
12+
APP["Application"]:::appBox
13+
14+
subgraph CS["ClusterSet"]
15+
direction TB
16+
17+
subgraph MGMT["ClusterSet management"]
18+
direction LR
19+
CR["PerconaServerMySQLClusterSet CR<br/>(ps-clusterset)"]:::mgmtBox
20+
CTRL["ClusterSet controller"]:::mgmtBox
21+
RUNNER["mysqlshell-runner Pod"]:::mgmtBox
22+
CR --- CTRL
23+
CTRL --- RUNNER
24+
end
25+
26+
subgraph PRIMARY["Kubernetes cluster — primary"]
27+
direction TB
28+
OP1["PS Operator"]:::innerBox
29+
subgraph GR1["Group Replication"]
30+
direction LR
31+
M1["MySQL Pod<br/>+ PVC"]:::innerBox
32+
M2["MySQL Pod<br/>+ PVC"]:::innerBox
33+
M3["MySQL Pod<br/>+ PVC"]:::innerBox
34+
end
35+
HA1["HAProxy"]:::innerBox
36+
OP1 --> GR1
37+
GR1 --> HA1
38+
end
39+
40+
subgraph REPLICA["Kubernetes cluster — replica"]
41+
direction TB
42+
OP2["PS Operator"]:::innerBox
43+
subgraph GR2["Group Replication"]
44+
direction LR
45+
R1["MySQL Pod<br/>+ PVC"]:::innerBox
46+
R2["MySQL Pod<br/>+ PVC"]:::innerBox
47+
R3["MySQL Pod<br/>+ PVC"]:::innerBox
48+
end
49+
HA2["HAProxy"]:::dashedBox
50+
OP2 --> GR2
51+
GR2 --> HA2
52+
end
53+
54+
subgraph ONPREM["On-premises MySQL"]
55+
direction TB
56+
subgraph NODES["MySQL nodes"]
57+
direction LR
58+
N1["MySQL node"]:::innerBox
59+
N2["MySQL node"]:::innerBox
60+
N3["MySQL node"]:::innerBox
61+
end
62+
HA3["HAProxy"]:::innerBox
63+
NODES --> HA3
64+
end
65+
66+
RUNNER -->|"mysqlsh AdminAPI"| PRIMARY
67+
RUNNER -->|"mysqlsh AdminAPI"| REPLICA
68+
RUNNER -->|"mysqlsh AdminAPI"| ONPREM
69+
70+
PRIMARY ==>|"Async replication"| REPLICA
71+
PRIMARY ==>|"Async replication"| ONPREM
72+
OP1 -.->|"Async replication"| OP2
73+
end
74+
75+
APP -->|"Writes and reads"| HA1
76+
APP -->|"Reads"| HA2
77+
APP -->|"Reads"| HA3
78+
79+
class CS clusterBox
80+
class PRIMARY,REPLICA,ONPREM clusterBox
81+
class MGMT clusterBox

‎docs/assets/diagrams/cross-site-replication-clusterset.py‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
#!/usr/bin/env python3
2+
"""Generate the cross-site replication ClusterSet architecture diagram.
3+
4+
Requires: pip install diagrams
5+
Run from repo root:
6+
python docs/assets/diagrams/cross-site-replication-clusterset.py
7+
8+
Output: docs/assets/images/cross-site-replication-clusterset-diagrams.png
9+
"""
10+
11+
from pathlib import Path
12+
13+
from diagrams import Cluster, Diagram, Edge
14+
from diagrams.k8s.compute import Pod
15+
from diagrams.k8s.others import CRD
16+
from diagrams.onprem.client import Client
17+
from diagrams.onprem.compute import Server
18+
from diagrams.onprem.database import MySQL
19+
from diagrams.onprem.network import HAProxy
20+
21+
OUTPUT = Path(__file__).resolve().parents[1] / "images" / "cross-site-replication-clusterset-diagrams.png"
22+
23+
graph_attr = {
24+
"fontsize": "13",
25+
"bgcolor": "white",
26+
"pad": "0.4",
27+
"splines": "spline",
28+
"nodesep": "0.8",
29+
"ranksep": "1.2",
30+
}
31+
32+
with Diagram(
33+
"Cross-site replication ClusterSet",
34+
filename=str(OUTPUT.with_suffix("")),
35+
outformat="png",
36+
show=False,
37+
direction="TB",
38+
graph_attr=graph_attr,
39+
):
40+
app = Client("Application")
41+
42+
43+
grouprepl = Edge(label="Group Replication", style="dashed", color="gray")
44+
asyncrepl = Edge(label="async replication", style="solid", color="green")
45+
46+
with Cluster("ClusterSet", direction="LR"):
47+
with Cluster("ClusterSet management", direction="LR"):
48+
clusterset_cr = CRD("PerconaServerMySQL\nClusterSet CR")
49+
controller = Pod("ClusterSet\ncontroller")
50+
runner = Pod("mysqlshell-\nrunner Pod")
51+
clusterset_cr << controller >> runner
52+
53+
54+
with Cluster(""):
55+
with Cluster("Kubernetes cluster — primary"):
56+
op_primary = Pod("PS Operator")
57+
with Cluster("MySQL Cluster primary"):
58+
mysql_primary1 = MySQL("Primary")
59+
mysql_primary2 = MySQL("Secondary")
60+
mysql_primary3 = MySQL("MySQL Pod")
61+
mysql_primary1 - mysql_primary2 - mysql_primary3 - mysql_primary1 << grouprepl
62+
haproxy_primary = HAProxy("HAProxy")
63+
op_primary >> mysql_primary1 >> haproxy_primary
64+
65+
with Cluster("Kubernetes cluster — replica"):
66+
op_replica = Pod("PS Operator")
67+
with Cluster("MySQL Cluster primary"):
68+
mysql_replica1 = MySQL("Primary")
69+
mysql_replica2 = MySQL("Secondary")
70+
mysql_replica3 = MySQL("Secondary")
71+
mysql_replica1 - mysql_replica2 - mysql_replica3 - mysql_replica1 << grouprepl
72+
haproxy_replica = HAProxy("HAProxy")
73+
op_replica >> mysql_replica1 >> haproxy_replica
74+
75+
with Cluster("On-premises MySQL"):
76+
mysql_nodes = Server("MySQL nodes")
77+
haproxy_onprem = HAProxy("HAProxy")
78+
mysql_nodes >> haproxy_onprem
79+
80+
runner >> Edge(label="mysqlsh AdminAPI") >> [op_primary, op_replica, mysql_nodes]
81+
mysql_primary1 >> Edge(label="Async replication", color="darkgreen") >> mysql_replica1
82+
mysql_primary1 >> Edge(label="Async replication", color="darkgreen") >> mysql_nodes
83+
op_primary >> Edge(label="Async replication", style="dashed", color="gray") >> op_replica
84+
85+
app >> Edge(label="Writes and reads") >> mysql_primary1
86+
app >> Edge(label="Reads") >> mysql_replica1
87+
app >> Edge(label="Reads") >> mysql_nodes
88+
89+
print(f"Wrote {OUTPUT}")

‎docs/assets/images/innodb_clusterset.png‎

187 KB
Loading

‎docs/clusterset-cr.md‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# ClusterSet Resource options
2+
3+
The `PerconaServerMySQLClusterSet` Custom Resource defines cross-site replication between Group Replication clusters as a MySQL InnoDB ClusterSet. Use the short name `ps-clusterset`.
4+
5+
This document describes all available options. For status fields and conditions, see [Custom resource statuses](cr-statuses.md).
6+
7+
## `apiVersion`
8+
9+
Specifies the API version of the Custom Resource.
10+
11+
| Value type | Example |
12+
| ---------- | ------- |
13+
| :material-code-string: string | `ps.percona.com/v1` |
14+
15+
## `kind`
16+
17+
Defines the type of resource: `PerconaServerMySQLClusterSet`.
18+
19+
## `metadata`
20+
21+
The metadata section identifies the ClusterSet object. It includes the following keys:
22+
23+
* `name` — Name of the ClusterSet resource.
24+
* `namespace` — Namespace where the ClusterSet Custom Resource and the `mysqlshell-runner` Pod run.
25+
* `finalizers` — ensure safe deletion of resources in Kubernetes under certain conditions. This subsection includes the following finalizers:
26+
27+
* `percona.com/clusterset-dissolve` — Runs `.dissolve()` on the InnoDB ClusterSet before the Custom Resource is deleted. Underlying clusters continue as standalone InnoDB Clusters.
28+
29+
## `spec`
30+
31+
This section contains the ClusterSet configuration.
32+
33+
### `spec.primaryCluster`
34+
35+
Name of the InnoDB cluster that should serve writes. Must match exactly one `spec.clusters[].innodbClusterName`. Must contain only alphanumeric characters (max 63).
36+
37+
Editing this field triggers a **planned switchover** when the current primary is reachable. For a **forced failover** when the current primary is unreachable, you must also set the [`spec.unsafeFlags.forcedFailover`](#specunsafeflagsforcedfailover) to `true`.
38+
39+
| Value type | Example |
40+
| ---------- | ------- |
41+
| :material-code-string: string | `pscluster1` |
42+
43+
### The `spec.unsafeFlags` subsection
44+
45+
Groups opt-in flags for destructive ClusterSet operations. Both default to `false`.
46+
47+
#### `spec.unsafeFlags.forcedFailover`
48+
49+
When `true`, permits `forcePrimaryCluster()` if you change `spec.primaryCluster` while the current primary is unreachable.
50+
51+
| Value type | Example |
52+
| ---------- | ------- |
53+
| :material-toggle-switch-outline: boolean | `false` |
54+
55+
!!! warning
56+
57+
Forced failover can promote a replica while the old primary may still accept writes, causing split-brain and lost transactions. Set this only when you are certain the old primary cannot recover.
58+
59+
#### `spec.unsafeFlags.forcedClusterRemoval`
60+
61+
When `true`, permits `removeCluster(..., {force: true})` for an unreachable replica removed from `spec.clusters[]`.
62+
63+
| Value type | Example |
64+
| ---------- | ------- |
65+
| :material-toggle-switch-outline: boolean | `false` |
66+
67+
!!! warning
68+
69+
Forced removal abandons unreplicated transactions on the removed cluster. The cluster may require manual cleanup or a full rebuild.
70+
71+
### `spec.sslMode`
72+
73+
SSL mode for ClusterSet async replication channels between primary and replica clusters.
74+
75+
| Value | Meaning |
76+
| ----- | ------- |
77+
| `AUTO` | TLS is enabled if the instance supports it; otherwise disabled. **Default.** |
78+
| `DISABLED` | TLS is disabled for replication channels. |
79+
| `REQUIRED` | TLS is required for replication channels. |
80+
| `VERIFY_CA` | Like `REQUIRED`, plus verify the peer certificate against configured CA certificates. Primary and replica certificates must be signed by the same CA. |
81+
| `VERIFY_IDENTITY` | Like `VERIFY_CA`, plus verify the peer certificate matches the connection host (SAN must match `spec.clusters[].endpoints[].host`). |
82+
83+
| Value type | Example |
84+
| ---------- | ------- |
85+
| :material-code-string: string | `AUTO` |
86+
87+
### `spec.credentialsSecret.name`
88+
89+
The name of a Secret in the same namespace that holds the `clusterset` user password.
90+
91+
| Value type | Example |
92+
| ---------- | ------- |
93+
| :material-code-string: string | `ps-cluster1-secrets` |
94+
95+
### `spec.credentialsSecret.key`
96+
97+
The field in the Secret that holds the `clusterset` user password value. This password must match on every participating cluster. Defaults to `clusterset`.
98+
99+
| Value type | Example |
100+
| ---------- | ------- |
101+
| :material-code-string: string | `clusterset` |
102+
103+
### The `spec.clusters` subsection
104+
105+
List of InnoDB clusters that participate in the ClusterSet. Minimum 1 entry (primary only at bootstrap); maximum 10. Each cluster name must be unique.
106+
107+
#### `spec.clusters[].innodbClusterName`
108+
109+
Logical name of the InnoDB cluster within the ClusterSet. Used in `spec.primaryCluster`, status, and MySQL Shell AdminAPI calls. The name must match the value from `status.innodbClusterName` of a corresponding `PerconaServerMySQL` Custom Resource.
110+
111+
Alphanumeric only, max 63 characters. Immutable in practice — rename by remove and re-add.
112+
113+
| Value type | Example |
114+
| ---------- | ------- |
115+
| :material-code-string: string | `pscluster1` |
116+
117+
#### `spec.clusters[].endpoints`
118+
119+
List of `host:port` pairs the controller uses to reach MySQL members of this cluster. Minimum 1, maximum 9 per cluster. The controller uses the first reachable endpoint. Multiple entries improve resilience to single-member failures.
120+
121+
Each `host` must be unique across **all** clusters in the ClusterSet. It can be the IP address or a DNS name reachable from the `mysqlshell-runner` Pod.
122+
123+
The `port` is MySQL port. Defaults to `3306`.
124+
125+
126+
| Value type | Example |
127+
| ---------- | ------- |
128+
| :material-text-long: subdoc | <pre>- host: ps-cluster1-mysql-primary.default.svc.cluster.local<br> port: 3306</pre> |
129+
130+
### `spec.createReplicaClusterOptions.recoveryMethod`
131+
132+
Preferred method for seeding the first member of a new replica cluster when it joins the ClusterSet.
133+
134+
Supported values:
135+
136+
* `clone` — Physical snapshot via MySQL CLONE plugin. **Default.** Best for small datasets and low-latency networks.
137+
* `incremental` — Apply GTID delta only. Use after restoring a backup from the primary into the replica. Best for large datasets or WAN links.
138+
139+
When you omit this field, the Operator uses `clone`.
140+
141+
| Value type | Example |
142+
| ---------- | ------- |
143+
| :material-code-string: string | `clone` |
144+
145+
146+
### `spec.mysqlshellRunner.image`
147+
148+
Container image for the `mysqlshell-runner` Pod and ClusterSet Jobs. Must contain `mysqlsh` on `PATH`. The mysqlsh major version must match the MySQL endpoints (8.0 endpoints → 8.0 image, 8.4 endpoints → 8.4 image).
149+
150+
| Value type | Example |
151+
| ---------- | ------- |
152+
| :material-code-string: string | `percona/percona-server:{{ ps84recommended }}` |

0 commit comments

Comments
 (0)