diff --git a/docs/arbiter.md b/docs/arbiter.md index ba55294d..44378f67 100644 --- a/docs/arbiter.md +++ b/docs/arbiter.md @@ -18,7 +18,7 @@ Besides the primary and regular secondaries in a MongoDB replica set, you can ha ### Arbiter nodes -An Arbiter node participates in the replica set elections but does not store any data. Its primary role is to act as a tiebreaker in a replica set with an even number of data-bearing nodes, ensuring that a primary can always be elected. By not storing data, Arbiter nodes require minimal resources, which can help reduce your overall costs. An does not demand a persistent volume. +An Arbiter node participates in the replica set elections but does not store any data. Its primary role is to act as a tiebreaker in a replica set with an even number of data-bearing nodes, ensuring that a primary can always be elected. By not storing data, Arbiter nodes require minimal resources, which can help reduce your overall costs. An arbiter does not demand a persistent volume. To add an Arbiter node, you can update your `deploy/cr.yaml` file by adding an `arbiter` section under `replsets` and setting the `enabled` and `size` options to your desired values. @@ -107,7 +107,7 @@ replsets: Find the description of other available options in the [replsets.nonvoting section](operator.md#replsetsnonvotingenabled) of the [Custom Resource options reference](operator.md). -Note that you can add a non-voting node in the edge location through the `externalNodes` option. Please see [cross-site replication documentation](replication.md) for details. +Note that you can add a non-voting node or an external arbiter in another location through the `externalNodes` option. See [Voting members across sites](replication.md#voting-members-across-sites) and [Deploy Primary-Secondary-Arbiter across sites](replication-multi-dc.md#deploy-primary-secondary-arbiter-across-sites) for details. ## Hidden nodes diff --git a/docs/expose.md b/docs/expose.md index 7b865f83..a42956bc 100644 --- a/docs/expose.md +++ b/docs/expose.md @@ -129,12 +129,13 @@ my-cluster-name-mongos-1 NodePort 10.38.155.250 27017:3 Starting from v1.14, the Operator configures replica set members using local fully-qualified domain names (FQDN), which are resolvable and available only from inside the Kubernetes cluster. Exposing the replica set using the options described above will not affect hostname usage in the replica set configuration. - !!! note Before v1.14, the Operator used the exposed IP addresses in the replica set configuration in the case of the exposed replica set. -It is still possible to restore the old behavior. For example, it may be useful to have the replica set configured with external IP addresses for [multi-cluster deployments](replication.md). The `clusterServiceDNSMode` field in the Custom Resource controls this Operator behavior. You can set `clusterServiceDNSMode` to one of the following values: +You can still restore the previous behavior. For example, to configure the replica set with external IP addresses for [multi-cluster deployments](replication.md). + +The `clusterServiceDNSMode` field in the Custom Resource controls which addresses the Operator writes into the replica set configuration. You can set it to one of the following values: 1. **`Internal`**: Use local FQDNs (i.e., `cluster1-rs0-0.cluster1-rs0.psmdb.svc.cluster.local`) in replica set configuration even if the replica set is exposed. **This is the default value.** @@ -142,7 +143,7 @@ It is still possible to restore the old behavior. For example, it may be useful 3. **`External`**: Use exposed IP addresses in replica set configuration if the replica set is exposed; otherwise, use local FQDN. **This copies the behavior of the Operator v1.13.** - !!! warning +!!! warning Be careful with the `clusterServiceDNSMode=External` variant. Using IP addresses instead of DNS hostnames is discouraged in MongoDB. IP addresses make reconfiguration and recovery more complicated, and are **generally problematic in scenarios where IP addresses change**. In particular, if you delete and recreate the cluster with `clusterServiceDNSMode=External` without deleting its volumes (having `percona.com/delete-psmdb-pvc` finalizer unset), your cluster will crash and there will be no straightforward way to recover it. @@ -156,7 +157,52 @@ To make a manual restart, run the `kubectl rollout restart sts Alternatively, you can simply [restart your cluster](pause.md). +### Override hostnames with external domain names + +When members must be reachable by externally resolvable DNS names such as through an Ingress, TransportServer, LoadBalancer hostname, or another cross-cluster DNS entry — use [`replsets.replsetOverrides`](operator.md#replsetsreplsetoverridesmember-namehost) instead of relying on `clusterServiceDNSMode=External` and IP addresses. + +With `replsetOverrides`, you replace the default local FQDN of each Pod with a custom host and, optionally, port in the MongoDB replica set configuration. Typical use cases include: + +* Cross-cluster or multi-data-center replica sets where sites reach each other by external domain names +* Ingress or TLS passthrough endpoints (for example `r1.example.com:443`) in front of each Pod’s Service + +**Important considerations:** + +- The Operator does **not** validate overridden hostnames. **You must ensure DNS resolution and network connectivity from every replica set member to those names**. +- Include every overridden hostname in your TLS certificates. Operator-generated certificates usually are not enough for external domain names; [generate certificates manually](tls-manual.md) when needed. +- [Expose the Pods](#connecting-from-outside-kubernetes) so the external name can route to the correct Service. +- After you set replica set name overrides, the Operator updates replica set hosts one by one (secondaries first, primary last). + +Set `replsetOverrides` under the replica set in `deploy/cr.yaml`. Each key must be the **Pod name**. The value of `host` is the external domain name. Also include the port if it is not the default `27017` port: + +```yaml +... +replsets: +- name: rs0 + size: 3 + expose: + enabled: true + type: ClusterIP + replsetOverrides: + my-cluster-name-rs0-0: + host: rs0-0.example.com:27017 + my-cluster-name-rs0-1: + host: rs0-1.example.com:27017 + my-cluster-name-rs0-2: + host: rs0-2.example.com:27017 +``` + +You can also add custom tags to members under the same override: + +```yaml +replsetOverrides: + my-cluster-name-rs0-0: + host: rs0-0.example.com:27017 + tags: + team: cloud +``` +You can use these external domain names when you [split a replica set across multiple data centers](replication-multi-dc.md). Override each local member with `replsetOverrides`, and register remote members with `externalNodes` using the same style of externally reachable hostnames. ### Application protocol support for service mesh integrations diff --git a/docs/operator.md b/docs/operator.md index 6516683c..0ca98d15 100644 --- a/docs/operator.md +++ b/docs/operator.md @@ -607,7 +607,7 @@ What to do with a Pod if it doesn't satisfy the [Kubernetes Pod Topology Spread ### `replsets.replsetOverrides.MEMBER-NAME.host` -Use if you need to [override the replica set members FQDNs with custom host names](replication-multi-dc.md). Each key (`MEMBER-NAME`) under `replsetOverrides` should be name of a Pod. The Operator won’t perform any validation for hostnames, so it's the user’s responsibility to ensure connectivity. +Use if you need to [override the replica set members FQDNs with custom host names](expose.md#override-hostnames-with-external-domain-names) (for example, external domain names). Each key (`MEMBER-NAME`) under `replsetOverrides` should be name of a Pod. The Operator won’t perform any validation for hostnames, so it's the user’s responsibility to ensure connectivity. | Value type | Example | | ----------- | ---------- | @@ -663,7 +663,7 @@ The [priority :octicons-link-external-16:](https://docs.mongodb.com/manual/refer ### `replsets.externalNodes.arbiterOnly` -Defines whether this [external replset instance](replication-main.md) acts as an [arbiter :octicons-link-external-16:](https://www.mongodb.com/docs/manual/core/replica-set-arbiter/) and makes an odd number of voting members in the multi-cluster setup. +Defines whether this [external replset instance](replication-multi-dc.md) acts as an [arbiter :octicons-link-external-16:](https://www.mongodb.com/docs/manual/core/replica-set-arbiter/) and makes an odd number of voting members in the multi-cluster setup. Available with the Operator version 1.23.0+. | Value type | Example | | ----------- | ---------- | diff --git a/docs/replication-interconnect.md b/docs/replication-interconnect.md index 9cb7af0f..a36fe919 100644 --- a/docs/replication-interconnect.md +++ b/docs/replication-interconnect.md @@ -1,11 +1,20 @@ # Interconnect sites for replication -At this step, you should let the clusters know about each other and interconnect them for replication. To do so, you need to add the Replica site's nodes as external nodes for the Main site. In the same way, you add the Main's site nodes as external ones for the Replica site. +At this step, you should let the clusters know about each other and interconnect them for replication. To do so, you need to add the Replica site's nodes as external nodes for the Main site. In the same way, you add the Main's site nodes as external ones for the Replica site. Every site has three replica set members and three config server replica set members. But you add only two of them as voting members, while the third member is added as a non-voting one. In doing so, you avoid split-brain situations and prevent the primary elections if the Replica site is down or there is a network disruption between the sites. -In this way, the `main` cluster managed by the Operator is able to reach the -the `replica` nodes. +In this way, the `main` cluster managed by the Operator is able to reach the `replica` nodes. + +!!! admonition "Using the arbiter node" + + You can also add an **external arbiter** as a voting member when you interconnect sites. For example, if you run the arbiter node in the third location. This ability requires the Operator version 1.23.0 and above. + + Set `arbiterOnly: true` with `votes: 1` and `priority: 0` on that `externalNodes` entry. + + The requirements to the site with the arbiter are the same as for the replica site. + + To learn how to deploy the site with the arbiter node, see [Deploy the Arbiter site](replication-multi-dc.md#deploy-the-arbiter-site). ## List the services @@ -45,21 +54,17 @@ kubectl get services main-cluster-rs0-2 ClusterIP 34.118.225.3 27017/TCP 42m ``` -## Add the Replica site nodes to the Main site +## Add the Replica site nodes to the Main site -1. Modify the `deploy/cr-main.yaml` file of the Main site and define the exposed nodes of the Replica site in the -`replsets.externalNodes` and `sharding.configsvrReplset.externalNodes` -subsections. For each node, specify the following: +1. Modify the `deploy/cr-main.yaml` file of the Main site and define the exposed nodes of the Replica site in the +`replsets.externalNodes` and `sharding.configsvrReplSet.externalNodes` subsections. For each node, specify the following: - set `host` to the URL of the external replset instance. When exposed, a node has its own service, recognized by the domain name `..svc.clusterset.local` - - set `port` to the port number of the external node. If not set, the default `27017` - port is used, - - set `priority` to define the [priority :octicons-link-external-16:](https://docs.mongodb.com/manual/reference/replica-configuration/#mongodb-rsconf-rsconf.members-n-.priority) - of the external node. The default priority for local members of the cluster is `2`. When you add external nodes, set the lower priority to avoid an unmanaged node being elected - as a primary. A zero `0` priority adds the node as a [non-voting member](arbiter.md#non-voting-nodes). - - set `votes` to the number of [votes :octicons-link-external-16:](https://docs.mongodb.com/manual/reference/replica-configuration/#mongodb-rsconf-rsconf.members-n-.votes) - an external node can cast in a replica set election (`0` is default and - should be used for non-voting members of the cluster). + - set `port` to the port number of the external node. If not set, the default `27017` port is used, + - set `priority` to define the [priority :octicons-link-external-16:](https://docs.mongodb.com/manual/reference/replica-configuration/#mongodb-rsconf-rsconf.members-n-.priority) of the external node. The default priority for local members of the cluster is `2`. When you add external nodes, set the lower priority to avoid an unmanaged node being elected + as a primary. A zero `0` priority adds the node as a [non-voting member](arbiter.md#non-voting-nodes). + - set `votes` to the number of [votes :octicons-link-external-16:](https://docs.mongodb.com/manual/reference/replica-configuration/#mongodb-rsconf-rsconf.members-n-.votes) an external node can cast in a replica set election (`0` is default and + should be used for non-voting members of the cluster). Here is an example: @@ -83,6 +88,9 @@ subsections. For each node, specify the following: replsets: - name: rs0 size: 3 + expose: + enabled: true + type: ClusterIP externalNodes: - host: replica-cluster-rs0-0.psmdb.svc.clusterset.local votes: 1 @@ -92,15 +100,15 @@ subsections. For each node, specify the following: priority: 1 - host: replica-cluster-rs0-2.psmdb.svc.clusterset.local votes: 0 - priority: 0 - expose: - enabled: true - type: ClusterIP + priority: 0 sharding: enabled: true configsvrReplSet: size: 3 + expose: + enabled: true + type: ClusterIP externalNodes: - host: replica-cluster-cfg-0.psmdb.svc.clusterset.local votes: 1 @@ -111,15 +119,11 @@ subsections. For each node, specify the following: - host: replica-cluster-cfg-2.psmdb.svc.clusterset.local votes: 0 priority: 0 - expose: - enabled: true - type: ClusterIP mongos: size: 3 expose: type: ClusterIP - ``` 2. Apply the changes: @@ -130,43 +134,43 @@ subsections. For each node, specify the following: ## Add Main site nodes to the Replica site -1. Modify the `deploy/cr-main.yaml` file of the Replica site and define the exposed nodes of the Main site in the -`replsets.externalNodes` and `sharding.configsvrReplset.externalNodes` -subsections. For each node, specify the following: - +1. Modify the `deploy/cr-replica.yaml` file of the Replica site and define the exposed nodes of the Main site in the +`replsets.externalNodes` and `sharding.configsvrReplSet.externalNodes` +subsections. + ```yaml apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDB metadata: name: replica-cluster spec: - ... + ... replsets: - name: rs0 size: 3 externalNodes: - host: main-cluster-rs0-0.psmdb.svc.clusterset.local + votes: 1 + priority: 1 + - host: main-cluster-rs0-1.psmdb.svc.clusterset.local + votes: 1 + priority: 1 + - host: main-cluster-rs0-2.psmdb.svc.clusterset.local + votes: 0 + priority: 0 + + sharding: + configsvrReplSet: + externalNodes: + - host: main-cluster-cfg-0.psmdb.svc.clusterset.local votes: 1 priority: 1 - - host: main-cluster-rs0-1.psmdb.svc.clusterset.local + - host: main-cluster-cfg-1.psmdb.svc.clusterset.local votes: 1 priority: 1 - - host: main-cluster-rs0-2.psmdb.svc.clusterset.local + - host: main-cluster-cfg-2.psmdb.svc.clusterset.local votes: 0 priority: 0 - - sharding: - configsvrReplSet: - externalNodes: - - host: main-cluster-cfg-0.psmdb.svc.clusterset.local - votes: 1 - priority: 1 - - host: main-cluster-cfg-1.psmdb.svc.clusterset.local - votes: 1 - priority: 1 - - host: main-cluster-cfg-2.psmdb.svc.clusterset.local - votes: 0 - priority: 0 ``` 2. Apply the configuration: diff --git a/docs/replication-multi-dc.md b/docs/replication-multi-dc.md index c2af4359..d2c86a38 100644 --- a/docs/replication-multi-dc.md +++ b/docs/replication-multi-dc.md @@ -1,99 +1,434 @@ -# Splitting replica set across multiple data centers +# Splitting a replica set across multiple data centers -Splitting the replica set of the database cluster over multiple Kubernetes clusters can be useful to get a fault-tolerant system in which all replicas are in different data centers. +Splitting a replica set over multiple Kubernetes clusters gives you a fault-tolerant system where members live in different data centers. -The Operator cannot deploy MongoDB replicas to other data centers, but this solution can be achieved with a number of Operator deployments, equal to the size of your replica set. So, you will need at least 3 Operator instances: one Operator to control the replica set via cross-site replication, and at least two Operators to bootstrap the unmanaged clusters. Each cluster will contain replica set with only one member, and the _Main_ site will manage instances from other sites as external nodes. All configuration of the replica set is done manually. +The Operator cannot place MongoDB Pods in other data centers by itself. You achieve this by deploying an Operator on each site: the Operator on the _Main_ site manages the replica set configuration through cross-site replication. Operators on other sites bootstrap the unmanaged clusters whose members you register on the Main site as `externalNodes`. You configure hostnames, TLS, and membership manually. -The solution has the following limitations to consider: +This solution has the following limitations: -* setting it up involves a number of manual operations, and the same applies to scaling such a manually configured replica, -* backups are supported on the _Main_ site only, not on the _Replica_ sites. +* Setup and scaling require manual operations. +* Backups are supported on the _Main_ site only, not on the _Replica_ sites. -## Configuring the _Main_ site +## Choose your topology -You will use the externally reachable URI for each of your replica set instances, manually overwiriting its default local fully-qualified domain name (FQDN) in the Custom Resource manifest. Also you will need including all these host names into TLS certificates. So the first thing needed is the list of these externally reachable names. In the above example we will use the following ones: +| Goal | Topology | Members | Go to | +|------|----------|---------|--------| +| Distribute a full data copy across data centers | 1 Main + 2 Replica sites, all data-bearing | 3 voting data members | [Distribute data-bearing members across sites](#distribute-data-bearing-members-across-sites) | +| You have (or want) only two data-bearing sites and need an odd vote count with a third site that stores no data | Primary-Secondary-Arbiter (PSA) across sites | 2 data members + 1 external arbiter | [Deploy Primary-Secondary-Arbiter across sites](#deploy-primary-secondary-arbiter-across-sites) | -* `r1.percona.local:443` URI for the `cluster-name-rs0-0` (1st replica set instance), -* `r2.percona.local:443` for the 2nd replica set instance, -* `r3.percona.local:443` for the 3rd replica set instance. +For either options, you must do these [Common steps](#common-steps) first: -Following steps will allow you to prepare the _Main_ site for cross-site replication, keeping in mind the multiple data centers deployment: +1. Generate TLS certificates +2. Deploy the main site +3. Export Secrets from the main site -TLS certificates generated by the Operator are not suitable and it’s required to [generate certificates manually](tls-manual.md) on the _Main_ site before creating a database cluster, with all names from `replsetOverrides` and `externalNodes`. +In the following example setups we deploy sites in separate namespaces within a single Kubernetes cluster to emulate isolation and communication boundaries between data centers. In production, you will use separate Kubernetes clusters or physically distinct data centers to achieve true fault isolation, scalability, and resilience. -1. Use [TLS ceritficates manual generation instruction](tls-manual.md) to prepare TLS certificates with the host names from your prepared list. +## Common steps -2. Deploy your [Main site](replication-main.md) as usual, with these manually generated certificates. Don't forget to turn on [Pods exposure on your Main cluster](expose.md#controlling-hostnames-in-replset-configuration). +### Initial preparation -2. Now override hostname of the first replica in the replica set configuration by using the `replsets.replsetOverrides` subsection in the Custom Resource options manifest with the externally reachable endpoint from your externally reachable URI list: +1. Clone the repository with all manifests and source code: - ```yaml hl_lines="9" - ... - unsafeFlags: - replsetSize: true - replsets: - - name: rs0 - size: 1 - replsetOverrides: - cluster-name-rs0-0: - host: r1.percona.local:443 - ... + ```bash + git clone -b v{{release}} https://github.com/percona/percona-server-mongodb-operator + cd percona-server-mongodb-operator ``` - The `unsafeFlags.replsetSize` option in the above example is needed to create replica set with less than 3 instances. +2. Export namespace variables. Use the set that matches your topology: - The actual approach to make the URI reachable from the outside of your Kubernetes cluster depends on the exposure type. It is different in case of the [NodePort exposure :octicons-link-external-16:](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport), [Load balancer of the cloud provider :octicons-link-external-16:](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer), etc. Operator won’t perform any validation for hostnames. It’s user’s responsibility to ensure connectivity. + === "Three data-bearing sites" - !!! note + ```bash + export MAIN=main + export REPLICA1=replica1 + export REPLICA2=replica2 + ``` - You can also add custom tags to the replset members, just to make their identification easier: + === "PSA architecture" - ```yaml hl_lines="5" - ... - replsetOverrides: - cluster-name-rs0-0: - host: r1.percona.local:443 - tags: - team: cloud - ... - ``` + ```bash + export MAIN=main + export REPLICA=replica + export ARBITER=arbiter + ``` -### Configuring _Replica_ sites +3. Create the namespaces for your topology: -To configure _Replica_ sites, you should [deploy your Relica sites](replication-replica.md), repeating the following steps for each Kubernetes cluster you are adding: + === "Three data-bearing sites" -1. Copy secrets from the _Main_ site, rename them according to the cluster name you use on the _Replica_ site (if needed), and apply. + ```bash + kubectl create namespace $MAIN + kubectl create namespace $REPLICA1 + kubectl create namespace $REPLICA2 + ``` - * `cluster1-ssl` (SSL certificates for client connections), + === "PSA architecture" - * `cluster1-ssl-internal` (SSL certificates for replication), + ```bash + kubectl create namespace $MAIN + kubectl create namespace $REPLICA + kubectl create namespace $ARBITER + ``` - * `cluster1-secrets` (user credentials), +4. Deploy the Operator in each namespace. Replace `` with each value you exported: - * `cluster1-mongodb-encryption-key` (encryption key). + ```bash + kubectl apply --server-side -f deploy/bundle.yaml -n + ``` -2. Deploy the database cluster on the _Replica_ site. Don't forgetting the following: +### Deploy the Main site + +TLS certificates generated by the Operator are not enough for multi-site layouts. You need to generate certificates manually on the Main site before you create the database cluster, and include the Main cluster name and the hostname of every replica set member you will connect. + +1. Prepare TLS certificates with the Main site cluster name, the hostnames of all members you will add later, and the `$MAIN` namespace. See [TLS certificates manual generation](tls-manual.md) for the steps. + +2. Rename `deploy/cr.yaml` to `deploy/cr-main.yaml`. Edit it and set: + + * `metadata.name` to a distinct cluster name (for example, `main-cluster`). It must match the name you used when generating certificates. + * `spec.secrets.ssl` and `spec.secrets.sslInternal` to the certificates you created + * `spec.unsafeFlags.replsetSize` to `true` to create a replica set with fewer than three instances + * `spec.replsets.size` to `1` + * `spec.sharding.enabled` to `false` + * [Expose](expose.md#controlling-hostnames-in-replset-configuration) the replica set Pod so other sites can reach it. For example, via MCS `svc.clusterset.local`, or by using an externally reachable Service type such as `LoadBalancer`/`NodePort` and matching hostnames in TLS certificates. + + The sample configuration looks like this: + + ```yaml title="cr-main.yaml" + apiVersion: psmdb.percona.com/v1 + kind: PerconaServerMongoDB + metadata: + name: main-cluster + finalizers: + - percona.com/delete-psmdb-pods-in-order + spec: + unsafeFlags: + replsetSize: true + updateStrategy: SmartUpdate + secrets: + users: main-cluster-secrets + encryptionKey: main-cluster-mongodb-encryption-key + ssl: main-cluster-ssl + sslInternal: main-cluster-ssl-internal + replsets: + - name: rs0 + size: 1 + expose: + enabled: true + type: ClusterIP + sharding: + enabled: false + # The rest of your configuration + ``` - 1. All _Replica_ sites must be deployed with the `unmanaged: true` Custom Resource option. This will stop the Operator in the _Replica_ cluster from touching the MongoDB replset configuration. Starting from this moment, only the Operator of the _Main_ cluster will be able to modify it. +3. Deploy the Main site: - 2. Backups must be disabled with the `backup.enabaled: false` Custom Resource option. + ```bash + kubectl apply -f deploy/cr-main.yaml -n $MAIN + ``` - 3. The `updateStrategy` Custom Resource option must be set to `RollingUpdate` or `OnDelete`. - - 4. In order to create a single-instance replica set, you will need to the `unsafeFlags.replsetSize` option to `true` as you did on the _Main_ site. +4. Wait until the cluster reports `Ready`: + + ```bash + kubectl get psmdb main-cluster -n $MAIN -o yaml | grep state: + ``` + + Expected output: `state: Ready`. + +### Export Secrets from the Main site + +All replica set sites must share the same user credentials and TLS certificates. For this reason, export them from the main site and create the corresponding Secrets on the replica sites. Follow the steps in the [Export the Secrets](replication-main.md#export-the-cluster-secrets-and-certificates-to-be-copied-from-main-to-replica) section. + +Also, update the namespace in each Secret to point to the target namespace. Depending on the topology you chose, the target namespaces are: `$REPLICA1`/`$REPLICA2` for the replica clusters, or `$REPLICA`/ `$ARBITER` for the PSA. + +Replace the ``, `-` placeholders as needed in the following command: + + ```bash + sed -i.bak "s/namespace: $MAIN/namespace: $REPLICA/" "-.yaml" + ``` + +As a result you should have the full set of Secret files for each unmanaged site you will deploy: + +* `-ssl.yaml` - SSL certificates for client connections, +* `-ssl-internal.yaml` - SSL certificates for replication, +* `-secrets.yaml` - user credentials, +* `-mongodb-encryption-key.yaml` - encryption key file. + +Continue with [Distribute data-bearing members across sites](#distribute-data-bearing-members-across-sites) or [Deploy Primary-Secondary-Arbiter across sites](#deploy-primary-secondary-arbiter-across-sites) for further configuration. + +## Distribute data-bearing members across sites + +In this site you deploy one Main site and two Replica sites. Each site contains a single data-bearing member. Together they form a three-member replica set with an odd number of voters and a full data copy in every data center. + +| Site | Role | Members | Name | Namespace | +|------|------|---------|------|-----------| +| Main (data center 1) | Data-bearing, Operator-managed | 1 Pod (`size: 1`) | `main-cluster` | `main` | +| Replica 1 (data center 2) | Data-bearing, unmanaged | 1 Pod (`unmanaged: true`, `size: 1`) | `replica1-cluster` | `replica1` | +| Replica 2 (data center 3) | Data-bearing, unmanaged | 1 Pod (`unmanaged: true`, `size: 1`) | `replica2-cluster` | `replica2` | + + +### Deploy the Replica sites + +Make sure you have completed the [Common steps](#common-steps) before you proceed. + +Repeat the following for **each** Replica site (`replica1-cluster` and `replica2-cluster`). The steps differ only by cluster name and namespace. + +The commands below show the configuration of the `replica1-cluster` cluster. + +1. Apply the Secret files you prepared for that site: + + ```bash + kubectl apply -f replica1-cluster-mongodb-encryption-key.yaml -n $REPLICA1 + kubectl apply -f replica1-cluster-secrets.yaml -n $REPLICA1 + kubectl apply -f replica1-cluster-ssl-internal.yaml -n $REPLICA1 + kubectl apply -f replica1-cluster-ssl.yaml -n $REPLICA1 + ``` + +2. Create a Custom Resource manifest for the site. Rename the `deploy/cr.yaml` to `deploy/cr-replica1.yaml`. Set the following keys: + + * `metadata.name` to a distinct name (`replica1-cluster` / `replica2-cluster`) + * `spec.secrets` to the Secrets you created for that site in `users`, `encryptionKey`, `ssl`, `sslInternal` + * `unmanaged: true` — only the Main site Operator may change the replica set configuration + * `backup.enabled: false` - backups are available only on the Main site + * `updateStrategy` to `RollingUpdate` or `OnDelete` + * `unsafeFlags.replsetSize: true` and `replsets.size: 1` + * Expose the Pod the same way you exposed the Main site member + + Here's the example configuration for the `replica1-cluster`: + + ```yaml title="cr-replica1.yaml" + apiVersion: psmdb.percona.com/v1 + kind: PerconaServerMongoDB + metadata: + name: replica1-cluster + finalizers: + - percona.com/delete-psmdb-pods-in-order + spec: + unmanaged: true + updateStrategy: RollingUpdate + secrets: + users: replica1-cluster-secrets + encryptionKey: replica1-cluster-mongodb-encryption-key + ssl: replica1-cluster-ssl + sslInternal: replica1-cluster-ssl-internal + unsafeFlags: + replsetSize: true + replsets: + - name: rs0 + size: 1 + expose: + enabled: true + type: ClusterIP + sharding: + enabled: false + backup: + enabled: false + # The rest of your configuration + ``` + +3. Deploy the site: + + ```bash + kubectl apply -f deploy/cr-replica1.yaml -n $REPLICA1 + ``` + +4. Repeat steps 2 and 3 and deploy the `replica2-cluster`. Replace the cluster name and the namespace. + +### Add replica sites to the Main site + +On the Main site, register both Replica members under `replsets.externalNodes` as voting members (`votes: 1`, `priority: 1`). + +Specify endpoints as `...svc.cluster.local:27017` (the service name is typically `-`). Confirm endpoint names with `kubectl get pods` and `kubectl get svc` on each site. If you use MCS or external hostnames, use those reachable names instead and include them in TLS certificates. + +```yaml title="cr-main.yaml" +replsets: +- name: rs0 + size: 1 + externalNodes: + - host: replica1-cluster-rs0-0.replica1-cluster-rs0.replica1.svc.cluster.local:27017 + votes: 1 + priority: 1 + - host: replica2-cluster-rs0-0.replica2-cluster-rs0.replica2.svc.cluster.local:27017 + votes: 1 + priority: 1 +``` + +Apply the updated Main site Custom Resource: -3. Now add the new _Replica_ site's Pod **to your _Main_ site's** `externalNodes` subsection of the Custom Resource options manifest: +```bash +kubectl apply -f deploy/cr-main.yaml -n $MAIN +``` - ```yaml hl_lines="8 9 10" - replsets: +## Deploy Primary-Secondary-Arbiter across sites + +Use this layout when you want **two data-bearing sites** and a third site that only participates in elections. This third site hosts the [arbiter](arbiter.md) node that stores no data and cannot become primary. The arbiter breaks election ties so the replica set keeps an odd number of voters. + +This topology uses `replsets.externalNodes.arbiterOnly` and requires Percona Operator for MongoDB 1.23.0 or later. + +| Site | Role | Members | Name | Namespace | +|------|------|---------|------|-----------| +| Main (data center 1) | Data-bearing, Operator-managed | 1 Pod (`size: 1`) | `main-cluster` | `main` | +| Replica (data center 2) | Data-bearing, unmanaged | 1 Pod (`unmanaged: true`, `size: 1`) | `replica-cluster` | `replica` | +| Arbiter (data center 3) | Arbiter only, unmanaged | 1 arbiter Pod | `arbiter-cluster` | `arbiter` | + +### Deploy the Replica site (data-bearing) + +Make sure you have completed the [Common steps](#common-steps) before you proceed. + +Deploy one unmanaged data-bearing Replica site following the steps from [Deploy the replica site](#deploy-the-replica-sites). +Create the Secrets from the files you created for the `replica-cluster`. + +Use the `$REPLICA` namespace and the cluster name `replica-cluster`. Apply the corresponding Secret files, set `unmanaged: true`, `backup.enabled: false`, `replsets.size: 1`, and expose the Pod. + +The example configuration is: + +```yaml title="cr-replica.yaml" +apiVersion: psmdb.percona.com/v1 +kind: PerconaServerMongoDB +metadata: + name: replica-cluster + finalizers: + - percona.com/delete-psmdb-pods-in-order +spec: + unmanaged: true + updateStrategy: RollingUpdate + secrets: + users: replica-cluster-secrets + encryptionKey: replica-cluster-mongodb-encryption-key + ssl: replica-cluster-ssl + sslInternal: replica-cluster-ssl-internal + unsafeFlags: + replsetSize: true + replsets: - name: rs0 size: 1 - replsetOverrides: - cluster1-rs0-0: - host: r1.percona.local:443 - externalNodes: - - host: r2.percona.local:443 - votes: 1 - priority: 1 + expose: + enabled: true + type: ClusterIP + sharding: + enabled: false + backup: + enabled: false + #The rest of your configuration +``` + +### Deploy the arbiter site + +Deploy a third cluster that hosts the arbiter node. You can run this site on Kubernetes (shown here) or as a standalone `mongod` arbiter outside Kubernetes. + +1. Apply the arbiter site Secrets: + + ```bash + kubectl apply -f arbiter-cluster-mongodb-encryption-key.yaml -n $ARBITER + kubectl apply -f arbiter-cluster-secrets.yaml -n $ARBITER + kubectl apply -f arbiter-cluster-ssl-internal.yaml -n $ARBITER + kubectl apply -f arbiter-cluster-ssl.yaml -n $ARBITER + ``` + +2. Configure the Custom Resource so this site runs an arbiter. Rename the file to `deploy/cr-arbiter.yaml`. + + Important settings: + + * A distinct cluster name (for example, `arbiter-cluster`) + * `unmanaged: true` + * `updateStrategy`: `RollingUpdate` or `OnDelete` + * `unsafeFlags.replsetSize: true` and `replsets.size: 0` + * `replsets.arbiter.enabled: true` and `arbiter.size: 1` + * Pod exposure enabled so other sites can reach the arbiter + * Secrets created in the previous step + + Example configuration: + + ```yaml title="cr-arbiter.yaml" + apiVersion: psmdb.percona.com/v1 + kind: PerconaServerMongoDB + metadata: + name: arbiter-cluster + spec: + unmanaged: true + updateStrategy: RollingUpdate + secrets: + users: arbiter-cluster-secrets + encryptionKey: arbiter-cluster-mongodb-encryption-key + ssl: arbiter-cluster-ssl + sslInternal: arbiter-cluster-ssl-internal + unsafeFlags: + replsetSize: true + replsets: + - name: rs0 + size: 0 + expose: + enabled: true + type: ClusterIP + arbiter: + enabled: true + size: 1 + sharding: + enabled: false + ``` + +3. Apply the configuration: + + ```bash + kubectl apply -f deploy/cr-arbiter.yaml -n $ARBITER ``` + The arbiter Pod is named `{cluster}-{replset}-arbiter-0` (for example, `arbiter-cluster-rs0-arbiter-0`). + +### Add the Replica member and external arbiter to the Main site + +On the Main site, register both remote members under `replsets.externalNodes`: + +* The Replica site data member as a normal voting member +* The arbiter site’s **arbiter Pod** (not its data Pod) with `arbiterOnly: true`, `votes: 1`, and `priority: 0`. + +Specify the member's endpoints in the format `...svc.cluster.local:27017` (the service name is typically `-`). To get the endpoint, run `kubectl get psmdb` on each site and check the output. + +```yaml title="cr-main.yaml" +replsets: +- name: rs0 + size: 1 + externalNodes: + - host: replica-cluster-rs0-0.replica-cluster-rs0.replica.svc.cluster.local:27017 + votes: 1 + priority: 1 + - host: arbiter-cluster-rs0-arbiter-0.arbiter-cluster-rs0.arbiter.svc.cluster.local:27017 + votes: 1 + priority: 0 + arbiterOnly: true +``` + +If the arbiter runs outside Kubernetes, use its externally reachable hostname instead. + +## Verify replica set + +1. Get the `databaseAdmin` password from the Main site: + + ```bash + kubectl get secret main-cluster-secrets -n $MAIN -o jsonpath="{.data.MONGODB_DATABASE_ADMIN_PASSWORD}" | base64 --decode + ``` + +2. Connect to the Main site Pod and list members: + + ```bash + kubectl -n $MAIN exec -it main-cluster-rs0-0 -- mongosh -u databaseAdmin -p + ``` + + ``` + rs.status().members + ``` + +The output differs per deployed topology: + +=== "Three data-bearing sites" + + You should see **three** data-bearing voting members (Main, Replica 1, Replica 2), all with `health: 1` when network and TLS configuration are correct. + +=== "PSA architecture" + + You should see **three** voting members: the Main data Pod, the Replica data Pod, and the external arbiter (`ARBITER` / `arbiterOnly`), all with `health: 1` when network and TLS configuration are correct. + +## Using more than one data-bearing member per site + +You can scale the same idea beyond single-member sites. For example, deploy two data-bearing members on the Main site and two on the Replica site (`size: 2` on each), then add a single external arbiter so the total number of voters stays odd (2 + 2 + 1). Use `unsafeFlags.replsetSize: true` when a site has an even number of local members without a local arbiter. The interconnect pattern is the same: list remote data members and the shared arbiter under `externalNodes`, with `arbiterOnly: true` on the arbiter entry. + diff --git a/docs/replication.md b/docs/replication.md index 9b1b52cd..9dced3a3 100644 --- a/docs/replication.md +++ b/docs/replication.md @@ -17,7 +17,7 @@ While the underlying mechanics such as replica sets, TLS, and service exposure a To maintain the same set of data in clusters within multi-cluster or multi-region deployment, the Operator uses the cross-site replication. This means that one cluster is the Main site and another one(s) - the Replica site(s). -The following diagram shows how the data is replicated between the sites +The following diagram shows how the data is replicated between the sites. ![image](assets/images/replication-pods.svg) @@ -26,6 +26,15 @@ The following diagram shows how the data is replicated between the sites This separation ensures consistency and avoids conflicts when managing distributed deployments. +### Voting members across sites + +MongoDB needs an **odd number of voting members** in each replica set for reliable primary elections. When you interconnect Main and Replica sites, you typically add remote members through `replsets.externalNodes` and keep the total voter count odd. Common approaches: + +* Add an even number of remote data-bearing members as voting, and one remote member as non-voting (`votes: 0`, `priority: 0`). The main [Interconnect sites](replication-interconnect.md) guide uses this pattern. +* Add an **external arbiter** as a voting member (`arbiterOnly: true`, `votes: 1`, `priority: 0`). The arbiter stores no data and cannot become primary, but it participates in elections. This is useful for a Primary-Secondary-Arbiter (PSA) layout distributed across data centers, or for two data-bearing sites plus a third arbiter site. + +For a full example that deploys two data-bearing clusters and an arbiter site, then registers the arbiter on the Main site, see [Splitting a replica set across multiple data centers](replication-multi-dc.md). + ## Why to use multi-cluster or multi-region? Choosing the right topology depends on your goals. Here are common use cases that you can achieve with these models: diff --git a/mkdocs-base.yml b/mkdocs-base.yml index 629768ab..c04c588b 100644 --- a/mkdocs-base.yml +++ b/mkdocs-base.yml @@ -295,7 +295,7 @@ nav: - "Interconnect sites": replication-interconnect.md - "Fail over workload to the Replica site": replication-failover.md - "Backups with cross-site replication": replication-backups.md - - "Splitting replica set across multiple data centers": replication-multi-dc.md + - "Splitting a replica set across multiple data centers": replication-multi-dc.md - "Monitor with Percona Monitoring and Management (PMM)": monitoring.md - "Add sidecar containers": sidecar.md