Skip to content

Commit cb27096

Browse files
nastena1606nastena1606
committed
K8SPS-849 Documented the support of binlog encryption
Restructured PITR docs to have a high-level overview page and PITR tutorials modified: docs/ReleaseNotes/Kubernetes-Operator-for-PS-RN1.2.0.md modified: docs/backups-encrypted.md modified: docs/backups-incremental.md modified: docs/backups-pitr.md new file: docs/backups-restore-pitr.md modified: docs/backups-restore-to-new-cluster.md modified: docs/backups-restore.md modified: docs/backups.md modified: docs/debug-backup-restore.md modified: docs/encryption.md modified: docs/features.md modified: docs/operator.md modified: docs/restore-cr.md modified: mkdocs-base.yml Updated links in affected pages
1 parent 5a65c18 commit cb27096

14 files changed

Lines changed: 643 additions & 452 deletions

‎docs/ReleaseNotes/Kubernetes-Operator-for-PS-RN1.2.0.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ This improvement makes the restore path straightforward. By separating binlog co
103103

104104
Point-in-time recovery remains a tech preview feature. We do not recommend using it in production environments yet, but we encourage you to try this workflow in staging or test clusters and share your feedback.
105105

106-
Read more about point-in-time recovery and restore to a new cluster in our [documentation](../backups-restore-to-new-cluster.md#restore-with-point-in-time-recovery).
106+
Read more about point-in-time recovery and restore to a new cluster in our [documentation](../backups-restore-pitr.md#restore-on-a-new-cluster).
107107

108108
### Disable NodePort allocation for LoadBalancer Services
109109

‎docs/backups-encrypted.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,3 +152,5 @@ See [Restore from a backup to a new Kubernetes-based environment](backups-restor
152152
!!! warning "Keep your backup encryption keys safe"
153153

154154
To restore from an encrypted backup, you **must have the original encryption key**. If the key is lost or rotated, your backups are irrecoverable. Always ensure you have a secure and reliable process for managing and backing up encryption keys separately from your database backups.
155+
156+
Binlogs collected for point-in-time recovery are stored separately from these backups. To encrypt those files in object storage, see [Binlog encryption](backups-pitr.md#binlog-encryption).

‎docs/backups-incremental.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ By default, the Operator uses the **latest full backup** as the base for both **
4747

4848
The restore flow is unified for both full and incremental backups. The Operator identifies the backup type by name or destination. To identify the increments and reconstruct the chain, the backup destination has now the `.incr` path segment. The Operator downloads the full backup and all related increments and sorts them in the correct order. Then it restores the full backup first and applies each incremental backup.
4949

50-
If you make a [point-in-time recovery](backups-pitr.md), it also applies binlogs on top, after restoring all backups. To learn more how it works, check [the point-in-time recovery workflow](backups-pitr.md#how-the-operator-performs-pointintime-recovery).
50+
If you make a [point-in-time recovery](backups-pitr.md), it also applies binlogs on top, after restoring all backups. To learn more, see [How it works](backups-pitr.md#how-it-works).
5151

5252
Here's how it works in detail:
5353

‎docs/backups-pitr.md‎

Lines changed: 200 additions & 93 deletions
Large diffs are not rendered by default.

‎docs/backups-restore-pitr.md‎

Lines changed: 296 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,296 @@
1+
# Perform a point-in-time recovery
2+
3+
Use this guide to restore your database up to a specific moment or a transaction target.
4+
5+
You can restore in place on the [same cluster](#restore-on-the-same-cluster) or onto a [new cluster](#restore-on-a-new-cluster). In both cases the Operator restores a base backup and then replays binary logs up to a timestamp or a GTID.
6+
7+
To restore from a base backup without replaying binlogs, see [Restore on the same cluster](backups-restore.md) or [Restore on a new cluster](backups-restore-to-new-cluster.md).
8+
9+
!!! warning
10+
11+
During point-in-time recovery, the Operator pauses the cluster, restores the base backup, applies binlogs up to your target, then brings the cluster back online. Binlog Server must be running so the Operator can find the required binlogs before it pauses. For more information about the workflow, see [How it works](backups-pitr.md#how-it-works).
12+
13+
## Before you begin
14+
15+
* [Enable binlog collection](backups-pitr.md#enable-binlog-collection) on the source cluster
16+
* Make a base backup **before** the target time or transaction
17+
* Choose a target timestamp (`type: date`) or a GTID set (`type: gtid`)
18+
* Set the `spec.backup.backoffLimit=0` in the cluster Custom Resource so a failed PITR Job does not retry automatically ([known limitations](backups-pitr.md#known-limitations))
19+
* If the source cluster encrypted binlogs, follow [Restore with encrypted binlogs](#restore-with-encrypted-binlogs)
20+
21+
22+
23+
## Restore on the same cluster
24+
25+
The Operator reads binlog storage from `spec.backup.pitr.binlogServer` on the cluster Custom Resource.
26+
27+
1. Confirm the cluster is running and list backups:
28+
29+
```bash
30+
kubectl get ps <cluster-name> -n <namespace>
31+
kubectl get ps-backup -n <namespace>
32+
```
33+
34+
2. Edit the [deploy/backup/restore.yaml :octicons-link-external-16:](https://github.com/percona/percona-server-mysql-operator/blob/v{{release}}/deploy/backup/restore.yaml) manifest. Set `spec.clusterName`, `spec.backupName`, and the `spec.pitr` target.
35+
36+
=== "Restore to a timestamp"
37+
38+
Use `type: date` when you know the wall-clock time to restore to.
39+
40+
```yaml
41+
apiVersion: ps.percona.com/v1
42+
kind: PerconaServerMySQLRestore
43+
metadata:
44+
name: restore-pitr-date
45+
spec:
46+
clusterName: ps-cluster1
47+
backupName: backup1
48+
pitr:
49+
type: date
50+
date: "2026-03-20 09:15:00"
51+
```
52+
53+
=== "Restore to a transaction"
54+
55+
Use `type: gtid` when you know the GTID to stop before. The value has the format `source_id:transaction_id`.
56+
57+
```yaml
58+
apiVersion: ps.percona.com/v1
59+
kind: PerconaServerMySQLRestore
60+
metadata:
61+
name: restore-pitr-gtid
62+
spec:
63+
clusterName: ps-cluster1
64+
backupName: backup1
65+
pitr:
66+
type: gtid
67+
gtid: "cc5e06e7-241e-11f1-a165-522d36bd0c5e:225"
68+
```
69+
70+
3. Apply the configuration:
71+
72+
```bash
73+
kubectl apply -f deploy/backup/restore.yaml -n <namespace>
74+
```
75+
76+
Then [watch the restore](#view-restore-details).
77+
78+
## Restore on a new cluster
79+
80+
The Operator starts a temporary Binlog Server from `spec.pitr.backupSource.binlogServer`, fetches binlogs from the **source** storage, then removes that Pod. Binlog storage is S3 or S3-compatible only, even when the base backup is on GCS or Azure.
81+
82+
### Checklist
83+
84+
* Target cluster is running
85+
* User password Secret matches the source cluster. Copy it as described in [Preconditions](backups-restore-to-new-cluster.md#preconditions)
86+
* If the base backup was [encrypted](backups-encrypted.md), create the same encryption-key Secret on the target and set `spec.backupSource.storage.encryptionKeySecret`
87+
* If source binlogs were encrypted, copy the keyring Secret and set `spec.pitr.keyringSecret`. See [Restore with encrypted binlogs](#restore-with-encrypted-binlogs)
88+
89+
Keep these three prefixes distinct:
90+
91+
| Prefix | Where you set it | Value |
92+
| ------ | ---------------- | ----- |
93+
| Backup | `spec.backupSource.storage` | Same folder as when the backup was taken |
94+
| Source binlogs | `spec.pitr.backupSource.binlogServer.storage.s3.prefix` | Copy from the source cluster's Binlog Server |
95+
| Target collection | cluster `spec.backup.pitr.binlogServer` **after** restore | New folder for binlogs this cluster will collect. Must differ from the source prefix if they share a bucket |
96+
97+
### Retrieve the backup destination
98+
99+
The Operator must know where to take the backup from. Run this command on the **source cluster**:
100+
101+
```bash
102+
kubectl get ps-backup -n <source-namespace>
103+
```
104+
105+
Look for the `destination` value. You will use it for the restore configuration.
106+
107+
### Create the restore object
108+
109+
Edit [deploy/backup/restore.yaml :octicons-link-external-16:](https://github.com/percona/percona-server-mysql-operator/blob/v{{release}}/deploy/backup/restore.yaml). Set these keys:
110+
111+
* `clusterName` - the name of the **target cluster**
112+
* `backupSource` section:
113+
114+
* `destination` - where the backup is located. See [Retrieve the backup destination](#retrieve-the-backup-destination) how to get the backup destination
115+
* `storage` - configure the storage where the backup is stored. Specify the bucket, region, credentials Secret for the Operator to access the storage. If you specified a separate folder for backups on the bucket, specify it for the `prefix` option.
116+
117+
* `pitr` section:
118+
119+
* `type` - Set `date` to restore to a time or `gtid` to restore to a transaction
120+
* `date` - Specify the time in the format `YYY-MM-DD hh:mm:ss`. Use if if `type=date`
121+
* `gtid` - Specify the GTID set in the format `source_id:transaction_id`
122+
* `backupSource.binlogServer` - configure access to the binlog storage on the source cluster. Use the same settings as in the source cluster’s `spec.backup.pitr.binlogServer`, including the `prefix` for the binlog folder.
123+
124+
```yaml
125+
apiVersion: ps.percona.com/v1
126+
kind: PerconaServerMySQLRestore
127+
metadata:
128+
name: restore-pitr
129+
spec:
130+
clusterName: ps-cluster1
131+
backupSource:
132+
destination: s3://S3-BUCKET-NAME/BACKUP-NAME
133+
storage:
134+
type: s3
135+
s3:
136+
bucket: S3-BUCKET-NAME
137+
credentialsSecret: ps-cluster1-s3-credentials
138+
region: us-west-2
139+
prefix: <BACKUP-PREFIX>
140+
# encryptionKeySecret: # if the base backup was encrypted
141+
# name: my-encryption-key
142+
# key: encryptionKey
143+
pitr:
144+
type: date # or gtid
145+
date: "2026-03-20 09:15:00" # or gtid: "uuid:1-100"
146+
backupSource:
147+
binlogServer:
148+
storage:
149+
s3:
150+
bucket: S3-BINLOG-BUCKET-NAME
151+
credentialsSecret: ps-cluster1-s3-credentials
152+
region: us-west-2
153+
prefix: binlogs # source binlog folder
154+
```
155+
156+
| If you need to… | Change |
157+
| --------------- | ------ |
158+
| Stop at a GTID | `pitr.type: gtid` and `pitr.gtid` instead of `date` |
159+
| Restore a GCS base backup | `backupSource.destination` (`gs://…`), `storage.type: gcs`, and the `gcs` keys. Binlogs stay under `pitr.backupSource.binlogServer.storage.s3` |
160+
| Decrypt an encrypted backup | Uncomment `encryptionKeySecret` |
161+
| Decrypt encrypted binlogs | See [Restore with encrypted binlogs](#restore-with-encrypted-binlogs) |
162+
163+
164+
Start the restore:
165+
166+
```bash
167+
kubectl apply -f deploy/backup/restore.yaml -n <namespace>
168+
```
169+
170+
### After the restore
171+
172+
* [Enable binlog collection](backups-pitr.md#enable-binlog-collection) on the restored cluster with a **new** prefix if you share the source bucket.
173+
* Take a fresh base backup to start a new timeline.
174+
175+
## Restore with encrypted binlogs
176+
177+
Use this section when the source cluster [encrypted binlogs](backups-pitr.md#binlog-encryption) in object storage. The restore Job decrypts each file with the key encryption key (KEK) recorded in that file's metadata. Unencrypted binlogs in the same bucket are applied as-is.
178+
179+
You **must have every KEK** that wrapped the binlogs you are replaying. If a key is missing from the keyring, those files cannot be decrypted.
180+
181+
### On the same cluster
182+
183+
If `spec.backup.pitr.binlogServer.keyringSecret` is still set on the cluster and the Secret still holds every KEK, run the restore as in [Restore on the same cluster](#restore-on-the-same-cluster). The Operator mounts that keyring automatically. No extra fields are required.
184+
185+
Set `spec.pitr.keyringSecret` on the restore object when the keys you need live in a **different** Secret than the one on the cluster (for example after you rotated keys into a new Secret):
186+
187+
```yaml
188+
apiVersion: ps.percona.com/v1
189+
kind: PerconaServerMySQLRestore
190+
metadata:
191+
name: restore-pitr-encrypted
192+
spec:
193+
clusterName: ps-cluster1
194+
backupName: backup1
195+
pitr:
196+
type: date
197+
date: "2026-03-20 09:15:00"
198+
keyringSecret:
199+
name: ps-cluster1-binlog-server-keyring
200+
key: keyring.json
201+
```
202+
203+
The `key` field defaults to `keyring.json` if you omit it.
204+
205+
### New cluster
206+
207+
The target cluster has no access to Secrets from the source environment. Copy the keyring Secret, then point the restore at it.
208+
209+
1. On the **source** cluster, export the keyring Secret. Use the name from `spec.backup.pitr.binlogServer.keyringSecret` on the source Custom Resource:
210+
211+
```bash
212+
kubectl get secret ps-cluster1-binlog-server-keyring -n <source-namespace> -o yaml > binlog-keyring.yaml
213+
```
214+
215+
2. Strip cluster-specific metadata:
216+
217+
```bash
218+
yq eval 'del(.metadata.ownerReferences, .metadata.annotations, .metadata.labels, .metadata.creationTimestamp, .metadata.resourceVersion, .metadata.selfLink, .metadata.uid, .metadata.namespace)' binlog-keyring.yaml > binlog-keyring-target.yaml
219+
```
220+
221+
3. Apply the Secret on the **target** cluster:
222+
223+
```bash
224+
kubectl apply -f binlog-keyring-target.yaml -n <target-namespace>
225+
```
226+
227+
Confirm the Secret's `keyring.json` still lists **every** KEK that encrypted the binlogs still in the source bucket (including retired `id` values after rotation).
228+
229+
4. Create the restore as in [Restore on a new cluster](#restore-on-a-new-cluster) and add `spec.pitr.keyringSecret`:
230+
231+
```yaml
232+
spec:
233+
clusterName: ps-cluster1
234+
backupSource:
235+
destination: s3://S3-BUCKET-NAME/BACKUP-NAME
236+
storage:
237+
type: s3
238+
s3:
239+
bucket: S3-BUCKET-NAME
240+
credentialsSecret: ps-cluster1-s3-credentials
241+
region: us-west-2
242+
prefix: <BACKUP-PREFIX>
243+
pitr:
244+
type: date
245+
date: "2026-03-20 09:15:00"
246+
keyringSecret:
247+
name: ps-cluster1-binlog-server-keyring
248+
key: keyring.json
249+
backupSource:
250+
binlogServer:
251+
storage:
252+
s3:
253+
bucket: S3-BINLOG-BUCKET-NAME
254+
credentialsSecret: ps-cluster1-s3-credentials
255+
region: us-west-2
256+
prefix: binlogs
257+
```
258+
259+
If the **base backup** was also encrypted, add `spec.backupSource.storage.encryptionKeySecret` as well. That key is separate from the binlog keyring. See [Encrypted backups](backups-encrypted.md).
260+
261+
!!! warning "Keep your binlog encryption keys safe"
262+
263+
If a KEK is lost or removed from the keyring, binlogs wrapped with that key are irrecoverable. Store a backup of the keyring Secret separately from the binlog bucket.
264+
265+
Apply the restore and [watch the restore](#view-restore-details). If decryption fails, see [Encrypted binlogs fail to decrypt](debug-backup-restore.md#encrypted-binlogs-fail-to-decrypt).
266+
267+
## Ignore SQL errors during binlog replay
268+
269+
Updates made by `mysql-shell` can produce replication errors during replay, for example `Error_code: 1032` / `HA_ERR_KEY_NOT_FOUND` on `mysql_innodb_cluster_metadata.instances`.
270+
271+
To ignore SQL errors, add `force: true` under `spec.pitr`. This passes `--force` to the MySQL client.
272+
273+
```yaml
274+
spec:
275+
pitr:
276+
force: true
277+
type: date
278+
date: "2026-04-16 21:12:00"
279+
```
280+
281+
!!! warning "`force: true` can hide data loss"
282+
283+
The client silently ignores **all** SQL errors during binlog replay, not only metadata errors. Use it only when you accept that risk.
284+
285+
## View restore details
286+
287+
Point-in-time recovery creates a base restore Job (`xb-restore-<name>`) and a PITR Job (`pitr-restore-<name>`). Check both:
288+
289+
```bash
290+
kubectl get job -n <namespace>
291+
kubectl get ps-restore -n <namespace>
292+
```
293+
294+
## Troubleshooting
295+
296+
If the restore fails, see [Troubleshoot backups and restores](debug-backup-restore.md#restores) and [Point-in-time recovery](debug-backup-restore.md#point-in-time-recovery).

0 commit comments

Comments
 (0)