Skip to content

K8SPS-678 Documented the support of page tracking for incremental backups #241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nastena1606 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

K8SPS-678 Documented the support of page tracking for incremental backups #241

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 67 additions & 59 deletions docs/backups-incremental.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,11 @@

This feature is in the tech preview stage. The behavior can change in future releases.

If your database grows quickly or sees heavy write traffic, you may want frequent backups. Taking a full backup every time costs more storage, takes longer to upload, and adds load to the cluster.
**Incremental** backups copy only what changed since the previous backup in the chain. They are smaller, faster to transfer, and cheaper to keep than taking a full backup every time.

**Incremental** backups copy only what changed since the previous backup in the chain. They are usually smaller, faster to transfer, and cheaper to keep in the backup storage.
## Backup chain

## What you need first

Every incremental backup belongs to a **chain** that starts with one **full** backup on the **same** storage. The Operator checks that a valid full backup exists before it starts an incremental one. Using the same storage for the whole chain enables the Operator to reuse the same credentials and paths, and simplifies restore.

You can use any [supported storage type](backups-storage.md).

## What the backup chain looks like

Incremental backups build on the full backup and on each other in order:
Every incremental backup belongs to a **chain** that starts with one **full** backup on the **same** storage. The Operator checks that a valid full backup exists before it starts an incremental one.

```mermaid
flowchart LR
Expand All @@ -28,81 +20,97 @@ flowchart LR
D --> IN["Increment N"]
```

By default, the Operator uses the **latest full backup** as the base for both **scheduled** and **on-demand** incremental backups. If you want more control, you can explicitly specify the base backup in the configuration for on-demand backups. The Operator validates the specified backup and derives the incremental chain from it.
By default, the Operator uses the **latest full backup** as the base. For on-demand backups, you can set a different base with `spec.incrementalBaseBackupName`.

You can use any [supported storage type](backups-storage.md).

## How an incremental backup runs

1. You create a `PerconaServerMySQLBackup` object with type **incremental**. Or, you configure the **backup schedule** in the cluster Custom Resource that creates incremental backups.
2. The Operator confirms that a **full** backup exists on the same storage and is valid. Unless you specify another backup, it uses the **most recent** full backup.
3. If the backup is valid, the Operator sets the `percona.com/base-backup-name` annotation on it so that it serves as the base for the incremental backup chain.
4. If there are already incremental backups derived from the base, the Operator takes the `to_lsn` value from the previous increment and sets it as the `from_lsn` value for the new increment so the chain stays continuous.
5. The Operator streams the new incremental to the same storage.
6. The Operator records the backup type on the backup object.
1. You [make an on-demand backup](backups-ondemand.md) with type **incremental**, or you [configure a schedule](backups-scheduled.md) that creates incremental backups.
2. The Operator confirms that a valid **full** backup exists on the same storage. Unless you specify another backup, it uses the most recent full backup.
3. The Operator marks that full backup as the chain base and streams the increment to the same storage.
4. Later increments use the previous increment’s `to_lsn` as `from_lsn` so the chain stays continuous.

!!! important

The `percona.com/base-backup-name` annotation is internal and serves to correctly link incremental backups to the base one. Removing or editing it will lead to unpredictable results and data corruption. Don't remove or edit this annotation.
The Operator sets the `percona.com/base-backup-name` annotation on the base backup to link the chain. Do not edit or remove it. Changing it can break the chain and lead to data corruption.

## How restore from an incremental backup works
## Restore from an incremental backup

The restore flow is unified for both full and incremental backups. The Operator identifies the backup type by name or destination. To identify the increments and reconstruct the chain, the backup destination has now the `.incr` path segment. The Operator downloads the full backup and all related increments and sorts them in the correct order. Then it restores the full backup first and applies each incremental backup.
The restore flow is the same for full and incremental backups. Follow [Restore the cluster from a previously saved backup](backups-restore.md) or [Restore from a backup to a new Kubernetes-based environment](backups-restore-to-new-cluster.md).

If you make a [point-in-time recovery](backups-pitr.md), it also applies binlogs on top, after restoring all backups. To learn more how it works, check [the point-in-time recovery workflow](backups-pitr.md#how-the-operator-performs-pointintime-recovery).
The Operator identifies the backup type by name or destination. Incremental destinations use an `.incr` path segment so the Operator can reconstruct the chain. It downloads the full backup and all related increments, restores the full backup first, then applies each increment.

Here's how it works in detail:
If you make a [point-in-time recovery](backups-pitr.md), the Operator applies binlogs after the chain. See [the point-in-time recovery workflow](backups-pitr.md#point-in-time-recovery-workflow).

1. You create a `PerconaServerMySQLRestore` object and reference the incremental backup with `backupName` (same cluster) or `backupSource` (remote path / another environment).
2. The Operator detects that the target is incremental and resolves the chain.
3. It connects to the storage and lists full backup plus all increments up to and including the one you chose, using paths that include the `.incr` segment so increments are unambiguous.
4. The Operator sorts increments in the right order.
5. The Operator pauses the cluster for the restore.
6. The restore Job downloads and applies the full backup, then each incremental one in order.
7. The cluster is unpaused when the restore completes.
Incremental objects in storage use an `.incr` path segment so the Operator can tell them apart from full backups:

!!! admonition "Path layout"
```
s3://bucket/prefix/
my-cluster-2026-04-06-full/ # base full backup
my-cluster-2026-04-06-full.incr/ # incremental chain directory
my-cluster-2026-04-07T000000-incr/
my-cluster-2026-04-08T000000-incr/
my-cluster-2026-04-09T000000-incr/
```

## Speed up incremental backups with page tracking

Incremental destinations use an `.incr` segment in the path so the Operator can tell full and incremental artifacts apart. A typical pattern resembles `prefix/<cluster>-<timestamp>-full.incr/<cluster>-<timestamp>-incr`; exact layout follows your `prefix` and storage settings.
!!! admonition "Version added: [1.3.0](ReleaseNotes/Kubernetes-Operator-for-PS-RN1.3.0.md)"

Here's the example:
Page tracking lets Percona XtraBackup copy only the InnoDB pages that changed since the last backup, instead of scanning every data file. The Operator installs the `mysqlbackup` component that this feature needs. For details, see [Take an incremental backup using page tracking :octicons-link-external-16:](https://docs.percona.com/percona-xtrabackup/latest/page-tracking.html).

```
s3://bucket/prefix/
my-cluster-2026-04-06-full/ # base full backup
my-cluster-2026-04-06-full.incr/ # incremental chain directory
my-cluster-2026-04-07T000000-incr/ # Monday's incremental
my-cluster-2026-04-08T000000-incr/ # Tuesday's incremental
my-cluster-2026-04-09T000000-incr/ # Wednesday's incremental
```
Enable page tracking by passing the `--page-tracking` flag in the `xtrabackup` container options. Set it on the **storage** so the whole chain uses it:

## Why you need incremental backups
```yaml title="deploy/cr.yaml"
spec:
backup:
storages:
s3-us-west:
containerOptions:
args:
xtrabackup:
- "--page-tracking"
```

With incremental backups, you gain the following benefits:
Override it for a single on-demand backup:

```yaml title="deploy/backup/backup.yaml"
spec:
clusterName: ps-cluster1
storageName: s3-us-west
type: incremental
containerOptions:
args:
xtrabackup:
- "--page-tracking"
```

* strengthen your backup strategy by creating multiple restore points
* increase storage efficiency by avoiding duplication of unchanged data
* lower system load, since smaller backups require fewer compute resources and reduce impact on your cluster
* reduce both storage and data‑transfer costs
Pass the `--page-tracking` flag on the **base** backup to start tracking for the whole chain. The next incremental backup then copies only changed pages. Keep the flag on later increments so tracking continues.

If you set the `--page-tracking` flag only on an incremental job, that job still scans all pages because the previous full backup did not start tracking.

For how cluster-level and per-job options interact, see [Fine-tuning backup and restore operations](backups-fine-tune.md).

## Implementation specifics and rules

### Backup chain rules

1. A full backup is required to start the incremental chain. If none exists, the incremental backup fails.
2. A base full backup and incremental backups derived from it must be **on the same storage**.
3. By default, the Operator uses the most recent full backup to start the incremental chain. You can explicitly specify the base full backup in the `spec.incrementalBaseBackupName` option in the backup configuration file. If the specified full backup is valid, the Operator starts the incremental backup chain from it.
4. If the base backup already has the incremental backup chain, the Operator uses the most recent increment to continue the chain.
5. Retention applies to the chain as a unit: deleting the **base** full backup removes the **entire** incremental chain that depends on it, so you do not leave orphaned increments. Specifying the retention policy for increments is not supported.
6. You cannot delete an increment in the middle of a chain as it would break its continuity. You can delete only the **last** increment in the chain or the base backup, which removes the whole chain.
7. The Operator prevents to run two concurrent incremental backups against the **same** chain to avoid ambiguous ordering. The Operator runs increments one by one.
1. A full backup is required to start the chain. If none exists, the incremental backup fails.
2. The base full backup and its increments must be on the **same** storage.
3. By default, the Operator uses the most recent full backup. You can set a different base with `spec.incrementalBaseBackupName`. If that backup is valid, the Operator starts the chain from it.
4. If the base backup already has increments, the Operator continues from the most recent increment.
5. Retention applies to the chain as a unit: deleting the **base** full backup removes the entire chain. You cannot set retention on increments alone.
6. You cannot delete an increment in the middle of a chain. You can delete only the **last** increment, or the base backup, which removes the whole chain.
7. The Operator does not run two incremental backups on the **same** chain at once. Increments run one by one.

### Restore rules

1. You can make either an in-place restore pointing at an incremental backup object in the `backupName` option, or make a cross-cluster restore specifying the incremental backup path for the `backupSource.destination` option.
2. Restores that use `backupSource` work across clusters and namespaces when the storage destination is reachable; incremental paths remain discoverable because of the `.incr` layout.
1. [Restore in place](backups-restore.md) with `backupName`, or [restore across clusters](backups-restore-to-new-cluster.md) with `backupSource.destination`.
2. `backupSource` restores work across clusters and namespaces when the storage path is reachable. The `.incr` layout keeps incremental paths discoverable.
3. Restore always needs the full chain: full backup first, then increments in order, up to the backup you selected.


## Known limitations

- If a backup in your chain fails but some data was already uploaded to storage, your restore still includes that failed backup, so the restore fails. Whenever any backup in the chain fails, start a new chain.
- If the checkpoint file is missing from a backup directory, your next incremental backup can hang. Start a new chain.
- If a backup in your chain fails but some data was already uploaded, restore still includes that failed backup and fails. Start a new chain.
- If the checkpoint file is missing from a backup directory, the next incremental backup can hang. Start a new chain.