Skip to content

K8SPS-763: ensure --defaults-file is first flag - #1468

Open
pooknull wants to merge 11 commits into
mainfrom
K8SPS-763
Open

K8SPS-763: ensure --defaults-file is first flag#1468
pooknull wants to merge 11 commits into
mainfrom
K8SPS-763

Conversation

@pooknull

@pooknull pooknull commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

https://perconadev.atlassian.net/browse/K8SPS-763

DESCRIPTION

Problem:
Xtrabackup requires --defaults-file to be the first option. The operator placed custom xtrabackup arguments after its own, causing backups using --defaults-file to fail.

Solution:
Place --defaults-file before all generated arguments during backups.

CHECKLIST

Jira

  • Is the Jira ticket created and referenced properly?
  • Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
  • Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

  • Is an E2E test/test case added for the new feature/change?
  • Are unit tests added where appropriate?

Config/Logging/Testability

  • Are all needed new/changed options added to default YAML files?
  • Are all needed new/changed options added to the Helm Chart?
  • Did we add proper logging messages for operator actions?
  • Did we ensure compatibility with the previous version or cluster upgrade process?
  • Does the change support oldest and newest supported PS version?
  • Does the change support oldest and newest supported Kubernetes version?

@pull-request-size pull-request-size Bot added the size/L 100-499 lines label Jul 24, 2026
@github-actions github-actions Bot added the build label Jul 24, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses XtraBackup’s requirement that --defaults-file must be the first option by (a) enforcing ordering at the CRD/schema level for user-provided xtrabackup args and (b) ensuring the backup sidecar promotes --defaults-file ahead of operator-generated arguments during backup execution.

Changes:

  • Add CRD validations and bounds (maxItems, maxLength, maxProperties) to restrict and validate xtrabackup args, including enforcing --defaults-file as the first xtrabackup argument when present.
  • Update backup sidecar argument construction to place --defaults-file before generated xtrabackup options when provided as the first custom arg.
  • Add unit coverage for xtrabackupArgs ordering behavior and reorder ${XB_EXTRA_ARGS} earlier in restore script xtrabackup invocations.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
deploy/cw-bundle.yaml Adds schema validation/limits for xtrabackup args (including --defaults-file ordering) in the CW bundle output.
deploy/crd.yaml Adds schema validation/limits for xtrabackup args (including --defaults-file ordering) in the CRD manifest.
deploy/bundle.yaml Adds schema validation/limits for xtrabackup args (including --defaults-file ordering) in the operator bundle output.
config/crd/bases/ps.percona.com_perconaservermysqls.yaml Propagates xtrabackup arg validation/limits into the base CRD for PerconaServerMySQL.
config/crd/bases/ps.percona.com_perconaservermysqlrestores.yaml Propagates xtrabackup arg validation/limits into the base CRD for restores.
config/crd/bases/ps.percona.com_perconaservermysqlbackups.yaml Propagates xtrabackup arg validation/limits into the base CRD for backups.
cmd/sidecar/handler/backup/create.go Promotes --defaults-file (when provided as the first custom arg) to the front of the xtrabackup argv.
cmd/sidecar/handler/backup/create_test.go Adds unit tests verifying xtrabackup arg ordering, including defaults-file handling.
build/run-restore.sh Reorders ${XB_EXTRA_ARGS} before other xtrabackup options in restore flows.
api/v1/perconaservermysql_types.go Adds kubebuilder validations for xtrabackup args and limits storages map size via MaxProperties.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cmd/sidecar/handler/backup/create.go Outdated
Comment thread cmd/sidecar/handler/backup/create_test.go Outdated
@pooknull
pooknull marked this pull request as ready for review August 19, 2026 13:06
Comment thread api/v1/perconaservermysql_types.go Outdated
Comment on lines 559 to 560
// +kubebuilder:validation:XValidation:rule="!self.exists(arg, arg == '--defaults-file' || arg.startsWith('--defaults-file=')) || self[0] == '--defaults-file' || self[0].startsWith('--defaults-file=')",message="--defaults-file must be the first xtrabackup argument"
Xtrabackup []string `json:"xtrabackup,omitempty"`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add an envtest to cover this

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread cmd/sidecar/handler/backup/create.go Outdated
Comment on lines +241 to +243
// kubebuilder validation guarantees that --defaults-file is the first custom argument if specified.
// XtraBackup requires it to be the first option: https://docs.percona.com/percona-xtrabackup/8.0/xtrabackup-option-reference.html#defaults-file
end := 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we already guarantee this on CRD level, what's the purpose of this code?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe because here we take this config over HTTP so we can trust nothing from the request body I guess

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe the comment needs a little rewording

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@egegunes egegunes added this to the v1.3.0 milestone Aug 20, 2026
Comment thread cmd/sidecar/handler/backup/create.go Outdated
@@ -236,7 +237,23 @@ func xtrabackupArgs(user, pass string, conf *xb.BackupConfig) []string {
}
}
if conf != nil && conf.ContainerOptions != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe not related to your changes, but in this function in general we could move conf !=nil early, revert the clause and return it instead of repeating it all the time

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread cmd/sidecar/handler/backup/create.go Outdated
end := 0
if len(customArgs) > 0 {
switch {
case strings.HasPrefix(customArgs[0], "--defaults-file=") && customArgs[0] != "--defaults-file=":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does xtrabackup actually honor the space-separated form here? As I read MySQL's get_defaults_options / load_defaults, --defaults-file is consumed before normal option parsing and only the --defaults-file= prefix is matched, so --defaults-file /etc/my.cnf wouldn't be picked up as a defaults file at all, and the path would fall through as an unrecognized positional arg. Is that correct?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +242 to +251
if conf.ContainerOptions != nil {
customArgs := conf.ContainerOptions.Args.Xtrabackup
// kubebuilder validation guarantees that --defaults-file=<path> is the first custom argument if specified.
// https://docs.percona.com/percona-xtrabackup/8.0/xtrabackup-option-reference.html#defaults-file
// We should move it to the beginning of args. Other custom arguments should be appended after the generated arguments.
if len(customArgs) > 0 && strings.HasPrefix(customArgs[0], "--defaults-file=") && customArgs[0] != "--defaults-file=" {
args = append([]string{customArgs[0]}, args...)
customArgs = customArgs[1:]
}
args = append(args, customArgs...)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're re-ordering it anyway, do we need a CRD validation? Can we also do the same for restore?

@hors
hors requested a review from gkech September 3, 2026 17:37
@JNKPercona

Copy link
Copy Markdown
Collaborator
Test Name Result Time
async-ignore-annotations-8-4 passed 00:00:00
async-global-metadata-8-4 passed 00:00:00
async-upgrade-8-0 passed 00:00:00
async-upgrade-8-4 passed 00:00:00
auto-config-8-4 passed 00:00:00
config-8-4 passed 00:00:00
config-router-8-0 passed 00:00:00
config-router-8-4 passed 00:00:00
custom-users-8-4 passed 00:00:00
demand-backup-8-0 passed 00:00:00
demand-backup-8-4 passed 00:00:00
gr-pitr-minio-8-4 passed 00:00:00
gr-pitr-encrypted-minio-8-4 passed 00:00:00
gr-pitr-one-pod-8-4 passed 00:00:00
async-pitr-minio-8-4 passed 00:00:00
demand-backup-cloud-8-4 passed 00:00:00
demand-backup-retry-8-4 passed 00:00:00
demand-backup-incremental-8-0 passed 00:00:00
demand-backup-incremental-8-4 passed 00:00:00
async-data-at-rest-encryption-8-0 passed 00:00:00
async-data-at-rest-encryption-8-4 passed 00:00:00
gr-cross-cluster-8-0 passed 00:00:00
gr-cross-cluster-8-4 passed 00:00:00
gr-cross-cluster-backup-8-0 passed 00:27:50
gr-cross-cluster-backup-8-4 passed 00:27:19
gr-global-metadata-8-4 passed 00:00:00
gr-data-at-rest-encryption-8-0 passed 00:00:00
gr-data-at-rest-encryption-8-4 passed 00:00:00
gr-demand-backup-8-4 passed 00:00:00
gr-demand-backup-cloud-8-4 passed 00:24:30
gr-demand-backup-haproxy-8-4 passed 00:00:00
gr-demand-backup-incremental-8-0 passed 00:00:00
gr-demand-backup-incremental-8-4 passed 00:00:00
gr-demand-backup-incremental-compressed-8-0 passed 00:00:00
gr-demand-backup-incremental-compressed-8-4 passed 00:00:00
gr-demand-backup-incremental-encrypted-8-0 passed 00:00:00
gr-demand-backup-incremental-encrypted-8-4 passed 00:00:00
gr-finalizer-8-4 passed 00:00:00
gr-haproxy-8-0 passed 00:00:00
gr-haproxy-8-4 passed 00:00:00
gr-ignore-annotations-8-4 passed 00:00:00
gr-init-deploy-8-0 passed 00:00:00
gr-init-deploy-8-4 passed 00:00:00
gr-one-pod-8-4 passed 00:00:00
gr-recreate-8-4 passed 00:00:00
gr-scaling-8-4 passed 00:00:00
gr-scheduled-backup-8-4 passed 00:00:00
gr-scheduled-backup-incremental-8-4 passed 00:35:06
gr-security-context-8-4 passed 00:00:00
gr-self-healing-8-4 passed 00:00:00
gr-tls-cert-manager-8-4 passed 00:00:00
gr-users-8-4 passed 00:00:00
gr-upgrade-8-0 passed 00:00:00
gr-upgrade-8-4 passed 00:00:00
haproxy-8-0 passed 00:00:00
haproxy-8-4 passed 00:00:00
init-deploy-8-0 passed 00:00:00
init-deploy-8-4 passed 00:00:00
limits-8-4 passed 00:00:00
monitoring-8-4 passed 00:00:00
one-pod-8-0 passed 00:00:00
one-pod-8-4 passed 00:00:00
operator-self-healing-8-4 passed 00:00:00
pvc-auto-resize-8-4 passed 00:00:00
pvc-resize-8-4 passed 00:00:00
recreate-8-4 passed 00:00:00
scaling-8-4 passed 00:00:00
scheduled-backup-8-0 passed 00:00:00
scheduled-backup-8-4 passed 00:00:00
scheduled-backup-incremental-8-0 passed 00:37:42
scheduled-backup-incremental-8-4 passed 00:00:00
service-per-pod-8-4 passed 00:00:00
sidecars-8-4 passed 00:06:27
smart-update-8-4 passed 00:00:00
storage-8-4 passed 00:05:28
switch-cluster-type-8-4 passed 00:00:00
telemetry-8-4 passed 00:00:00
tls-cert-manager-8-4 passed 00:00:00
users-8-0 passed 00:00:00
users-8-4 passed 00:00:00
version-service-8-4 passed 00:00:00
Summary Value
Tests Run 81/81
Job Duration 01:03:33
Total Test Time 02:44:25

commit: 8e6ad52
image: perconalab/percona-server-mysql-operator:PR-1468-8e6ad528

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build size/L 100-499 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants