Skip to content

Scheduled backups are not created after the operator misses a cron tick #2526

Description

@guanchzhou

Scheduled backups are registered with robfig/cron inside the operator leader (createOrUpdateBackupTaskcrons.AddFunc). They are not Kubernetes CronJobs. robfig/cron does not fire ticks that were due while the process was down.

If the leader pod is restarted or evicted at the scheduled time, no PerconaServerMongoDBBackup is created for that interval. The next backup waits until the next cron fire. There is no catch-up.

This is silent: the cluster stays Ready and nothing records that a scheduled backup was skipped.

Expected: when the schedule is re-registered (leader start or failover) and the most recent backup for that task is older than the last due tick, create one catch-up backup. Do not create a backup on first install when no prior scheduled backup exists. Do not start a second backup if one is already running.

A PodDisruptionBudget on the operator Deployment with replicaCount >= 2 reduces how often this happens; it does not replace catch-up.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions