Scheduled backups run twice: upstream CronJob and Percona in-process scheduler both fire for the same schedule, racing for the pgBackRest backup lock

[leandro-clint]

Report

In operator 3.0.0, every backup schedule defined in spec.backups.pgbackrest.repos[].schedules is executed by two independent mechanisms simultaneously:

1. The upstream (CrunchyData) reconciler creates a Kubernetes CronJob (<cluster>-repo1-<type>) whose pod runs pgbackrest backup directly (internal/controller/postgrescluster/pgbackrest.go, ~L3184).
2. The Percona layer registers an in-process cron for the same schedule (percona/controller/pgcluster/schedule.go — reconcileScheduledBackup → createScheduledBackup) which creates a PerconaPGBackup CR (GenerateName: <cronjob-name>-, no annotations) ~30s later.

The PerconaPGBackup controller never "adopts" the CronJob's job: findBackupJob() (percona/controller/pgbackup/controller.go, ~L702-739) only matches manual-type jobs annotated with the CR's name. So reconciling the scheduler-created CR calls startBackup() and launches a second, manual-type backup job (<cluster>-backup-xxxx) for the same slot. Both pgbackrest processes race for the stanza's backup lock on the repo host:

ERROR: [050]: unable to acquire lock on file '/tmp/pgbackrest/db-backup-1.lock': Resource temporarily unavailable
HINT: is another pgBackRest process running?

Additionally, reconcileBackupJob() (percona/controller/pgcluster/backup.go, ~L187) creates a third object — a tracking PerconaPGBackup CR for the CronJob's job (annotated with pgv2.percona.com/pgbackrest-backup-job-name) — so two CRs accumulate per schedule slot.

More about the problem

The duplicate execution cascades into a self-perpetuating failure loop:

• Whichever process loses the lock race fails with exit 50; scheduled CronJob jobs intermittently end BackoffLimitExceeded (slots missing their backup, retries succeeding at odd times).
• When the duplicate manual job's pods exhaust retries and the job is removed, its CR remains in Starting forever — the BackupStarting branch requeues "Waiting for backup to start" every 5s with no timeout (percona/controller/pgbackup/controller.go, ~L240).
• The stuck CR holds the controller's lease (handleLease), freezing reconciliation of all pg-backup CRs for the cluster. In our production cluster, 936 unreconciled CRs accumulated over ~6 weeks behind a CR stuck in Starting.
• Deleting the stuck CR unfreezes the queue; the backlog then fires manual backups at arbitrary times, colliding with the scheduled CronJobs again and producing a new stuck CR. We reproduced and recorded this live.

Operator log while stuck (repeats every 5s, indefinitely):

INFO Waiting for backup to start {"controller": "perconapgbackup", "controllerKind": "PerconaPGBackup", "name": "postgres-ash-vega-prod-repo1-incr-7nskz", ...}

ps on the repo host during a slot — the CR-driven backup (note the --annotation) holding the lock while the scheduled CronJob's pods crash-loop with exit 50:

pgbackrest backup --stanza=db --repo=1 --type=incr --annotation=percona.com/backup-name=postgres-ash-vega-prod-repo1-incr-9qg9g

Also: stuck manual jobs remain Terminating indefinitely holding the internal.percona.com/keep-job finalizer after their CR is deleted.

Steps to reproduce

1. Deploy a PerconaPGCluster with operator 3.0.0 and backups.pgbackrest.repos[0].schedules set (e.g. incremental 20 0,3,9,12,15,18,21 * * *, S3 repo with dedicated repo host).
2. Watch a schedule slot: a CronJob job runs pgbackrest directly AND ~30s later an unannotated PerconaPGBackup CR is created; its reconcile spawns a <cluster>-backup-* job → lock collision on the repo host.
3. Let the manual job fail past its backoff limit (easy when the backup is long enough to overlap): its CR stays in Starting forever and all subsequent pg-backup CRs stop being reconciled (empty state).

Versions

• Operator: docker.io/percona/percona-postgresql-operator:3.0.0
• Database: PostgreSQL 18, percona/percona-pgbackrest:2.58.0-2 (pgBackRest 2.58.0)
• Kubernetes: v1.36.1+rke2r2 (RKE2, bare metal)
• Backup storage: S3-compatible object storage (path-style), dedicated repo host, archive-async=y

Anything else?

Production workaround we deployed: a cron "janitor" that (a) deletes scheduler-created CRs — no pgv2.percona.com/pgbackrest-backup-job-name annotation and name matching <cluster>-repo1-(incr|full)-[a-z0-9]{5} — before they spawn duplicate jobs, and (b) strips the orphaned keep-job finalizer from Terminating jobs. With the upstream CronJob as the single executor, every slot completes cleanly.

Suggested fixes:

• Gate the upstream CronJob creation when the in-process scheduler is active (or vice versa), or make the scheduler-created CR adopt the CronJob's running job in findBackupJob();
• Add a timeout/failure transition to the BackupStarting wait so a CR whose job disappeared cannot hold the lease forever.

[egegunes]

Interesting, last few days I was testing a problem in backups and scheduled backups to run every 10 minutes for 2 days and never seen seen something like this.

[egegunes]

@leandro-clint could you please share PerconaPGCluster and PostgresCluster yamls?

[leandro-clint]

Thanks for looking into this! YAMLs attached (public IPs redacted).

A few notes that may explain why a 10-minute test schedule wouldn't show visible failures — and a quick way to check whether the duplication is happening in your environment too:

1. The duplication is silent unless backups are slow. The double execution only becomes visible (failed jobs) when a backup holds the stanza lock longer than the retry budget of the competing job. Our database is 3.9TB; incrementals take 30-60+ min, so the two executors collide hard. With a small test database, backups finish in seconds: the loser of the lock race simply succeeds on its next retry and both backups complete — no Failed jobs, just twice the backups.

2. Instant check for the duplication (any env, any size): right after a scheduled slot fires, list the pg-backup objects:

kubectl get pg-backup -o jsonpath='{range .items[*]}{.metadata.name} anno={.metadata.annotations.pgv2\.percona\.com/pgbackrest-backup-job-name}{"\n"}{end}'

In our 3.0.0 environment, every slot produces two CRs: one annotated with pgbackrest-backup-job-name (created by reconcileBackupJob() in percona/controller/pgcluster/backup.go to track the upstream CronJob's job) and one without any annotations (created by the in-process scheduler in percona/controller/pgcluster/schedule.go — createScheduledBackup()). The unannotated one is reconciled through the manual-backup path (findBackupJob() only matches manual-type jobs annotated with the CR name, so it never adopts the CronJob's job) and spawns a <cluster>-backup-xxxx job that runs a second pgbackrest backup. You can also just count backups in pgbackrest info vs. the number of scheduled slots.

3. The lease/zombie chain makes it self-hiding over time. When a duplicate manual job dies past its backoff limit (lock contention, or transient S3 errors) and the job object goes away, its CR stays in Starting forever (Waiting for backup to start requeues every 5s with no timeout) holding the pg-<cluster>-backup-lock lease — which freezes reconciliation of ALL pg-backup CRs. From that point on the duplicate executor stops firing (queue frozen), so the env looks "normal" again while unreconciled CRs pile up (we accumulated 936 over ~6 weeks). Deleting the stuck CR unfreezes the queue and the collisions start again. So depending on when you look, you may see no duplication at all — just a growing pile of state-less pg-backup CRs, which is the telltale.

4. Related defect found while debugging: tracking CRs for scheduled jobs can never progress even when everything else is healthy — the label switch at percona/controller/pgbackup/controller.go (~L264 in v3.0.0) doesn't handle the scheduled value:

ERROR Reconciler error {... "error": "update PGBackup: unknown value for postgres-operator.crunchydata.com/pgbackrest-backup label: scheduled"}

Environment details: operator 3.0.0, PG 18 (percona-distribution-postgresql:18.4-5), pgBackRest 2.58.0-2, dedicated repo host, S3-compatible repo (path-style), repo1-block=y, repo1-bundle=y, archive-async=y, process-max=8, schedules: full 0 5 * * 0,4, incr 20 0,3,9,12,15,18,21 * * *.

postgrescluster-upstream-sanitized.yaml
perconapgcluster-sanitized.yaml

[nmarukovich]

@leandro-clint hey-hey! thank you for your report.
could you please check do you have any cronjobs in namespace where cluster is running.

kubectl -n postgres-ash-vega-prod get cronjob -o yaml

[leandro-clint]

Hi @nmarukovich, thanks for looking into this! Both answers below.

1) Full YAML of the CronJob

kubectl -n postgres-ash-vega-prod get cronjob postgres-ash-vega-prod-repo1-incr -o yaml — untouched, 100% operator-generated (managed-by: percona-postgresql-operator, owned by the PostgresCluster, single generation: 1, we never patch it):

apiVersion: batch/v1
kind: CronJob
metadata:
  annotations:
    kubectl.kubernetes.io/default-container: pgbackrest
  creationTimestamp: "2026-07-27T13:06:55Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: postgres-ash-vega-prod
    app.kubernetes.io/managed-by: percona-postgresql-operator
    app.kubernetes.io/name: percona-postgresql
    app.kubernetes.io/part-of: percona-postgresql
    pgv2.percona.com/version: 3.0.0
    postgres-operator.crunchydata.com/cluster: postgres-ash-vega-prod
    postgres-operator.crunchydata.com/pgbackrest: ""
    postgres-operator.crunchydata.com/pgbackrest-backup: scheduled
    postgres-operator.crunchydata.com/pgbackrest-cronjob: incr
    postgres-operator.crunchydata.com/pgbackrest-repo: repo1
  name: postgres-ash-vega-prod-repo1-incr
  namespace: postgres-ash-vega-prod
  ownerReferences:
  - apiVersion: upstream.pgv2.percona.com/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: PostgresCluster
    name: postgres-ash-vega-prod
    uid: 03770f1f-4d51-420a-a439-d277c86f7bbe
  resourceVersion: "1642116229"
  uid: 5795aa26-1ab8-4c52-b6b0-4bfebfd2f7e4
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    metadata:
      annotations:
        kubectl.kubernetes.io/default-container: pgbackrest
      labels:
        app.kubernetes.io/instance: postgres-ash-vega-prod
        app.kubernetes.io/managed-by: percona-postgresql-operator
        app.kubernetes.io/name: percona-postgresql
        app.kubernetes.io/part-of: percona-postgresql
        pgv2.percona.com/version: 3.0.0
        postgres-operator.crunchydata.com/cluster: postgres-ash-vega-prod
        postgres-operator.crunchydata.com/pgbackrest: ""
        postgres-operator.crunchydata.com/pgbackrest-backup: scheduled
        postgres-operator.crunchydata.com/pgbackrest-cronjob: incr
        postgres-operator.crunchydata.com/pgbackrest-repo: repo1
    spec:
      template:
        metadata:
          annotations:
            kubectl.kubernetes.io/default-container: pgbackrest
          labels:
            app.kubernetes.io/instance: postgres-ash-vega-prod
            app.kubernetes.io/managed-by: percona-postgresql-operator
            app.kubernetes.io/name: percona-postgresql
            app.kubernetes.io/part-of: percona-postgresql
            pgv2.percona.com/version: 3.0.0
            postgres-operator.crunchydata.com/cluster: postgres-ash-vega-prod
            postgres-operator.crunchydata.com/pgbackrest: ""
            postgres-operator.crunchydata.com/pgbackrest-backup: scheduled
            postgres-operator.crunchydata.com/pgbackrest-cronjob: incr
            postgres-operator.crunchydata.com/pgbackrest-repo: repo1
        spec:
          containers:
          - command:
            - /opt/crunchy/bin/pgbackrest
            env:
            - name: COMMAND
              value: backup
            - name: COMMAND_OPTS
              value: --stanza=db --repo=1 --type=incr
            - name: COMPARE_HASH
              value: "true"
            - name: CONTAINER
              value: pgbackrest
            - name: NAMESPACE
              value: postgres-ash-vega-prod
            - name: SELECTOR
              value: postgres-operator.crunchydata.com/cluster=postgres-ash-vega-prod,postgres-operator.crunchydata.com/pgbackrest=,postgres-operator.crunchydata.com/pgbackrest-dedicated=
            image: percona/percona-pgbackrest:2.58.0-2
            imagePullPolicy: Always
            name: pgbackrest
            resources: {}
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop:
                - ALL
              privileged: false
              readOnlyRootFilesystem: true
              runAsNonRoot: true
              seccompProfile:
                type: RuntimeDefault
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /opt/crunchy
              name: crunchy-bin
            - mountPath: /etc/pgbackrest/conf.d
              name: pgbackrest-config
              readOnly: true
          dnsPolicy: ClusterFirst
          enableServiceLinks: false
          initContainers:
          - command:
            - /usr/local/bin/init-entrypoint.sh
            image: docker.io/percona/percona-postgresql-operator:3.0.0
            imagePullPolicy: Always
            name: pgbackrest-init
            resources: {}
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop:
                - ALL
              privileged: false
              readOnlyRootFilesystem: true
              runAsNonRoot: true
              seccompProfile:
                type: RuntimeDefault
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /opt/crunchy
              name: crunchy-bin
          restartPolicy: Never
          schedulerName: default-scheduler
          securityContext:
            fsGroupChangePolicy: OnRootMismatch
          serviceAccount: postgres-ash-vega-prod-pgbackrest
          serviceAccountName: postgres-ash-vega-prod-pgbackrest
          terminationGracePeriodSeconds: 30
          volumes:
          - emptyDir: {}
            name: crunchy-bin
          - name: pgbackrest-config
            projected:
              defaultMode: 420
              sources:
              - secret:
                  name: postgres-ash-vega-prod-external-secret
              - configMap:
                  items:
                  - key: pgbackrest_repo.conf
                    path: pgbackrest_repo.conf
                  - key: config-hash
                    path: config-hash
                  - key: pgbackrest-server.conf
                    path: ~postgres-operator_server.conf
                  name: postgres-ash-vega-prod-pgbackrest-config
              - secret:
                  items:
                  - key: pgbackrest.ca-roots
                    path: ~postgres-operator/tls-ca.crt
                  - key: pgbackrest-client.crt
                    path: ~postgres-operator/client-tls.crt
                  - key: pgbackrest-client.key
                    mode: 384
                    path: ~postgres-operator/client-tls.key
                  name: postgres-ash-vega-prod-pgbackrest
  schedule: 20 0,3,9,12,15,18,21 * * *
  successfulJobsHistoryLimit: 3
  suspend: false
status:
  active:
  - apiVersion: batch/v1
    kind: Job
    name: postgres-ash-vega-prod-repo1-incr-29774360
    namespace: postgres-ash-vega-prod
    resourceVersion: "1642116226"
    uid: 54e6212c-fc93-4f0a-ad35-694fcd6377db
  lastScheduleTime: "2026-08-11T15:20:00Z"

The full CronJob (0 5 * * 0,4) is identical except for --type=full.

2) Other CronJobs in the namespace

Yes — full disclosure, there are two, and both are ours. Neither one creates backups: they only delete/clean up leftover objects, and both exist because of this bug (they are the workarounds described in my first comment). Here is the current list:

NAME                                           SCHEDULE                     SUSPEND   ACTIVE   AGE
pitr-sub-monitor                               */5 * * * *                  False     0        34d
postgres-ash-vega-prod-backup-active-cleaner   */15 * * * *                 False     0        47d
postgres-ash-vega-prod-repo1-full              0 5 * * 0,4                  False     0        15d
postgres-ash-vega-prod-repo1-incr              20 0,3,9,12,15,18,21 * * *   False     1        15d

pitr-sub-monitor — our logical-replication watchdog (unrelated to backups originally). We bolted a "janitor" onto it that runs kubectl delete on stuck pg-backup CRs: (a) the CRs created by the operator's in-process scheduler that have no pgv2.percona.com/pgbackrest-backup-job-name annotation (i.e. the ones that spawn the second, duplicate -backup-xxxxx Job); (b) tracking CRs stuck in Starting whose Job is already gone, which hold the pg-<cluster>-backup-lock lease forever. It is guarded with active == 0 && succeeded+failed >= 1 so it can never touch a running backup. It creates nothing.

postgres-ash-vega-prod-backup-active-cleaner — 20 lines of kubectl, it does exactly one thing: for CronJobs labelled pgbackrest-backup=scheduled, if status.active references a Job that no longer exists, it patches status.active: null. We needed this because the operator re-parents the CronJob's Job (j.OwnerReferences = nil / owner swapped to the pg-backup CR), so when that CR is deleted the Job is garbage-collected while it is still listed in the CronJob's status.active. With concurrencyPolicy: Forbid that stale entry permanently blocks every future scheduled backup — the CronJob just stops firing. Same root cause also explains why successfulJobsHistoryLimit: 3 / failedJobsHistoryLimit: 1 never take effect and finished backup Jobs pile up (19 alive at one point).

Important for your reproduction: the duplicate backup does not depend on either of these CronJobs. The duplicate is the operator's own pgbackrest Job from postgres-ash-vega-prod-repo1-incr-<ts> racing the -backup-xxxxx Job created by the in-process scheduler's pg-backup CR for the same slot, both fighting over db-backup-1.lock (ERROR [050] ... unable to acquire lock, exit 50). Our cronjobs only hide the symptom by deleting the extra CR before its Job starts (~5 min window), so if you reproduce on a clean namespace with no extra cronjobs you should see the collision much more often than we do today.

Current state right now, for reference (15:20 UTC slot in flight): one Job from the CronJob, plus a single tracking CR — which, as noted in my first comment, is stuck in Starting because pgbackrest-backup: scheduled falls through to default: in the pg-backup controller's switch (unknown value for pgbackrest-backup label: scheduled):

NAME                                               STATE      JOB-ANNOT                                    CREATED
postgres-ash-vega-prod-repo1-incr-29774360-t55lc   Starting   postgres-ash-vega-prod-repo1-incr-29774360   2026-08-11T15:20:00Z

Happy to run any other command or to drop the janitor for a few slots if you want to see the unmitigated behaviour.

[nmarukovich]

@leandro-clint could you please also run these two commands:

kubectl -n postgres-ash-vega-prod get postgrescluster postgres-ash-vega-prod -o jsonpath='{range .metadata.managedFields[*]}{.manager}{"  op="}{.operation}{"  time="}{.time}{"\n"}{end}'

kubectl get deploy -A -l app.kubernetes.io/name=percona-postgresql-operator -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image'

[leandro-clint]

@nmarukovich here you go — and your second command turned out to be the right question to ask, because chasing it down changed my understanding of the root cause. Summary first: there is only one operator, and the two upstream CronJobs in this namespace are orphans that operator 3.0.0 created and then abandoned.

1) managedFields on the PostgresCluster

postgrescluster-controller  op=Update  time=2026-07-27T09:16:06Z
postgres-operator           op=Update  time=2026-08-05T21:21:05Z
postgres-operator           op=Update  time=2026-08-11T00:11:42Z
postgrescluster-controller  op=Update  time=2026-08-12T18:20:01Z

Broken down by what each manager actually owns (note -o json strips managedFields unless you pass --show-managed-fields):

manager | owns | latest write -- | -- | -- postgres-operator | f:spec (incl. f:backups, f:instances, f:patroni, f:users, …) + f:metadata | 2026-08-11T00:11:42Z postgres-operator | f:status | 2026-08-05T21:21:05Z postgrescluster-controller | f:status | 2026-08-12T18:20:01Z postgrescluster-controller | f:metadata → only the postgres-operator.crunchydata.com/finalizer | 2026-07-27T09:16:06Z

So: no kubectl, no ArgoCD, no third party writing to the PostgresCluster — only the two controllers that live inside the operator binary. Our GitOps (ArgoCD) applies the PerconaPGCluster (pg) CR instead:

argocd-controller   op=Apply   time=2026-08-11T07:49:51Z   top=f:metadata,f:spec
kubectl-patch       op=Update  time=2026-08-02T00:58:35Z   top=f:spec
postgres-operator   op=Update  time=2026-08-05T21:25:50Z   top=f:metadata,f:spec
postgres-operator   op=Update  time=2026-08-12T18:20:13Z   top=f:status

2) Operator deployments

Your selector returns nothing — the Helm chart labels it app.kubernetes.io/name: pg-operator, not percona-postgresql-operator:

$ kubectl get deploy -A -l app.kubernetes.io/name=percona-postgresql-operator -o custom-columns=...
NS    NAME   IMAGE
(empty)

Searching by image instead:

NS                   NAME                 IMAGE                                                 READY
percona-pg-operator  percona-pg-operator  docker.io/percona/percona-postgresql-operator:3.0.0    1/1

• exactly one Deployment, replicas: 1, and exactly one pod cluster-wide running that image (percona-pg-operator-7df6fdd4bd-8xvb4, started 2026-08-01T22:36:59Z)
• WATCH_NAMESPACE empty (cluster-wide), PGO_CONTROLLER_LEADER_ELECTION_ENABLED=true, and the lease 08db3feb.percona.com is held by that single pod
• both ReplicaSets in the namespace are 3.0.0 (oldest created 2026-06-16) — the operator has been 3.0.0 since well before any of the objects discussed below, so nothing here is a leftover from a 2.x install

Labels, for a future selector: app.kubernetes.io/name: pg-operator, pgv2.percona.com/control-plane: postgres-operator.

3) The actual finding: schedules never reach the PostgresCluster, but the CronJobs live on

This is what I did not know when I opened the issue. In the pg CR the schedules are declared; in the PostgresCluster the operator generates, schedules is empty — and that is true for all three Percona clusters in this Kubernetes:

== postgres-ash-vega-prod/postgres-ash-vega-prod
   pg schedules:              {"full":"0 5 * * 0,4","incremental":"20 0,3,9,12,15,18,21 * * *"}
   PostgresCluster schedules: {}
   cronjobs:                  postgres-ash-vega-prod-repo1-full, postgres-ash-vega-prod-repo1-incr
== postgres-clintplay-ash-vega-prod/postgres-clintplay-ash-vega-prod
   pg schedules:              {"full":"0 4 * * 0,3","incremental":"20 4 * * 1,2,4,5,6"}
   PostgresCluster schedules: {}
   cronjobs:                  <none>
== postgres-sequin-ash-vega-prod/postgres-sequin-ash-vega-prod
   pg schedules:              {"full":"0 5 * * 0,4","incremental":"20 5 * * *"}
   PostgresCluster schedules: {}
   cronjobs:                  <none>

The two clusters with no CronJobs behave exactly as I assume 3.0.0 intends: the in-process scheduler creates one pg-backup CR per slot, it runs, it succeeds, no lock contention at all. E.g. postgres-sequin-ash-vega-prod:

postgres-sequin-ash-vega-prod-repo1-incr-tg849  [--type=incr]  Succeeded  2026-08-09T05:20:35Z
postgres-sequin-ash-vega-prod-repo1-incr-hzr5t  [--type=incr]  Succeeded  2026-08-10T05:20:37Z
postgres-sequin-ash-vega-prod-repo1-incr-br9jv  [--type=incr]  Succeeded  2026-08-11T05:20:33Z
postgres-sequin-ash-vega-prod-repo1-incr-sgpsn  [--type=incr]  Succeeded  2026-08-12T05:20:57Z

Only postgres-ash-vega-prod still has the upstream CronJobs, and they are orphans:

• written by postgrescluster-controller, op=Apply, time=2026-07-27T13:06:54Z/:55Z — exactly once, generation: 1, never touched again (only kube-controller-manager writes f:status since)
• created ~4h after the PostgresCluster itself (postgrescluster-controller put its finalizer on at 09:16:06Z) — that afternoon we were rebuilding this cluster from a dataSource pgBackRest restore
• they carry pgv2.percona.com/version: 3.0.0, so they were produced by 3.0.0
• they have been firing on schedule for 16 days while PostgresCluster.spec.backups.pgbackrest.repos[0].schedules is {} — whatever cleanup exists for de-scheduled repos never removed them

So the "two schedulers" I reported is not two components racing by design — it is one leftover CronJob plus the in-process scheduler. My working hypothesis: some bootstrap/restore path in 3.0.0 propagates spec.backups...schedules into the generated PostgresCluster (long enough for the upstream controller to Apply the CronJobs), while the steady-state reconcile clears it — and nothing ever garbage-collects the CronJobs that the first pass created. That would explain why only the cluster we rebuilt via dataSource has them, and why the two clusters created normally never got any.

Two questions, then:

1. Is schedules: {} on the generated PostgresCluster the intended 3.0.0 behaviour (i.e. scheduling is fully in-process now)? If yes, the bug is simply that the upstream CronJobs are not deleted when schedules is cleared — and any cluster that ever passed through that window keeps double-executing forever.
2. If the answer to (1) is yes, can I just kubectl delete cronjob postgres-ash-vega-prod-repo1-{incr,full} and expect the operator not to recreate them? That would leave the in-process scheduler as the single executor, matching the other two clusters. Happy to do it on your word and report back.

If you want a repro to aim at: create a cluster with dataSource (pgBackRest restore) and repos[].schedules set, then check whether PostgresCluster ... repos[0].schedules ends up {} while the *-repo1-incr CronJob still exists.

[nmarukovich]

@leandro-clint
I tried to reproduce it.

1. dataSource restore alone does not reproduce it. Operator 3.0.0: source cluster → full backup to S3 → delete cluster → new cluster with dataSource.pgbackrest and repos[].schedules set, no CronJob was created. Restore and scheduled backups worked fine.

2. Writing schedules directly into the PostgresCluster does reproduce it. We ran kubectl patch postgrescluster ... --type=merge and added schedules. What happened:
   the upstream controller created the *-repo1-incr CronJob (owned by the current PostgresCluster, label pgv2.percona.com/version: 3.0.0, generation: 1 — same as yours);
   then the operator removed schedules again, back to {};
   the CronJob was never deleted
   within one schedule slot we got the same failure chain you reported: tracking CR stuck in Starting, the backup lease held, state-less CRs piling up.

Note about your managedFields check: after the operator removed schedules, our kubectl-patch entry disappeared from managedFields too. Ownership is only kept while the field exists. So our object also shows only the two controllers now — same as yours. Meaning: "no kubectl / no ArgoCD in managedFields" does not prove nobody wrote there.

Your questions:

1. Yes, schedules: {} on the PostgresCluster is intended — scheduling is fully in-process since v2.3.1 (K8SPG-493). Your two other clusters show the intended behaviour.
2. Yes, delete both CronJobs — the operator will not recreate them (verified on 3.0.0 and main). Also delete any pg-backup CR stuck in Starting (it holds the lease) and Jobs with the keep-job finalizer.

About the origin: something wrote schedules into the PostgresCluster (not the pg CR) around 2026-07-27 13:06. The objects can't tell who — the trace erases itself, as shown above.