[1.23.0] delete-backup finalizer still fails with x509 "unknown authority" — DeletePITRChunks missing the MinIO caBundle/skip-verify handling

[raphideb]

Report

We have our own Root CA and the certificate is deployed in the Pods and also in the Operator Namespace. However, even after update to 1.23 (which apparently fixed some of the issues by setting skipverify), our logs still get filled.

psmdb-backup Setup
backup.storages.backup-storage:
type: minio
minio:
endpointUrl: :8443
secure: true
caBundle:
name:
key: ca.crt

More about the problem

** Logs: **
deleting/expiring backups loops forever, backup objects stay stuck on the
percona.com/delete-backup finalizer:

ERROR failed to run finalizer ... finalizer: percona.com/delete-backup
error: failed to delete PITR: delete pitr chunk 'pbmPitr/rs0/.../....oplog.gz' from storage:
list with parts for mw delete op: ... get using minio:
Get "https:////?location=": tls: failed to verify certificate:
x509: certificate signed by unknown authority
...backup.(*pbmC).DeletePITRChunks (pkg/psmdb/backup/pbm.go:1563)
...perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).deleteBackupFinalizer (psmdb_backup_controller.go:817)

We upgraded 1.22.0 -> 1.23.0 specifically because 1.23.0 added handling for this
(operator pod has no CA bundle mounted, so it falls back to InsecureSkipTLSVerify when
minio.caBundle is set). The error is unchanged.

** Analysis: **

in 1.23.0 (and current main) the operator creates object-storage clients in four
places, but the MinIO workaround was only added to some of them:

pkg/psmdb/backup/pbm.go:1353 deleteBackupImpl -> InsecureSkipTLSVerify set OK
pkg/psmdb/backup/pbm.go:1547 DeletePITRChunks -> NOT set BUG?
pkg/psmdb/backup/pbm.go:1398 deleteIncremetalChainImpl -> NOT set BUG?
pkg/psmdb/backup/pbm.go:1022 ValidateBackupInStorage -> set by restore validate.go OK

DeletePITRChunks takes cfg.Storage from b.GetConfig(ctx) and hands it to
util.StorageFromConfig unmodified, so the MinIO client uses the system trust store and
fails on our private CA. The same applies to incremental-chain deletion.
The workaround in psmdb_backup_controller.go getPBMStorage() does not help here,
because deleteBackupFinalizer calls b.PBM().DeletePITRChunks(), not getPBMStorage().

** Solutions **

• apply the same handling to DeletePITRChunks and deleteIncremetalChainImpl
• or much more preferred: have the operator actually read the caBundle secret and pass the CA to the
  storage client (mio.Config has no CA/RootCAs option today) instead of disabling verification

Steps to reproduce

1. Use SSL Certificates signed by a private CA and have the ca.cert deployed as a secret

Versions

1. Kubernetes (Vanilla)
2. Operator 1.23 (helm)
3. Database MongoDB 8.0.17-6

Anything else?

No response