exportServices/SSL cleanup should ignore Delete NotFound after cached read

[ChaoHuang2018]

Hi Percona MongoDB Operator maintainers,

I noticed three remaining cleanup delete paths where a cached read/list can be followed by a live Delete that returns NotFound.

I saw #1751 and the merged fix PR #2389, which addressed repeated redundant delete calls by introducing DeleteIfExists in several cleanup paths. This issue is narrower: in the paths below, the controller has already read or listed a target object, but if that object is gone by the time the delete reaches the API server, the delete-side NotFound is still returned as a cleanup error.

In current main, exportServices lists ServiceExport objects and deletes each one when multicluster support is disabled:

pkg/controller/perconaservermongodb/service.go:130-L147

This branch returns any delete error. A later stale ServiceExport cleanup branch in the same function already ignores k8serrors.IsNotFound(err):

pkg/controller/perconaservermongodb/service.go:173-L176

The SSL certificate rotation path has the same delete-result shape. It gets a Secret, compares CA data, then deletes the Secret when it should be recreated:

pkg/controller/perconaservermongodb/ssl.go:232-L256

The old-CA cleanup path also gets an old Secret and deletes it when the merged CA is already present:

pkg/controller/perconaservermongodb/ssl.go:365-L389

A possible interleaving is:

a cleanup path reads or lists an object that should be deleted
the object is deleted by a previous reconcile attempt or another controller action
the cache has not observed that deletion yet
the cleanup path still calls Delete(ctx, object)
Delete returns NotFound from the API server
the cleanup path returns an error even though the object is already absent

For these paths, absence of the target object seems to be the desired cleanup state. Would it be reasonable to use client.IgnoreNotFound(err) or an explicit k8serrors.IsNotFound(err) check around these delete calls, while still reporting other delete failures?

Chao

[ChaoHuang2018]

I noticed two additional cleanup paths in the same operator that appear to have the same delete-result shape as this issue.

1. deleteSecrets first gets each Secret and skips the path if the Secret is missing, then deletes the fetched Secret:

pkg/controller/perconaservermongodb/finalizers.go:312-L330

If the Secret is deleted after the get and before the delete reaches the API server, Delete can return NotFound.

2. deleteConfigMapIfExists similarly gets the ConfigMap, returns on get-side NotFound, checks ownership, then returns the raw delete result:

pkg/controller/perconaservermongodb/psmdb_controller.go:1401-L1417

If this issue is accepted, these two call sites may be worth checking at the same time.

[egegunes]

Yes, ignoring NotFound errors in these paths makes sense. I created https://perconadev.atlassian.net/browse/K8SPSMDB-1798 for this.

[ChaoHuang2018]

Thanks for working on this. I agree that the service.go branch may be unreachable.
One scope check: my follow-up comment also identified deleteSecrets in finalizers.go and deleteConfigMapIfExists in psmdb_controller.go. #2533 currently changes only the two ssl.go sites. Should the remaining two same-shape sites be included in K8SPSMDB-1798 or tracked separately?