Avoid creating CatalogSource pods on every poll even when catalog hasn't changed #3733
Description
Bug Report
What did you do?
Observed that OLMv0 creates a CatalogSource pod on every polling interval, even when the catalog image has not changed. This happens as part of the normal catalog upgrade polling loop.
What did you expect to see?
Ideally, OLM should be determining whether the catalog image has actually changed before creating a CatalogSource pod, avoiding unnecessary API calls to kube api server and etcd writes when nothing has changed.
What did you see instead? Under which circumstances?
On every polling interval (historically 10 minutes, 15 minutes by default), OLMv0 performs the following:
- Pull the CatalogSource image
- Create a CatalogSource pod
- Compare it with the existing catalog pod using a pod hash
- Switch traffic if a real update is detected
Even when no update exists, OLMv0 still performs:
- GET ServiceAccount
- GET NetworkPolicy (x2)
- GET Service
- LIST Pods
- CREATE Pod (writes to etcd)
- Wait for the Pod to become Ready
- DELETE Pod (writes to etcd)
- PATCH CatalogSource/status (writes to etcd)
With 4 default CatalogSources, this results in approximately:
- 72 etcd writes per hour
- 1,728 etcd writes per day
Possible Solution
A potential improvement would be to query the container registry (e.g., via manifest digest) before creating a CatalogSource pod and only proceed with pod creation if the image has changed.
Additional context
PR in operator-marketplace for the short term workaround that increases the default catalog polling interval to 4h to improve performance: operator-framework/operator-marketplace#695