fix: add caching to Entitlement

[dattito]

Fixes #256.

Because a lot of Entitlement CRs in a cluster lead to a burst of API requests against the BTP server, this PR introduces a chaching layer that prevents to many GET calls to the BTP server. The cache invalidation timeout is 10 seconds.

Some modifications, like Create, Update or Delete, will automatically invalidate the cache, because we can be sure that the external state has changed.

[maximilianbraun]

additionally to my comment:

The final implementation (we should validate how well the approach works in real life, beforehand) should expose metrics (https://book.kubebuilder.io/reference/metrics.html), ie. how the cache hit / miss rate is. This would anyway be interesting for evaluation in real life and to fine tune parameters. The Library is exposing different metrics already. Maybe logging them every 30s might already help for validation.

I wonder on the other side how its about syncronisation. The cache might only be effective if consecuritve runs wait for each other, the entitlement get call is quite expensive (terms of size) and might take some time to transport.

I think the loader can help with that. https://maypok86.github.io/otter/user-guide/v2/examples/#loading

[dattito]

Thank you for your feedback. All changes are incorporated!

The new Prometheus metrics were not tested, as I first would have to build the setup for this. I can however do this if required.

[jtucci]

Hey @dattito , great work on this PR! I noticed it also fixes the bug reported in #597 (entitlement quota being transiently zeroed when deleting one of multiple CRs).

Since the caching feature is getting some review feedback, would you consider splitting this into two PRs, one with just the bug fix so it can be fast-tracked, and one for the caching layer? That way the bug fix can land sooner without being blocked by the caching discussion. Alternatively, if @maximilianbraun is not expecting much more work on this PR, is there a chance it could be merged soon as-is? Happy to help if useful!

[dattito]

just rebased on main and fixed merge conflicts.