MintMaker: Deploy Redis for Renovate cache #6085
Conversation
|
Skipping CI for Draft Pull Request. |
querti
force-pushed
the
mintmaker-redis-cache
branch
from
ef7075c to
e22bfec
Compare
|
/retest |
|
This looks great! My only question is whether you tested it on some repos (especially Go based) and if you have any numbers/observations if this cache reduces the amount of HTTP requests and how much. |
I tried checking how the x-ratelimit-used header number changes based on whether the cache is used and whether the repos have already been cached. The results have been pretty odd so far - many DB records get crated in the cache, for a file like https://github.com/querti/renovate-tests/blob/master/go.mod as many as 250. On subsequent runs, I can observe cache hits, so the cache is functioning properly. But this doesn't correspond to the values of x-ratelimit-used, which are 6-20 regardless of whether cache is even used. This makes me think that Go updates don't use github API at all and instead query Github directly, which doesn't count toward the API limit. The limits being hit may simply come from regular git operations - many PRs being created and managed by renovate. Enabling go exacerbated this situation since this manager usually creates many PRs. Rate limit graph on the Mintmaker dashboard sort-of suggests it as well, rate limits are all over the place even on a non-go day. This is only the global cache, so enabling Redis for repository cache should hopefully improve the rate-limiting issue. I think that following this approach is still the best path forward, even if the results haven't been very promising so far. |
|
/retest |
|
Great, thanks for the comment. There might be many factors, Renovate uses GraphQL for some GitHub API operations, but REST for some other (I think), and AFAIK we also don't know how the rate limit is reset on GitHub's side, but if there are hits on the cache, that's a good sign imo. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: querti, staticf0x The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest |
querti
force-pushed
the
mintmaker-redis-cache
branch
6 times, most recently
from
51bf3c7 to
8fbc68c
Compare
Redis will be used by renovate PLRs to store and retrieve cache in order to improve performance and decrease the number of API calls. Connection is not password protected, but a network policy exists to restrict access only to pods in the "mintmaker" namespace. I think that a password is unnecessary since it would complicate the deployment and no sensitive or important data will be stored in the database. The persistent volume exists to back up the data so that Redis can reload it after restart. The memory limit was chosen somewhat arbitrarily, but we can increase it if high use is observed over time. OOM errors should not happen since Redis limit is configured lower than Openshift limit.
querti
force-pushed
the
mintmaker-redis-cache
branch
from
8fbc68c to
a3f9862
Compare
|
/test all |
2 similar comments
|
/test all |
|
/test all |
|
/retest |
|
@querti: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Redis will be used by renovate PLRs to store and retrieve cache in order to improve performance and decrease the number of API calls. Connection is not password protected, but a network policy exists to restrict access only to pods in the "mintmaker" namespace. I think that a password is unnecessary since it would complicate the deployment and no sensitive or important data will be stored in the database. The persistent volume exists to back up the data so that Redis can reload it after restart.
The memory limit was chosen somewhat arbitrarily, but we can increase it if high use is observed over time. OOM errors should not happen since Redis limit is configured lower than Openshift limit.