Discussion on HLS cache management.

[vidit-od]

HLS cache limits have been a concern for some time. we are always close to our 10GB limit. This made me dig deeper into our cache eviction policies. Below is my documentation for it and blockers that needs more opinions.

some ways to improve cache utilization :

1. Evict cache entries for PRs that are merged / closed ( attempted here )
2. Evict old cache entries when new cache entries are created.

These are only some of the ideas, welcoming more ways ;

Blockers :

• PR Fix cache clearing job #4868 and PR Migrate stackage build on GHA #4865 introduced cache cleaning when a PR is closed / merged but currently Github workflows do not have access to write.

• To begin working on any of the cache eviction we first need to give write access to our workflows.
• will this / could this create any unwanted side effects later is a point to discuss.

[fendor]

To begin working on any of the cache eviction we first need to give write access to our workflows.

What does this entail? Could this be dangerous as workflows might be executing user provided code with elevated privileges?

---

It might also be helpful to specify our current caching strategy. In theory, according to cache.yml, we run a job nightly to obtain caches for cabal dependencies. How much is that for our current matrix of jobs? Do 10 GB suffice in principle?

Are we generating caches for PRs? Is that a big time save?
It is when adding new GHC versions or updating dependencies, since then you have complete cache miss. How much of the cache right now is caused by these PR caches?

Are there other ways to improve our caching situation? Some time ago, cabal-cache and a HLS specific S3 instance was discussed, perhaps this something to ponder on?

[vidit-od]

What does this entail? Could this be dangerous as workflows might be executing user provided code with elevated privileges?

according to rferences 1 and 2 what we are doing should be fine. we do provide permissions : actions : write; This should ideally work, i am not sure what is blocking permissions right now. will have to create a local repo and experiment there;

this will makes sure our workflow has only cache clearing permissions ; also this will only trigger on merge / close.

Threat :
if what we are doing somehow works, we still have 1 problem. our workflow triggers on close / merge of a PR. i.e someone can change the workflow commands to clear all cache entries. open a PR. then close it. This could clear our cache right ?

if so, ideally we should do cache clearing only on merge.

It might also be helpful to specify our current caching strategy. In theory, according to cache.yml, we run a job nightly to obtain caches for cabal dependencies. How much is that for our current matrix of jobs? Do 10 GB suffice in principle?

Are we generating caches for PRs? Is that a big time save?

Reference 3 is a good doc to understand our current caching strategy.

we refresh every 8 hours to makes sure master caches is present. we do generate caches for PRs. not sure how much time they save. Reference doc of caching PRs : 4

How much of the cache right now is caused by these PR caches?

in theory we should not worry about PR caches evicting master caches. every time a PR is created we use master cache (making them hot), then build PR cache and save it. on average ( from observation ) i have seen aprox 30 entries reside in our caches. out of this 12 -15 entries are for PRs.

Once a PR cache is created, PR will use its PR caches. on successive commits. more PR cache entries pileup. i.e only way to evict master cache would be if a few Prs successively creates caches. But the 8 hourly refresh makes sure to reintroduce master caches again.

Are there other ways to improve our caching situation? Some time ago, cabal-cache and a HLS specific S3 instance was discussed, perhaps this something to ponder on?

i dont have much experience with s3 but seems interesting. will look into it :) any other references for this (other open issues / discussions) from which i can get more context ?