ci: remove cache steps in nix jobs

[Alizter]

Change

We remove the cache steps in the nix jobs in the CI. We replace them with a cachix cache hosted at https://ocaml-dune.cachix.org.

The flake is configured so that it consults this cache by default. This should speed up setting up local development too.

I've added instructions on how to warm nix cache which has to happen manually between flake updates.

Motivation

The main motivation for this change was to give some relief to the GitHub actions cache which was evicting itself often leading to CI times suffering.

Notes

We are currently caching a subset of all the nix jobs, I haven't for instance added the full oxcaml nix job because we override dune with the current scope which invalidates cache. If we can work out how to do this without, it should be no problem to add.

I have not cached anything for macos, because I don't currently know how to do this.

p.s.

I can give permissions to push to this cache as needed from other maintainers.

cc @shonfeder @anmonteiro @rgrinberg

[rgrinberg]

Manually pushing to this cache does not seem like an attractive option. Can't we push at then of a successful CI run?

[Alizter]

@rgrinberg We only have to do this every time we update the nix flake. The dune that we build here doesn't affect the packages we pull in (that's how we have it set up currently).

We could push every PR but I would have to add some secrets to this repo in order to allow it. Most of the time nothing changes, so they would be noops. When somebody updates a flake, it would invalidate the cache.

Cachix has a 5GB limit at the moment in the free tier, our nix jobs should fit nicely in that for now since there is a lot of sharing. When the limit is reached they will evict LRU.

[shonfeder]

We only have to do this every time we update the nix flake.

Could we have a CI job that is only run when the flake file changes that pushes updates to cache?

[rgrinberg]

There's a github secrets thing to prevent the key from being committed.

I am hoping that this push step would be a no-op when the flake hasn't changed anyway.