Caching dependencies

[panglesd]

The README says:

This action intentionally does not cache the results of opam install . --deps-only. Unlike package managers such as npm or Cargo, opam does not use a lock file by default — dependency versions are resolved against the current state of opam-repository at install time.

If these resolved dependencies were cached, opam-repository updates (bug fixes, security patches, new package versions) would not be picked up for as long as the cache remains valid. On active repositories where CI runs frequently, the cache would be hit continuously and never expire, effectively freezing dependencies indefinitely. This would make CI unreliable, as it could pass with stale dependencies whilst failing on a fresh install.

I don't understand why not cache the result of opam install . --deps-only, and then do something along the lines of:

$ opam install . --deps-only --depext-only  # to install depexts (they are not cached) 
$ opam upgrade                              # To ensure installed packages are up-to-date
$ opam install . --deps-only                # Just in case a *.opam file changed

This way, the last two command will say "nothing to do" when they are up to date, and upgrade only the packages needed otherwise, and we escape the dependency freeze.

Am I missing something?

[smorimoto]

Thank you for the suggestion. We have considered this approach, but believe it would be difficult to adopt for several reasons.

The biggest concern is cache size. The current cache contains only the opam root and the switch (compiler). If we were to include the results of opam install . --deps-only in the cache, all compiled dependencies would be stored in _opam, causing the cache size to increase substantially in proportion to the number of project dependencies. GitHub Actions imposes a 10 GB per-repository cache limit, and with matrix builds (multiple OCaml versions × multiple operating systems), this limit would be reached very quickly.

The speed improvement would also be limited. opam upgrade still runs the solver and recompiles packages as needed, so there is unlikely to be a significant speed difference compared with running opam install . --deps-only from scratch. When all dependencies are already up to date, both commands effectively result in "nothing to do".

Furthermore, opam upgrade includes the compiler itself as an upgrade candidate. Whilst switch invariants provide some protection, this action is commonly used with a matrix strategy that specifies particular OCaml versions for testing, and there is a risk of unintended compiler upgrades.

[panglesd]

Thanks for the answer. However:

How did you come up with such list of problems, that are completely orthogonal to what the README states as a reason for not caching deps?

The speed improvement would also be limited. opam upgrade still runs the solver and recompiles packages as needed, so there is unlikely to be a significant speed difference compared with running opam install . --deps-only from scratch. When all dependencies are already up to date, both commands effectively result in "nothing to do".

Without cache, all dependencies are not up to date, so I don't see the point of this last sentence. If there are many dependencies to install, and the dependencies had no new version (which is often the case when several commits are made in a row), then the speed improvement is (very) big.

Furthermore, opam upgrade includes the compiler itself as an upgrade candidate.

As you said, the compiler version is in a switch invariant, so this is an absence of a problem.

The biggest concern is cache size.

I agree that this needs handling. I don't understand though why not offer the possibility to cache or not, for instance through an input? Not everyone has such a big matrix that it reaches the quota. setup-dune offers to cache the whole dune cache.