unexpected growth in files under reticulate/uv/cache/archive-v0

[vjcitn]

This may all be working as intended. I have been using the pattern

library(reticulate)
py_require([pkg])  # often not pinning the version
p = import([pkg])

In ~/.cache/R/reticulate/uv/cache/archive-v0 a new uniquely named folder is
created whenever the py_require spec is not satisfied. Speculation: if any
upstream dependency of the required python package has been updated, a
new folder in archive-v0 will be produced to store the updated dependency. (I believe I've
seen doc on a retention policy for these cache folders.) This process would
occur even if the version of the required package was pinned, if its dependencies
were not explicitly identified and pinned.

Through a relatively brief period of development I have seen the number of
folders under cache/archive-v0 exceed 100. Should there be some guidance on
pruning this area? Maybe an option to py_config that if set would add some information
on this to the report.

I am aware of the best practices doc
that suggests avoiding restrictive version constraints. If the speculation above is correct, the
implication for monitoring/pruning the cache might be worthy of documentation.

[t-kalinowski]

I did a bit of a deep dive on this a few months ago in #1821 and #1828. The short answer is that this is the expected behavior of uv: the uv cache grows unbounded (at least with how we are using it to stand up ephemeral venvs).

The solution we implemented in reticulate is to automatically clear the full reticulate cache directory at a configurable interval, defaulting to every 120 days. This wipes out all downloaded Python packages, and all the ephemeral venvs (which are mostly thin lightweight symlinks). It also removes downloaded Python binaries, helping clear out versions that are generally unused (e.g., prior patch versions). Finally, it also updates uv itself.
You can manually clear the cache yourself at anytime with

unlink(tools::R_user_dir("reticulate", "cache"), recursive = TRUE)

or change the update interval by placing this in ~/.Rprofile:

options(reticulate.max_cache_age = as.difftime(30, units = "days"))

Some more detail here:
https://rstudio.github.io/reticulate/reference/py_require.html#clearing-the-cache

[vjcitn]

Thanks, I appreciate the details. Can close if appropriate. I noticed in the referenced issue that you were looking for user feedback. All the facts are clear enough. I have found that clearing the cache, even though one hopes most of the content consists of lightweight symlinks, even though it accumulated over a period of far less than 120 days, was a nontrivial process. Perhaps I was working on an non-performant file system, but getting it fully cleared took quite some time. Maybe this is a niche case, maybe folks who are doing a lot of interop programming need to create tools to give more visibility and economy in this space. (The session-info package in python becomes a significant tool for managing this kind of interop, and I would propose that its functionality becomes a first-class element of the reticulate ecosystem.)

[t-kalinowski]

I sympathize. Hitting an unexpected one- or two-minute pause can be surprising, even a bit jarring.

The broader context is that uv’s cache grows without bound, so some cleanup mechanism is necessary. Of the available options, an infrequent automatic reset (roughly every four months by default) concentrates that cost into a rare event rather than introducing ongoing overhead or requiring manual management. I think that, given the alternatives, this design introduces the smallest overall disruption to users’ time and attention.

[t-kalinowski]

It would be good to update the message shown when this happens, to set expectations like Clearing cache. This happens automatically every 4 months. It can take a few minutes...

[eddelbuettel]

Another thought: You could also add a helper function name to the displayed text as a hint that users could cron-job this if the unexpected delay poses an issue "in production" / "normal use".