Determining if package cache was used

[dominiquekwiek]

It seems that there is no way to determine if the package cache was used to resolve packages from. Repro:

[16:32:26][dkwiek] ~/scratch $> rez-env rez
You are now in a rez-configured environment.

resolved by dkwiek@ip-10-255-208-120.ec2.internal, on Fri Apr 07 16:32:28 2023, using Rez v2.88.4

requested packages:
rez               
~platform==linux  (implicit)
~arch==x86_64     (implicit)
~os==CentOS-7.9   (implicit)

resolved packages:
arch-x86_64     /tmp/rez/cache/rez-pkg-cache/arch/x86_64/761a/a     (cached)
os-CentOS-7.9   /tmp/rez/cache/rez-pkg-cache/os/CentOS-7.9/3875/a   (cached)
platform-linux  /tmp/rez/cache/rez-pkg-cache/platform/linux/36e8/a  (cached)
python-3.7.7    /tmp/rez/cache/rez-pkg-cache/python/3.7.7/c2b8/a    (cached)
rez-2.88.4      /tmp/rez/cache/rez-pkg-cache/rez/2.88.4/4c68/a      (cached)
> [16:32:29][dkwiek] ~/scratch $> python -c "from rez.resolved_context import ResolvedContext; ctx = ResolvedContext.get_current(); print(ctx.resolved_packages)"
[Variant(FileSystemVariantResource({'index': None, 'location': '/software/rez/packages', 'name': 'platform', 'repository_type': 'filesystem', 'version': 'linux'})), 
Variant(FileSystemVariantResource({'index': None, 'location': '/software/rez/packages', 'name': 'arch', 'repository_type': 'filesystem', 'version': 'x86_64'})), 
Variant(FileSystemVariantResource({'index': None, 'location': '/software/rez/packages', 'name': 'os', 'repository_type': 'filesystem', 'version': 'CentOS-7.9'})), 
Variant(FileSystemVariantResource({'index': 0, 'location': '/software/rez/packages', 'name': 'python', 'repository_type': 'filesystem', 'version': '3.7.7'})), 
Variant(FileSystemVariantResource({'index': 0, 'location': '/software/rez/packages', 'name': 'rez', 'repository_type': 'filesystem', 'version': '2.88.4'}))]

The ResolvedContext is obfuscating the fact it is being loaded from the cache and there appears to be no method available on the ResolvedContext in order to determine the cache was used. The package_cache_present seems to return True if the caching directory exists (even if it is empty). It seems the only way is to inspect the environment for the context for the existence of _ORIG_ROOT env vars. Is there any other way to determine this?

[JeanChristopheMorinPerso]

You can use the rez-pkg-cache command line tool to query the state of the cache. Depending on your use case, that might be enough.

Alternatively, you can do something like

import rez.config
import rez.package_cache

cache = rez.package_cache.PackageCache(rez.config.config.cache_packages_path)
cached_variants = cache.get_variants()

In both cases, you'll have to manually iterate your resolved packages (variants) and check if they are in the cache.

If you tell us more about what you are trying to achieve, we can see if we could improve the situation a little bit. For example, maybe we could add a cached_resolved_packages property on the ResolvedContext class.

[dominiquekwiek]

Thanks for the response. Basically we want to launch a job on a different box and propagate the Rez environment. We are creating our package cache in the \tmp directory on our boxes which obviously cannot be seen by another box. For now, I can just force a re-evaluation without package caching i.e.

ctx = ResolvedContext.get_current()
ctx_dict = ctx.to_dict()
# Turn off package caching
ctx_dict["package_caching"] = False
ctx_non_cached = ResolvedContext.from_dict(ctx_dict)
rez_env_vars = ctx_non_cached.get_environ()

It would be nicer to only regenerate the resolved context if anything is cached.

[JeanChristopheMorinPerso]

What do you mean by "propagate the Rez environment"? Also, how do you generate your job environment? From what I see in your last snippet, you seem to take all the environment variables from the context and inject that into the jobs?

[dominiquekwiek]

We are passing the environment along to a render farm. We therefore need to ensure no cached package paths remain in the env vars.

[JeanChristopheMorinPerso]

Um ok, that's not really supported per say. We don't recommend copying the whole environment like that anywhere. We generally recommend to pass the context (the content of the file set in $REZ_RXT_FILE) or to pass the whole list of resolved packages (fully baked, i.e replace - with ==, etc). In both cases, you let the job in the render farm re-do a resolve, but it'll be guaranteed to resolve exactly the same set of packages.

[dominiquekwiek]

Yeah for sure. This is an interim measure before we handle re-resolving the environment in the pre-job hook.

[JeanChristopheMorinPerso]

Alright, thanks for the additional info. I'll re-label the issue as an enhancement. I suggest to add a new property on the ResolvedContext class to get the packages that were cached. Because I don't think we want to add an attribute on the objects returned by ResolvedContext.resolved_packages. but maybe I'm wrong. Everybody is welcome to suggest something else too.