Allow enable/disable package cache for specific directories.

[hughetop]

Caching packages locally is an incredible way to increase performance that we use all the time, but it would be great if we could mark certain directories as uncachable (or do the opposite and provide a list of cacheable directories). The reason is that we have a special location for "test" packages that can change frequently and we don't want those to be cached. Often times we forget to add cachable = False to the package.py, install to the test location, test it, make an update, install again, but then we don't see the change because the package has been cached and the version hasn't changed.

Motivation
Having the ability to allow or disallow certain directories from being cached would help simplify the development and testing workflow.

[maxnbk]

Does this satisfy the requirements?
https://github.com/nerdvegas/rez/wiki/Configuring-Rez#default_cachable

[hughetop]

That is set to True for our rez config because most of the time we want packages to cache. What I'm trying out now is setting the 'cachable' attribute in package.py depending on the root. I tried

@late()
def cachable():
    return 'Rez/packages/test' not in this.root

but that didn't seem to work, it still cached when the package was within the test package directory. So I'm not quite sure how I can conditionally set a package's 'cachable' attribute correctly.

Edit: I updated the example to show what was not working before. 1) wrong slashes on windows, and 2) this.root is all lowercase

[maxnbk]

Sorry, I wasn't clear, I was also pointing you to the variables below that one, which are to be set in a rezconfig, allowing you to control per-repository defaults, or per-package defaults, independent of the rez package.py

[hughetop]

Right, but aside from a few specific packages, I want most to cache by default. I'm already handling those special cases in their own package.py files, I just want to prevent certain builds from caching. Like if I do

rez-build --install --prefix=/some/test/area

I want to prevent anything in that directory from caching.

[maxnbk]

Hmm. For some reason I thought caching was disabled during rez-build in general...

[hughetop]

Caching happens after the resolve. So if I do

rez-env --paths=/some/test/area package1

it'll pick up my test package and then start a subprocess to cache it locally. Subsequent rez-env calls will then find that package in the cache and use that instead. This is where the issue comes. If I make a change and run

rez-build --install --prefix=/some/test/area

again, it will update the package in /some/test/area, but the cache will not update. The cache will now be stale.

[nerdvegas]

I think we're talking about two slightly different things here: 1. local installs should not use the cache for the reason Topher shows above (absolutely agree with this) 2. inability to default packages to non-caching for a specific pkg repo. Wrt (2), see: https://github.com/nerdvegas/rez/wiki/Configuring-Rez#default_cachable_per_repository Specifically: """Set cachable on a per-package-repository basis. Overrides 'default_cachable_per_package' and 'default_cachable' for any repos listed here.""" So you should be able to set default_cachable=True, _and_ use default_cachable_per_repository to disable caching on a specific repo, at the same time. Is that not working? If (1) is indeed happening with default_cachable=True, then perhaps we simply disable pkg caching on local packages completely (I can't imagine why you would ever want this), and update https://github.com/nerdvegas/rez/wiki/Managing-Packages#package-caching appropriately. But I would like to know if my suggestion works anyway - we need to file a bug if it doesn't. Thx A

…

On Fri, Apr 8, 2022 at 10:23 AM Topher ***@***.***> wrote: Caching happens after the resolve. So if I do rez-env --paths=/some/test/area package1 it'll pick up my test package and *then* start a subprocess to cache it locally. Subsequent rez-env calls will then find that package in the cache and use that instead. This is where the issue comes. If I make a change and run rez-build --install --prefix=/some/test/area again, it will update the package in /some/test/area, but the cache will not update. The cache will now be stale. — Reply to this email directly, view it on GitHub <#1281 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSQM2NCCYHS6QM6KDQTVD537JANCNFSM5S2UOK5Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[hughetop]

Sorry, I'm not explaining this very well. We want the majority of packages to be cachable, so our rezconfig has default_cachable = True. There are a few packages that we never want cached, and we explicitly set cachable = False in their respective package.py files.

The workflow for testing locally is simple, just rez-build --install and it goes to my home directory. But if I want someone else to test my changes, like an artist, I'll build to a network location like rez-build --install --prefix=/some/network/test/area. They will get an updated "launcher script" that will explicitly use that directory in the --paths argument like:
rez-env --paths=/some/network/test/area:~/packages:/some/network/path/int:/some/network/path/ext package1 package2

This works well as it allows anyone to test the changes, but the issue is when I need to make an update. Once an artist has gone into a rez environment with that test path, it will automatically start to cache the packages. So when I update the package and then run rez-build --install --prefix=/some/network/test/area again, the test build on the network gets updated, but the artist does not get the update because the package has been cached. We can sort of get around this by setting cachable = False in the package.py before building to the network test area, but we sometimes forget and it causes a lot of confusion. (And sometimes we forget to remove it before we do rez-release and that package version never caches).

So the feature request has to do with this testing workflow: being able to restrict certain directories from caching, or only allowing specific directories to be cached.

[hughetop]

Update: I have found a way to conditionally prevent caching. Adding this to the package.py

@late()
def cachable():
    import os
    return os.path.join('rez', 'packages', 'test') not in this.base.lower()

I tried something similar before, but it didn't work because 1) I was using the wrong slashes on windows, and 2) the this.base was all lowercase and there was 1 directory that was capitalized. So I have a workaround, but ideally there would be a way within rez to prevent certain directories from caching.

[ColinKennedy]

A bit late to the discussion. I see certain parallels between what's being requested here and what is written in REP-003 - Package Immutability. Maybe there's a way to solve this problem using that as a reference?

From what I read so far, the only packages which @hughetop does not want to cache are the packages related to testing, which are built somewhere on a network. That link I posted above addresses this as well. "One point to note is that caching is only useful if we can assume that the contents of a package do not change....The case where they cannot is local packages, which in practice get re-installed over all the time."

(Note: I'm making this syntax / functionality up) what if you could do this

cd thing_you_dont_want_cached
rez-build --clean --install --prefix /some/network/root/installed_packages
rez-env foo-1 bar-2 --output base.rxt  # everything that you want cached
rez-package-cache --populate base.rxt /some/network/root/cache  # the "/cache" folder has the base request now
rez-env foo bar-1 thing_you_dont_want_cached-3 --package-cache /some/network/root/cache --output /some/network/root/user_tester.rxt

Now you have a "user_tester.rxt", which, I assume, gets cache requests for everything but thing_you_dont_want_cached-3. So you can continually rez-build thing_you_dont_want_cached and make new versions, the other requests are cached, and all the end user needs to do is keep calling the same command, rez-env -i /some/network/root/user_tester.rxt (whether via the terminal or in a GUI or something) over and over to get the updates.

Admittedly that's a lot more code than having a simple rez-config list-of-str paths to prevent caching. But worth mentioning that there's potential overlap with past topics.

[hughetop]

"One point to note is that caching is only useful if we can assume that the contents of a package do not change....The case where they cannot is local packages, which in practice get re-installed over all the time."
Yeah, basically I would want to extend that functionality to a list of directories. Setting the cachable attribute to a function that determines if it's within the test area is working so far, but it's another thing we need to remember to remember.

[nerdvegas]

Hi Topher, Ok cool I understand the situation you have. But again, can I check that you've tried this, I need to know that it works as intended: https://github.com/nerdvegas/rez/wiki/Configuring-Rez#default_cachable_per_repository Specifically: """Set cachable on a per-package-repository basis. Overrides 'default_cachable_per_package' and 'default_cachable' for any repos listed here.""" You should be able to use this setting like so: ``` default_cachable_per_repostitory = { '/some/network/test/area': False } ``` This should override `default_cachable` for all packages in that repo. Thx A

…

On Sat, Apr 9, 2022 at 7:35 AM Topher ***@***.***> wrote: Update: I have found a way to conditionally prevent caching. Adding this to the package.py @late()def cachable(): import os return os.path.join('rez', 'packages', 'test') not in this.base.lower() I tried something similar before, but it didn't work because 1) I was using the wrong slashes on windows, and 2) the this.base was all lowercase and there was 1 directory that was capitalized. So I have a workaround, but ideally there would be a way within rez to prevent certain directories from caching. — Reply to this email directly, view it on GitHub <#1281 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSV4VLHZQCNRZVYT4CTVECRBXANCNFSM5S2UOK5Q> . You are receiving this because you commented.Message ID: ***@***.***>

[hughetop]

I am so sorry, I completely misinterpreted your suggestion because I was thinking repository == package. However, I can't seem to get that to work, but I could be using incorrect syntax. In my rezconfig I have default_cachable = True and I just added

default_cachable_per_repostitory = {
    '/some/network/test/area': False
}

but when I build a package like rez-build --install --prefix=/some/network/test/area and then do
rez-env --paths=/some/network/test/area:$REZ_PACKAGES_PATH package1 it still caches the test package. I see a note about canonical paths, but I tested on both linux and macos with the same result.

[hughetop]

OH! I see the problem...a typo...

- default_cachable_per_repostitory = {
+ default_cachable_per_repository = {

That'll teach me to copy+paste without proofreading. Yes, @nerdvegas, it works perfectly fine. Even across platforms:

import os, sys

def _default_cachable_per_repository():
    prefix = '/some'
    if sys.platform == 'darwin':
        prefix = '/Volumes/SOME'

    return {
        os.path.join(prefix, 'network/test/area'): False
    }

default_cachable_per_repository = _default_cachable_per_repository()