Question about disk hits and environment variables

[ttanimura]

Hey everyone. So we are gearing up for a trial run of Rez for our department at Imageworks. There has been a good amount of interest, and if all goes well we will have a strong case to make for other departments to adopt.

One concern that we have is the extra number of disk hits it will take when looking for plugins/scripts/libraries etc. Since every package installs itself into its own location, we face the possibility of having to search through many more directories than we currently do. We are being diligent with the number of packages we are creating, but it seems to me that over time the number of paths is only going to grow.

I'm curious if other studios have run into this problem, and if so, how they solved it. We have talked about having some sort of process that takes a resolved environment and centralises (or localises) all of its data into a single directory, although that seems fraught with problems... any insight would be much appreciated.

Thanks!

[nerdvegas]

Hey Taisuke, you should read my post yesterday that talks specifically about this exact problem ;) Will find and paste...

…

On Wed, Feb 1, 2017 at 7:16 AM, ttanimura ***@***.***> wrote: Hey everyone. So we are gearing up for a trial run of Rez for our department at Imageworks. There has been a good amount of interest, and if all goes well we will have a strong case to make for other departments to adopt. One concern that we have is the extra number of disk hits it will take when looking for plugins/scripts/libraries etc. Since every package installs itself into its own location, we face the possibility of having to search through many more directories than we currently do. We are being diligent with the number of packages we are creating, but it seems to me that over time the number of paths is only going to grow. I'm curious if other studios have run into this problem, and if so, how they solved it. We have talked about having some sort of process that takes a resolved environment and centralises (or localises) all of its data into a single directory, although that seems fraught with problems... any insight would be much appreciated. Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#380>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABjqSjj4Cb6DulDB8evwYpuAehwojPCaks5rX5angaJpZM4LzH3G> .

[nerdvegas]

<snip> Hey all, So, I've thought a lot about this. And I think there's a good way to solve the problem, though suites isn't where to do it. Rez contexts lack anywhere on disk where packages have the opportunity to update the filesystem in a way that's specific to that context. In other words, it'd be nice if a package could create (soft/hard) links somewhere on the filesystem for a given resolve, rather than appending PYTHONPATH. IMO this feature should be added. You should be able to configure a location on your NFS where these 'context filesystems' go, and rez should know when it can reuse them. If this is not configured, the context filesystem would be created in /tmp (almost certainly in the same tmpdir that rez already creates). If adding a context to a suite, you should be able to tell rez to put the context filesystem within the suite. It'd work something like this: # in package.py bind_fs = [ { "source_path": "{root}/python" "dest_path": "python" "hardlink": True } ] def commands(): env.PYTHONPATH.append("{root}/python") Here, the 'bind_fs' entry is telling rez that as part of this package's actions, it needs to hardlink everything in the package under {root}/python, into the 'python' directory in the context filesystem. The following PYTHONPATH.append call actually knows to map the path its given back to the new bound path. It will avoid appending identical paths. The bind_fs call also tracks everything that happens, and uses this info to create a hash, and it's this hash that decides where the context fs is stored on disk. So for example, let's say we have a maya_utils and nuke_utils package. This is what happens: 1. the resolve is done 2. the commands are run. Two bind_fs entries are found - one for maya_utils, one for nuke_utils. 3. a hash is generated, unique to the list of fs bindings that have occurred (could just be a hash of a list of (path, is_soft_link) for example). 4. the PYTHONPATH.append call in both cases become the same path - only one append occurs. 5. Now we have the hash, we know the context filepath. It might be: /rez/context_fs/ab295c32fa2486fdd126. 6. If that path doesn't exist, it is created, and the bindings are executed - this creates the relevant links for each package, in that dir. If the path does exist, it's reused. So here, we end up with /rez/context_fs/ ab295c32fa2486fdd126/python/mas_utils and /rez/context_fs/ ab295c32fa2486fdd126/python/nuke_utils created. 7. The rez-env is started as per usual. </snip> * (2) is a little off, because initially I thought that bind_fs() calls belonged in the commands. That actually complicates things, and it's better off if the fs bindings are stored separately to the commands.

…

On Wed, Feb 1, 2017 at 7:28 AM, Allan Johns ***@***.***> wrote: Hey Taisuke, you should read my post yesterday that talks specifically about this exact problem ;) Will find and paste... On Wed, Feb 1, 2017 at 7:16 AM, ttanimura ***@***.***> wrote: > Hey everyone. So we are gearing up for a trial run of Rez for our > department at Imageworks. There has been a good amount of interest, and if > all goes well we will have a strong case to make for other departments to > adopt. > > One concern that we have is the extra number of disk hits it will take > when looking for plugins/scripts/libraries etc. Since every package > installs itself into its own location, we face the possibility of having to > search through many more directories than we currently do. We are being > diligent with the number of packages we are creating, but it seems to me > that over time the number of paths is only going to grow. > > I'm curious if other studios have run into this problem, and if so, how > they solved it. We have talked about having some sort of process that takes > a resolved environment and centralises (or localises) all of its data into > a single directory, although that seems fraught with problems... any > insight would be much appreciated. > > Thanks! > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#380>, or mute the thread > <https://github.com/notifications/unsubscribe-auth/ABjqSjj4Cb6DulDB8evwYpuAehwojPCaks5rX5angaJpZM4LzH3G> > . >

[ttanimura]

Yes! This is exactly what we are looking for. Sorry about not reading your earlier post, and thank you for re-posting :)