Allow "common" or "shared" payloads between variants.

[herronelou]

Packages should have a way to specify a payload that is shared between all variants.

Motivation
Reduced data storage and file-wrangling.

Example:
Today I had to install 10 variants of a package (5 nuke versions x 2 OSes). Each variant is 300MB, but of that only 75MB is compiled for the OS/Nuke version, everything else is python and supporting data.
We want to modify the menu.py provided by the vendor slightly to suit our pipeline, and so I need to do that 10 times.
With a shared payload, we could have a package that's 975MB total, with a single menu.py to modify, instead of 3000MB and 10 menu.py.

Possible implementation
Being able to access {base} like we access {root} in the package definition might be sufficient, along with a way for certain rez commands such as rez-cp to understand what is a variant and what is a "shared" file.

[maxnbk]

Parts of this can be already be possible to optimize depending on the specifics. The part that isn't possible to optimize is the OS part, if there are different OSes, but possible if it's the same OS but a different version number.

For example, what you can do is:

• Write a single package.py, with a requires of the full range of nuke versions.
• Write a build that creates the payload structure you want; If there is a range of nuke-specific files, you'd have to manually encode that part, or read it out of the package.py yourself.
• Then commands() picks up on which nuke version is being resolved, and constructs the env vars by using the variables created by the nuke package in-play to use the payload path-ing that you designed yourself, presumably into "variant-like variantless structure" you chose.

(If the OS is different versions of the same OS, that can work the same way since a single requirement will cover it, but if it's two different OSes, then that has to be split into two variants, at least.)

I've used this strategy with, for example, maya or nuke plugins whose installations unpack multiple versions of plugins in their own version-specific structure. You can be very granular, since rez makes specific variables for major, minor, patch, making a variety of constructions possible.

All of this doesn't necessarily mean that the idea is without merit - I've wished for something similar a few times myself - But this is in fact a workable alternative in a number of scenarios to reduce disk-space consumed.

[herronelou]

We could do that indeed, we've also cheated it another way with accessing the base path via this and working it into paths in the past, but it feels a bit like a hack.

"variant-like variantless structure" essentially sounds like bypassing the variant system instead of improving it :)

[maxnbk]

Sure does! ;) Just offering up workarounds. But I sort of think that a varianted-package with a single payload would kind of have to work that way under the hood anyway, no? At least, if you have any need whatsoever for the env-vars to be set differently per-version-of-nuke, or whatever app. Can you describe how you'd imagine that aspect working? Or is the point that you wouldn't be able to make the env any different?

The implications of involving the package base in the equation are pretty rough. For example, package-caching occurs on a variant-level, so would it cache the base as if it were a separate variant? How does it differentiate what the actual contents of the base are, given that every variant will be contained within it?

How would releasing a new variant work? Presumably the permissions of the released artifact are locked down at that point, so adding things to it wouldn't be possible at that point.

[herronelou]

Yes you're right, the base was a workaround I did that does not take into account all the features of rez.

Maybe a specific folder name alongside the variant folders?

[maxnbk]

The best way I can think of for such a thing to interoperate with the current system in a way that wouldn't require extensive retrofitting or seriously break other important systems, would be something like the ability to specify an attribute in a package of the effect of common_base_variant = "common" , which would cause the build system, on building the first(?) or any(?) variant, to invoke a separate base_variant_build_command = ..., which would be used to build an equal-in-all-ways-to-any-other-variant variant, but which would generate a variant for that package named common (per the variable setting), which can act just like any hashable variant.

The variant would get hashed like any other (if set), any attempt to cache the package would have to also cache the common_base_variant, and this variant would technically be not-directly-resolvable (importantly, it would never exist in the variants = [] list), but the introduction of any variant in the solve for a package with this feature, would result in some new variable being set for pre/post/commands() such as {this.common_base}, so that the commands for the variant doing the actual command-work could add environment variables to introduce the required behavior to make that common_base useful in some way. Perhaps some new env vars like REZ_<PKGNAME>_COMMON_BASE would be introduced as well.

This approach would certainly introduce some other "Oops" and Gotchas, that I'm pretty sure are obvious at the moment, but I won't railroad the suggestion with all the challenges before even agreeing whether the above proposal might work for the situation you're describing.