Remote backend for CacheDir

[aaronborden-rivianvw]

Describe the Feature
Support remote backend implementations for SCons using the custom CacheDir class.

Required information

• Link to SCons Users thread discussing your issue https://discord.com/channels/571796279483564041/1139589636855955587
• Version of SCons 4.5.2
• Version of Python 3.8
• Which python distribution if applicable python.org
• How you installed SCons pypi
• What Platform are you on? Ubuntu Linux 20.04
• How to reproduce your issue? Please include a small self contained reproducer. Likely a SConstruct should do for most issues.
• How you invoke scons scons ecu=<ecu> config=<config>

Additional context
Add any other context or screenshots about the feature request here.

The existing cache logic in SCons works well for us. This feature is about making that cache available remotely/distributed rather than assuming the cache is a local directory. This doesn't have to be implemented using the custom CacheDir class, but the existing interface makes sense for a simple remote backend. The docs recommend using an NFS for a shared/remote cache which essentially offloads the remote handling to the OS. We've found NFS doesn't scale across our CI fleet and are looking for a remote/distributed cache.

Here's an example interface/implementation that uses the build signature for a unique key that can be used to store and fetch from a remote backend.

class CustomCacheDir(Scons.CacheDir.CacheDir):
    @classmethod
    def retrieve(cls, env, bsig, src, dst) -> bool:
        """Retrieve an object from the remote cache. Return False if the operation fails, True otherwise."""
        return cls.backend.retrieve(bsig, dst)

    @classmethod
    def store(cls, env, bsig src, dst) -> bool:
        """Store an object in the cache."""
        return cls.backend.store(bsig, src)

    @classmethod
    def exists(cls, env, bsig, src, dst) -> bool:
        """Check if an object exists in the remote cache."""
        return cls.backend.exists(bsig)

    @classmethod
    def copy_from_cache(cls, env, bsig, src, dst) -> str:
        """Copy a file from cache."""
        # if object already exists in the local cache, no need to fetch it.
        if not os.path.exists(src):
            cls.retrieve(env, bsig, src, dst)

        # copy the local cached object to the destination
        return super().copy_from_cache(env, bsig, src, dst)

    @classmethod
    def copy_to_cache(cls, env, bsig src, dst) -> str:
        # store the object in the remote cache
        cls.store(env, bsig, src, dst)

        # store the object in the local cache
        return super().copy_to_cache(env, bsig, src, dst)

The behavior is modeled on ccache's remote backend strategy where by default, you have local pull-through cache. If SCons handles the local cache, then the existing logic mostly works as is. SCons would only delegate the determination of if an object exists in the cache to a custom cachedir class.

Local storage	Remote storage	What happens
miss	miss	Compile, write to local, write to remote[1]
miss	hit	Read from remote, write to local
hit	-	Read from local, don’t write to remote[2]

[1] Unless remote storage has attribute read-only=true.
[2] Unless local storage is set to share its cache hits with the reshare option.

Determining whether or not the object is in the cache might be unnecessary. If copy_from_cache returns a negative answer (e.g. False), SCons should assume the object wasn't there, compile and then store the object as necessary.

[aaronborden-rivianvw]

Just to be clear, I went ahead and tried to implement this with the existing CacheDir interface on S3. However, because SCons checks the local cache directory to determine if an object is in the cache or not, it never calls copy_from_cache on an empty local cache directory, even though the object exists on the remote. If instead it called copy_from_cache and handle the case where copy_from_cache copies no files or returns a negative response, it might end up working pretty well.

import re
import boto3

class S3CacheDir(SCons.CacheDir.CacheDir):
    log = s3_cachedir_logger
    _bucket = None
    tmpfile_re = re.compile(r'.tmp[0-9a-f]{32}$')

    @classmethod
    def key(cls, cachefile):
        signature = cls.tmpfile_re.sub('', os.path.basename(cachefile))

        key = "%s/%s" % (signature[:2], signature)
        prefix = os.getenv("CACHEDIR_S3_PREFIX", False)
        if prefix:
            key = "%s/%s" % (prefix, key)

        return key

    @classmethod
    def bucket(cls):
        if not cls._bucket:
            bucket_name = os.getenv("CACHEDIR_S3_BUCKET")
            cls._bucket = boto3.resource("s3").Bucket(bucket_name)

        return cls._bucket

    @classmethod
    def copy_from_remote(cls, cachefile):
        key = cls.key(cachefile)
        cls.log.debug(f'copy_from_remote key={key} cachefile={cachefile}')
        try:
            bucket = cls.bucket()
            bucket.download_file(key, cachefile)
        except e:
            cls.log.warning(f'failed to copy from bucket error={e}')

    @classmethod
    def copy_to_remote(cls, tmpfile):
        key = cls.key(tmpfile)
        cls.log.debug(f'copy_to_remote key={key} tmpfile={tmpfile}')
        try:
            bucket = cls.bucket()
            bucket.upload_file(tmpfile, key)
        except e:
            cls.log.warning(f'failed to copy to bucket error={e}')

    @classmethod
    def copy_from_cache(cls, env, src, dst) -> str:
        """Copy a file from cache."""
        if not os.path.exists(src):
            cls.copy_from_remote(src)

        return super().copy_from_cache(env, src, dst)

    @classmethod
    def copy_to_cache(cls, env, src, dst) -> str:
        result = super().copy_to_cache(env, src, dst)

        cls.copy_to_remote(dst)

        return result

[aaronborden-rivianvw]

Linking #3971 which implements a remote cache using a Bazel Remote backend, but directly in SCons, without the CacheDir implementation.

[mwichmann]

I like the pull-through cache idea. But considering such things does open up architectural questions. Does SCons want to grow a more comprehensive solution for caching, or does it want to incrementally improve as patches come along? I think #3971 in part stalled because of this - the submitter didn't want to put in a lot more work there without knowing the direction it was taking would be considered okay in the end, which I sympathize with.