(SOLVED) meta-demo-ci problems with sstate mirroring

[robwoolley]

I have been working on getting meta-demo-ci to work to store my downloads and sstate-cache in S3 buckets. I noticed that the downloads were being mirrored but the sstate-cache was not.

I managed to resolve the problem, but I am sharing the details here to help anyone else that runs into this issue.

This is the configuration that I was adding to local.conf based on the documentation:

    # Configuration for populating the download mirrors
    BB_GENERATE_MIRROR_TARBALLS = "1"
    UPDATE_DOWNLOADS_MIRROR = "1"
    DOWNLOADS_MIRROR_URL = "s3://oeros-bitbake-cache/downloads"

    # Configuration for sstate mirroring
    UPDATE_SSTATE_MIRROR = "1"
    SSTATE_MIRROR_URL = "s3://oeros-bitbake-cache/sstate-cache"

    # Configuration for downloading from the mirrors
    SOURCE_MIRROR_URL = "s3://oeros-bitbake-cache/downloads"
    INHERIT += "own-mirrors"
    SSTATE_MIRRORS = "file://.* s3://oeros-bitbake-cache/sstate-cache/PATH;downloadfilename=PATH"

    # Configuration for uploading to mirrors
    INHERIT += "mirror_updates"

Instrumentation showed the "if" block in mirror_updates.bbclass (https://github.com/OE4T/tegra-demo-distro/blob/master/layers/meta-demo-ci/classes/mirror_updates.bbclass#L173) wasn't running because "sstate_task_postfunc" wasn't present in the task's postfuncs varflag.

This is because the sstate.bbclass had not yet run its code to add the varflag (https://github.com/openembedded/openembedded-core/blob/scarthgap/meta/classes-global/sstate.bbclass#L161)

I tracked this back to the use of INHERIT += "mirror_updates" in local.conf. This caused mirror_updates to be appended to the end of an empty INHERIT variable followed by sstate being appended afterwards by defaultsetup.conf.

This resulted in an INHERIT variable that looked like this: INHERIT=" own-mirrors mirror_updates report-error rm_work package_ipk debian devshell sstate license remove-libtool sanity"

One possible solution would be to move the INHERIT line to auto.conf or layer.conf which I believe is parsed afterwards. My solution was to use the following which instead appended the string at the end of parsing:
INHERIT:append = " mirror_updates"

[madisongh]

Thanks, @robwoolley ! I have a note about using INHERIT:append in my own CI builder's configuration files, but when I wrote this wiki page I missed transcribing that rather important bit of information. I've updated the page accordingly.

[robwoolley]

Cool. Glad to have helped. I think the wiki page may need to be updated to say mirror_updates instead of sstate_mirror_updates.

Thanks for developing this and sharing it. My AWS builds are much faster and I appreciate not having to manage volumes between runners.