Using S3 cache in GHA

[ipriva]

I looked to S3 cache and it looked interesting, but I had problems to setup it using roles in GitHub Actions. The documentation states:

"Any system using environment variables / config files supported by the AWS Go SDK. The configuration must be available for the buildkit daemon, not for the client."

But I tried to find how you add environment variables or mount directories for buildkit daemon, but could not find any configuration for this. Should I somehow spin up buildkit container instead using docker/setup-buildx-action?

To my undestanding this is would be only way to use S3 cache with configuration/roles in GHA because IAM Instance profiles are only usable inside AWS and providing parameters do not support assume role.

[crazy-max]

Yes for containerized BuildKit instance it might be tricky as you need to create your own BuildKit image with AWS credentials inside /root/.aws/credentials and then run the setup-buildx-action with:

      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          driver-opts: image=user/custombuildkit:latest

You can also setup a buildkitd container and use the remote driver:

      -
        name: Set up buildkitd
        run: |
          docker run -d \
            --privileged \
            --name remote-buildkit \
            -p 1234:1234 \
            -v "/root/.aws/credentials:/root/.aws/credentials:ro" \
            moby/buildit:master \
            --addr unix:///run/buildkit/buildkitd.sock \
            --addr tcp://0.0.0.0:1234
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          # remote driver feature not yet available so build on-fly buildx
          version: https://github.com/docker/buildx#master
          driver: remote
          endpoint: docker-container://remote-buildkit
      -
        name: Build
        ...
      -
        name: Cleanup
        if: always()
        run: |
          docker rm -f remote-buildkit

This is not ideal and I think we should provide a dedicated configuration in buildkitd.toml that would be handled in buildx to propagate local credentials like we do for registry certs:

# /etc/buildkitd.toml
debug = true
[cache.s3."<bucket>-<region>"]
  credentials="/root/.aws/credentials"

[ipriva]

Custom image is too hazard for my purposes. I can live with couple extra setups in GHA, so I test suggested container spinup later today and let you know results. Thanks!