Multi level image builds - only base image is cached

[oerp-odoo]

Behaviour

Not sure if I am doing it wrong, but I can't make specific images to be cached (if I use docker buildx bake locally, it caches everything properly), but using this action, it only caches my base image.

Steps to reproduce this issue

For example I tried:

      - name: Login to docker registry
        uses: docker/login-action@v2
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2.9.1
      - name: Build and Push Projects
        uses: docker/bake-action@v3.1.0
        env:
          PROJECT_TAG: cmt-${{ github.sha }}
        with:
          # targets: "${{ needs.project_names.outputs.names_comma }}"
          targets: base,specific_image1,specific_image2
          push: true
          set: |
            base.cache-from=type=gha,scope=build-base
            base.cache-to=type=gha,scope=build-base,mode=max
            specific_image1.cache-from=type=gha,scope=build-specific_image1
            specific_image1.cache-to=type=gha,scope=build-specific_image1,mode=max
            specific_image2.cache-from=type=gha,scope=build-specific_image2
            specific_image2.cache-to=type=gha,scope=build-specific_image2,mode=max

I also tried specifying cache generically like:

      *.cache-from=type=gha
      *.cache-to=type=gha,mode=max

Expected behaviour

I would expect to cache all image built, not just base one.

Actual behaviour

Base image is cached, and specific image is always rebuilt even if nothing have changed.

Configuration

My bake file looks like this:

variable "BASE_TAG" {
  default = "16.0"
}
variable "PROJECT_TAG" {
  default = "16.0"
}

target "base" {
  dockerfile = "src/base/Dockerfile"
  contexts = {
    base-src = "src/base"
  }
  tags = ["ghcr.io/myorg/base:${BASE_TAG}"]
}

target "_all" {
  contexts = {
    base = "target:base",
    extra-src = "src/extra"
  }
}

target "specific_image_1" {
  inherits = ["_all"]
  contexts = {
    project-src = "src/projects/specific_image_1"
  }
  dockerfile = "${PROJECT_DOCKERFILE}"
  tags = ["ghcr.io/myorg/specific_image_1:${PROJECT_TAG}"]
}

Logs

In logs, part that is cached is actually coming from base image, not specific one.:

...
#24 [specific_image_1 stage-0  7/16] RUN python3 -m venv /opt/odoo/venv     && pip3 install --no-cache-dir pip-tools==6.13.0 wheel     && pip-compile-install /opt/odoo/requirements
#24 CACHED

#25 [specific_image_1 stage-0 15/16] COPY --from=monodoo-src --chown=odoo:odoo ./entrypoint.py entrypoint.py
#25 CACHED

#26 [specific_image_1 stage-0  4/16] COPY --from=monodoo-src --chown=odoo:odoo ./odoo/requirements.txt /opt/odoo/requirements/odoo-requirements.in
#26 CACHED

#27 [specific_image_1 stage-0 10/16] RUN pip3 install --no-cache-dir -e /opt/odoo/odoo     && pip3 install --no-cache-dir "/opt/odoo/bins/anthem-0.13.1.dev33+gcf73513-py2.py3-none-any.whl"
#27 CACHED

#28 [specific_image_1 stage-0 12/16] RUN pip3 install --no-cache-dir -e /opt/odoo/songs
#28 CACHED

#29 [specific_image_1 stage-0  8/16] COPY --from=monodoo-src --chown=odoo:odoo ./odoo /opt/odoo/odoo
#29 CACHED

#30 [specific_image_1 stage-0 14/16] COPY --chown=odoo:odoo ./pytest.ini /opt/odoo/pytest.ini
#30 CACHED

#31 [specific_image_1 stage-0  5/16] COPY --from=monodoo-src --chown=odoo:odoo ./requirements.txt /opt/odoo/requirements/requirements.in
#31 CACHED

#32 [specific_image_1 stage-0 16/16] COPY --from=monodoo-src --chown=odoo:odoo ./addons /opt/odoo/projects/monodoo
#32 sha256:287b999fa4f23b153628e0306124c5821df8499eed930d33db2fb0631cfcd35c 0B / 216B 0.2s
#32 ...
...

#35 [base] exporting to GitHub Actions Cache
#35 preparing build cache for export 2.6s done
#35 DONE 2.6s

#32 [base stage-0 16/16] COPY --from=base-src --chown=odoo:odoo ./addons /opt/odoo/projects/base
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 52.43MB / 314.72MB 6.0s
#32 extracting sha256:cf92e523b49ea3d1fae59f5f082437a5f96c244fda6697995920142ff31d59cf 1.7s done
#32 extracting sha256:89c0313aa29e2dd962db1c104533845c3b8a4417e3f9498ed0b277a5c8d96901 0.0s done
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 40.89MB / 140.09MB 5.4s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 49.28MB / 156.02MB 5.7s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 49.28MB / 140.09MB 6.2s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 57.67MB / 156.02MB 6.5s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 70.25MB / 314.72MB 7.7s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 56.62MB / 140.09MB 6.9s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 66.06MB / 156.02MB 7.4s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 65.01MB / 140.09MB 7.7s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 88.08MB / 314.72MB 9.3s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 73.40MB / 140.09MB 8.4s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 80.74MB / 140.09MB 9.2s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 89.13MB / 140.09MB 9.9s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 105.91MB / 314.72MB 11.0s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 74.45MB / 156.02MB 10.4s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 97.52MB / 140.09MB 10.7s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 82.84MB / 156.02MB 11.4s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 104.86MB / 140.09MB 11.4s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 122.68MB / 314.72MB 12.5s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 112.20MB / 140.09MB 12.0s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 92.27MB / 156.02MB 12.5s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 119.54MB / 140.09MB 12.8s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 140.51MB / 314.72MB 14.1s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 100.66MB / 156.02MB 13.5s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 127.93MB / 140.09MB 13.5s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 136.31MB / 140.09MB 14.3s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 109.05MB / 156.02MB 14.6s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 140.09MB / 140.09MB 14.7s done
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 158.33MB / 314.72MB 15.8s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 117.44MB / 156.02MB 15.5s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 175.11MB / 314.72MB 17.3s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 125.83MB / 156.02MB 16.5s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 134.22MB / 156.02MB 17.4s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 192.94MB / 314.72MB 18.9s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 142.61MB / 156.02MB 18.5s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 150.99MB / 156.02MB 19.5s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 210.76MB / 314.72MB 20.6s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 156.02MB / 156.02MB 20.0s done
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 228.59MB / 314.72MB 22.4s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 246.42MB / 314.72MB 24.5s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 264.24MB / 314.72MB 26.6s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 281.02MB / 314.72MB 28.5s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 297.80MB / 314.72MB 30.5s
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 314.72MB / 314.72MB 32.4s done
#32 extracting sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438
#32 extracting sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 11.5s done
#32 extracting sha256:bcdb739d5aef016392e512fa997b8d9d306e64497bbaaa60c3b98a2f778884c7 0.0s done
#32 DONE 44.6s

#32 [specific_image_1 stage-0 16/16] COPY --from=base-src --chown=odoo:odoo ./addons /opt/odoo/projects/base
#32 extracting sha256:161804423ce2990c2dbff86ee156e9a25c4b230934ff56f121be1506ff494762 0.0s done
#32 extracting sha256:c9285c82326f171fa758940fa64cc967cd87dac0d945976540eb67e911f66b10 0.0s done
#32 extracting sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710
#32 extracting sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 6.6s done
#32 extracting sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf
#32 extracting sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 17.3s done
#32 extracting sha256:3ae774462f6508dbc08f158ce375fa336d7d572b7bab20043ddabb7ce540b373 0.0s done
#32 extracting sha256:f906c05f9d6e04001861baa255081d8a36ff0e569f6d2b6799d16bf90b8ee58f 0.0s done
#32 extracting sha256:903c40a7931a03fd4d19722115d49b0e0080e91fdc280beb6056360452051f44 0.0s done
#32 DONE 68.6s

#32 [specific_image_1 stage-0 16/16] COPY --from=base-src --chown=odoo:odoo ./addons /opt/odoo/projects/base
#32 extracting sha256:fba3414c2be77074d36dff6a0128e4027a39049f2d1130795cd46e9e206066ea 0.0s done
#32 extracting sha256:154d71d26556a7378017cb7151f6f4644d27c9e4a27d8f25d5688f6ee62dbc69 0.0s done
#32 extracting sha256:32c43fb07e4e86e64922daf97eb8939eba6f39b47d0db3501b80ff8c6a6ce21e 0.0s done
#32 extracting sha256:287b999fa4f23b153628e0306124c5821df8499eed930d33db2fb0631cfcd35c 0.0s done
#32 extracting sha256:9c83cbfacf8c82b77a547c595566a6a9ea060ab0e635825e9fef63a117a38e61
#32 extracting sha256:9c83cbfacf8c82b77a547c595566a6a9ea060ab0e635825e9fef63a117a38e61 0.1s done
#32 DONE 68.8s

#36 [specific_image_1 stage-0 1/9] COPY --from=project-src --chown=odoo:odoo ./requirements.txt /opt/odoo/requirements/custom-requirements.in
#36 DONE 0.2s

#37 [specific_image_1 stage-0 2/9] COPY --from=extra-src --chown=odoo:odoo ./connector/requirements.txt /opt/odoo/requirements/connector-requirements.in
#37 DONE 0.0s

#38 [specific_image_1 stage-0 3/9] RUN pip-compile-install /opt/odoo/requirements
#0 0.145 + pip-compile-install /opt/odoo/requirements
#38 43.66 #
#38 43.66 # This file is autogenerated by pip-compile with Python 3.10
#38 43.66 # by the following command:
#38 43.66 #
#38 43.66 #    pip-compile --output-file=/opt/odoo/requirements/requirements.txt --resolver=backtracking /opt/odoo/requirements/connector-requirements.in /opt/odoo/requirements/custom-requirements.in /opt/odoo/requirements/odoo-requirements.in /opt/odoo/requirements/requirements.in
#38 43.66 #

Whats interesting, looks like last COPY of base image is not cachhed (not sure why):

#30 [specific_image_1 stage-0 14/16] COPY --chown=odoo:odoo ./pytest.ini /opt/odoo/pytest.ini
#30 CACHED

#31 [specific_image_1 stage-0  5/16] COPY --from=monodoo-src --chown=odoo:odoo ./requirements.txt /opt/odoo/requirements/requirements.in
#31 CACHED

#32 [specific_image_1 stage-0 16/16] COPY --from=monodoo-src --chown=odoo:odoo ./addons /opt/odoo/projects/monodoo
#32 sha256:287b999fa4f23b153628e0306124c5821df8499eed930d33db2fb0631cfcd35c 0B / 216B 0.2s
#32 ...

It just copies source code, but maybe its not supposed to be cached?

[crazy-max]

It's actually cached:

#32 [base stage-0 16/16] COPY --from=base-src --chown=odoo:odoo ./addons /opt/odoo/projects/base
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 52.43MB / 314.72MB 6.0s
#32 extracting sha256:cf92e523b49ea3d1fae59f5f082437a5f96c244fda6697995920142ff31d59cf 1.7s done
#32 extracting sha256:89c0313aa29e2dd962db1c104533845c3b8a4417e3f9498ed0b277a5c8d96901 0.0s done
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 40.89MB / 140.09MB 5.4s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 49.28MB / 156.02MB 5.7s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 49.28MB / 140.09MB 6.2s

#32 extracting and logs like sha256:... 52.43MB / 314.72MB 6.0s means it downloads and extracts the cache.

This is similar to docker/build-push-action#766 (comment)

[oerp-odoo]

It's actually cached:

#32 [base stage-0 16/16] COPY --from=base-src --chown=odoo:odoo ./addons /opt/odoo/projects/base
#32 sha256:16fd52525449f870f1ec2d0096b352bae611ff3876a7e733950c4628813f3438 52.43MB / 314.72MB 6.0s
#32 extracting sha256:cf92e523b49ea3d1fae59f5f082437a5f96c244fda6697995920142ff31d59cf 1.7s done
#32 extracting sha256:89c0313aa29e2dd962db1c104533845c3b8a4417e3f9498ed0b277a5c8d96901 0.0s done
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 40.89MB / 140.09MB 5.4s
#32 sha256:f9b1fff5a0fb46796fbac4bda595f1590813ae6302b13dd025fc40eac739afbf 49.28MB / 156.02MB 5.7s
#32 sha256:834c5693942525652d5952da9f7578d39838d6f94c63c52e729328f5615d9710 49.28MB / 140.09MB 6.2s

#32 extracting and logs like sha256:... 52.43MB / 314.72MB 6.0s means it downloads and extracts the cache.

This is similar to docker/build-push-action#766 (comment)

So then its not related. Though still not sure why specific images are not cached, when running same thing directly with bake, caches everything.

I also tried using local cache with bake-action, but it still works same way (a bit faster, because don't need to download/upload cache, but way slower than directly running bake).

[Geczy]

running into this as well here https://github.com/dotabod/backend/blob/master/docker-compose.yml

it only caches the first base image, stops caching after yarn install, and it keeps pushing new images to ghcr even though all the layers are the exact same. i can run teh workflow twice and i'll get two new pushes to the registry

i'm expecting only 1 push in that scenario. why is this happening @crazy-max ? you can see the job logs here https://github.com/dotabod/backend/actions/runs/7292673115/job/19874180120

[Geczy]

i fixed my issue by adding ghcr registry cache-from/to

[vanleeuw]

Is this related to and fix by doing this?
docker/build-push-action#286 (comment)