Another containerd issue

[dmikusa]

Summary

Seeing what looks like another issue related to GHA & containerd. This one is with pack, when you do a build on GHA:

Warning: Exporting to docker daemon (building without --publish) and daemon uses containerd storage; performance may be significantly degraded.
For more information, see https://github.com/buildpacks/pack/issues/2272.
0.0.78: Pulling from paketobuildpacks/ubuntu-noble-builder-buildpackless
89732bc75041: Pulling fs layer
16d7a2d6d0f3: Pulling fs layer
33fe83d5dd91: Pulling fs layer
b8ca7013fd5b: Pulling fs layer
f208496158d2: Pulling fs layer
2ac2a660ad99: Pulling fs layer
500a46ce47c1: Pulling fs layer
b21444ab8d88: Pulling fs layer
b21444ab8d88: Download complete
89732bc75041: Download complete
33fe83d5dd91: Download complete
f208496158d2: Download complete
2ac2a660ad99: Download complete
500a46ce47c1: Download complete
16d7a2d6d0f3: Download complete
b8ca7013fd5b: Download complete
b21444ab8d88: Pull complete
2ac2a660ad99: Pull complete
b8ca7013fd5b: Pull complete
500a46ce47c1: Pull complete
16d7a2d6d0f3: Pull complete
89732bc75041: Pull complete
33fe83d5dd91: Pull complete
f208496158d2: Pull complete
Digest: sha256:484fd7477a7cae13329f880d46d7fa2dcb7e5df6592e5801ea2a371db745c280
Status: Downloaded newer image for paketobuildpacks/ubuntu-noble-builder-buildpackless:0.0.78
latest: Pulling from paketobuildpacks/ubuntu-noble-run
5d754b86dbff: Pulling fs layer
813ca5898330: Pulling fs layer
e7ab608c8949: Pulling fs layer
5d754b86dbff: Download complete
e7ab608c8949: Download complete
813ca5898330: Download complete
5d754b86dbff: Pull complete
e7ab608c8949: Pull complete
813ca5898330: Pull complete
Digest: sha256:ad9e88c345a5e2a1d896f3f5f8882d15cf7cb49fd52e9ac2d61cbf9780538429
Status: Downloaded newer image for paketobuildpacks/ubuntu-noble-run:latest
latest: Pulling from paketobuildpacks/go-dist
f9c162d5355a: Pulling fs layer
f9c162d5355a: Download complete
f9c162d5355a: Pull complete
Digest: sha256:9ab3987697a96b9f144af0c71aa569e8635949390aa1ac69786bb9210a177432
Status: Downloaded newer image for paketobuildpacks/go-dist:latest
latest: Pulling from paketocommunity/build-plan
f6d0a858a9e9: Pulling fs layer
f6d0a858a9e9: Download complete
f6d0a858a9e9: Pull complete
Digest: sha256:29f431ebaa3dc7296ed5a1322ea9af749f790ed767f3e395eab616ce350a6ec0
Status: Downloaded newer image for paketocommunity/build-plan:latest
latest: Pulling from paketobuildpacks/procfile
026c6dba3379: Pulling fs layer
026c6dba3379: Download complete
026c6dba3379: Pull complete
Digest: sha256:244d771287677280c49212fc51ffd8240da3b3d32fafd9f389f2b55a00b29de7
Status: Downloaded newer image for paketobuildpacks/procfile:latest
Warning: Builder is trusted but additional modules were added; using the untrusted (5 phases) build flow
docker.io/buildpacksio/lifecycle@sha256:6eaa70cbaa7823b3befdaa6d4babcdd040f3815e1fe2f8b70ae4008f7c171f87: Pulling from buildpacksio/lifecycle
ce49d46500be: Pulling fs layer
52630fc75a18: Pulling fs layer
3a6a574b4221: Pulling fs layer
dd64bf2dd177: Pulling fs layer
406ab5c5f811: Pulling fs layer
ebddc55facdc: Pulling fs layer
c172f21841df: Pulling fs layer
3214acf345c0: Pulling fs layer
2780920e5dbf: Pulling fs layer
7c12895b777b: Pulling fs layer
b839dfae01f6: Pulling fs layer
bdfd7f7e5bf6: Pulling fs layer
d6b1b89eccac: Pulling fs layer
c172f21841df: Download complete
dd64bf2dd177: Download complete
b839dfae01f6: Download complete
bdfd7f7e5bf6: Download complete
d6b1b89eccac: Download complete
ebddc55facdc: Download complete
2780920e5dbf: Download complete
7c12895b777b: Download complete
3214acf345c0: Download complete
52630fc75a18: Download complete
ce49d46500be: Download complete
3a6a574b4221: Download complete
c172f21841df: Pull complete
ce49d46500be: Pull complete
dd64bf2dd177: Pull complete
b839dfae01f6: Pull complete
bdfd7f7e5bf6: Pull complete
d6b1b89eccac: Pull complete
ebddc55facdc: Pull complete
2780920e5dbf: Pull complete
7c12895b777b: Pull complete
3214acf345c0: Pull complete
52630fc75a18: Pull complete
3a6a574b4221: Pull complete
406ab5c5f811: Download complete
406ab5c5f811: Pull complete
Digest: sha256:6eaa70cbaa7823b3befdaa6d4babcdd040f3815e1fe2f8b70ae4008f7c171f87
Status: Downloaded newer image for buildpacksio/lifecycle@sha256:6eaa70cbaa7823b3befdaa6d4babcdd040f3815e1fe2f8b70ae4008f7c171f87
ERROR: failed to build: failed to fetch base layers: open /tmp/imgutil.local.image.2344428218/blobs/sha256/b9f0d77a987e097d1577fc7e5c756581f21392771dd9c2dd6e1635860b977a91: no such file or directory
Error: Process completed with exit code 1.

---

Reproduction

Steps

1. Example build command:

pack build applications/procfile \
          --builder docker.io/paketobuildpacks/ubuntu-noble-builder-buildpackless:latest \
          --lifecycle-image buildpacksio/lifecycle:e10cf01 \
          --buildpack docker.io/paketobuildpacks/go-dist \
          --buildpack docker.io/paketocommunity/build-plan \
          --buildpack docker.io/paketobuildpacks/procfile \
          --trust-builder

Sample app: https://github.com/paketo-buildpacks/samples/tree/main/procfile/procfile-sample

Current behavior

Fails to fetch lifecycle image.

Expected behavior

Should fetch lifecycle image.

---

Environment

pack info

0.40.0 and 0.39.1.

docker info

Docker 29 & GHA

[dmikusa]

Workaround: disable containerd

     - name: Disable containerd snapshotter
        run: |
          echo '{"features": {"containerd-snapshotter": false}}' | sudo tee /etc/docker/daemon.json
          sudo systemctl restart docker

[jjbustamante]

@dmikusa I tried to reproduce this error (locally) with the latest pack 0.40.1 version, and I didn't manage to do it. Could you re-try?

My Local Configuration

• I am running Docker 29 and switched to containerd storage

juan@pop-os:~/go/src/github.com/paketo-buildpacks/samples/procfile/procfile-sample$ docker info -f '{{ .DriverStatus}}'
[[driver-type io.containerd.snapshotter.v1]]
juan@pop-os:~/go/src/github.com/paketo-buildpacks/samples/procfile/procfile-sample$ pack --version
0.40.1+git-bb546d6.build-6779
juan@pop-os:~/go/src/github.com/paketo-buildpacks/samples/procfile/procfile-sample$ docker info
Client: Docker Engine - Community
 Version:    29.3.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.31.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.1.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 19
 Server Version: 29.3.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc version: v1.0.0-296-ge999e29a
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.17.9-76061709-generic
 Operating System: Pop!_OS 22.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.67GiB
 Name: pop-os
 ID: 4f71215f-ffa4-460b-bc50-8e45e5c17b00
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: jjbustamante
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

Local Build

juan@pop-os:~/go/src/github.com/paketo-buildpacks/samples/procfile/procfile-sample$ pack build applications/procfile \
          --builder docker.io/paketobuildpacks/ubuntu-noble-builder-buildpackless:latest \
          --lifecycle-image buildpacksio/lifecycle:e10cf01 \
          --buildpack docker.io/paketobuildpacks/go-dist \
          --buildpack docker.io/paketocommunity/build-plan \
          --buildpack docker.io/paketobuildpacks/procfile \
          --trust-builder
Warning: Ignoring the provided lifecycle image as the builder is trusted, running the creator in a single container using the provided builder
Warning: Exporting to docker daemon (building without --publish) and daemon uses containerd storage; performance may be significantly degraded.
For more information, see https://github.com/buildpacks/pack/issues/2272.
latest: Pulling from paketobuildpacks/ubuntu-noble-builder-buildpackless
89732bc75041: Pull complete 
f208496158d2: Pull complete 
33fe83d5dd91: Pull complete 
500a46ce47c1: Pull complete 
bd06205d1b21: Pull complete 
3906af3250bb: Pull complete 
11ad43dc9c63: Pull complete 
b21444ab8d88: Pull complete 
df74f0172247: Pull complete 
Digest: sha256:2b5a2cfac171923d67d552dfb3ea9d7f55573984213a02d70ddfe39475991af3
Status: Downloaded newer image for paketobuildpacks/ubuntu-noble-builder-buildpackless:latest
latest: Pulling from paketobuildpacks/ubuntu-noble-run
45b87fc92e38: Pull complete 
d30a9e8a3f1e: Pull complete 
8b0272a36a61: Pull complete 
Digest: sha256:8bea9e1a60b05071e3721056ac057618262403fb9385a88e894eb825b113c79f
Status: Downloaded newer image for paketobuildpacks/ubuntu-noble-run:latest
latest: Pulling from paketobuildpacks/go-dist
9ed76ba221c8: Pull complete 
Digest: sha256:0810f8484a518edce72b01f9f2ee7bdd1ad83ca9bf2f547659655dc2693b073f
Status: Downloaded newer image for paketobuildpacks/go-dist:latest
latest: Pulling from paketocommunity/build-plan
f6d0a858a9e9: Pull complete 
Digest: sha256:29f431ebaa3dc7296ed5a1322ea9af749f790ed767f3e395eab616ce350a6ec0
Status: Downloaded newer image for paketocommunity/build-plan:latest
latest: Pulling from paketobuildpacks/procfile
d8a18e651d89: Pull complete 
Digest: sha256:4c160fd0f526cdbea9c1896c6ea6ed9f67b41983d8d72161bad39eb1dda089f3
Status: Downloaded newer image for paketobuildpacks/procfile:latest
Warning: Builder is trusted but additional modules were added; using the untrusted (5 phases) build flow
docker.io/buildpacksio/lifecycle@sha256:6eaa70cbaa7823b3befdaa6d4babcdd040f3815e1fe2f8b70ae4008f7c171f87: Pulling from buildpacksio/lifecycle
406ab5c5f811: Pull complete 
bdfd7f7e5bf6: Pull complete 
b839dfae01f6: Pull complete 
ce49d46500be: Pull complete 
52630fc75a18: Pull complete 
2780920e5dbf: Pull complete 
d6b1b89eccac: Pull complete 
dd64bf2dd177: Pull complete 
ebddc55facdc: Pull complete 
7c12895b777b: Pull complete 
3214acf345c0: Pull complete 
3a6a574b4221: Pull complete 
c172f21841df: Pull complete 
Digest: sha256:6eaa70cbaa7823b3befdaa6d4babcdd040f3815e1fe2f8b70ae4008f7c171f87
Status: Downloaded newer image for buildpacksio/lifecycle@sha256:6eaa70cbaa7823b3befdaa6d4babcdd040f3815e1fe2f8b70ae4008f7c171f87
===> ANALYZING
[analyzer] Image with name "applications/procfile" not found
===> DETECTING
[detector] target distro name/version labels not found, reading /etc/os-release file
[detector] paketo-buildpacks/go-dist   2.10.1
[detector] paketo-community/build-plan 0.3.21
[detector] paketo-buildpacks/procfile  5.13.0
===> RESTORING
[restorer] No run metadata found at path "/cnb/run.toml"
===> BUILDING
[builder] target distro name/version labels not found, reading /etc/os-release file
[builder] Paketo Buildpack for Go Distribution 2.10.1
[builder]   Resolving Go version
[builder]     Candidate version sources (in priority order):
[builder]       <unknown> -> ""
[builder] 
[builder]     Selected Go version (using <unknown>): 1.25.7
[builder] 
[builder]   Executing build process
[builder]     Installing Go 1.25.7
[builder]       Completed in 5.821s
[builder] 
[builder]   Generating SBOM for /layers/paketo-buildpacks_go-dist/go
[builder]       Completed in 0s
[builder] 
[builder] 
[builder] Paketo Buildpack for Procfile 5.13.0
[builder]   https://github.com/paketo-buildpacks/procfile
[builder]   Process types:
[builder]     web: go run main.go
===> EXPORTING
[exporter] Adding layer 'paketo-buildpacks/go-dist:go'
[exporter] Adding layer 'buildpacksio/lifecycle:launch.sbom'
[exporter] Added 1/1 app layer(s)
[exporter] Adding layer 'buildpacksio/lifecycle:launcher'
[exporter] Adding layer 'buildpacksio/lifecycle:config'
[exporter] Adding layer 'buildpacksio/lifecycle:process-types'
[exporter] Adding label 'io.buildpacks.lifecycle.metadata'
[exporter] Adding label 'io.buildpacks.build.metadata'
[exporter] Adding label 'io.buildpacks.project.metadata'
[exporter] Adding label 'io.buildpacks.exec-env'
[exporter] Setting default process type 'web'
[exporter] Saving applications/procfile...
[exporter] *** Images (83a367506b83):
[exporter]       applications/procfile
Successfully built image applications/procfile