podman push: unsupported MIME type for compression: application/vnd.docker.image.rootfs.diff.tar.gzip

[Timoses]

Issue Description

Pulling an image, retagging it and trying to push results in the following error:

Error: creating an updated image manifest: preparing updated manifest, layer "sha256:56b812a48e20bdc4a1e18637e445987e10673a4ad229d9ff5c4f1486ae1e1f8c": unsupported MIME type for compression: application/vnd.docker.image.rootfs.diff.tar.gzip

Steps to reproduce the issue

Steps to reproduce the issue

1. podman pull registry.local/project/image:1
2. podman tag registry2.local/project/image:1
3. podman push registry2.local/project/image:1

Describe the results you received

> podman push registry2.local/project/image:1
Getting image source signatures
Copying blob fd190c1a6475 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 863eee1bec4c skipped: already exists
Copying blob 158dc2bb708e skipped: already exists
Copying blob 600c98360e08 skipped: already exists
Copying blob d8fc108cfe22 skipped: already exists
Copying blob 5c91ba2acd18 skipped: already exists
Copying blob 32f7015b9422 skipped: already exists
Copying blob 80071f9ed774 skipped: already exists
Copying blob 5c16b2868d93 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob 14163d527796 skipped: already exists
Copying blob 56b812a48e20 skipped: already exists
Copying blob 3e74f2c81bd3 skipped: already exists
Copying blob d8a374451c7b skipped: already exists
Copying blob 611a36c75723 skipped: already exists
Error: creating an updated image manifest: preparing updated manifest, layer "sha256:56b812a48e20bdc4a1e18637e445987e10673a4ad229d9ff5c4f1486ae1e1f8c": unsupported MIME type for compression: application/vnd.docker.image.rootfs.diff.tar.gzip

Describe the results you expected

It should push it successfully.

(Repeating the same commands with docker instead of podman works)

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 97.64
    systemPercent: 0.74
    userPercent: 1.62
  cpus: 2
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "38"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.3.8-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 312930304
  memTotal: 8309153792
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 5106642944
  swapTotal: 8308912128
  uptime: 1803h 59m 48.00s (Approximately 75.12 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/funke_t1/.local/share/containers/storage
  graphRootAllocated: 52595523584
  graphRootUsed: 26896195584
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 8
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 1685123928
  BuiltTime: Fri May 26 19:58:48 2023
  GitCommit: ""
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

I have found this: podman-container-tools/buildah#3668
which suggests the image does not fulfill the OCI spec.

However, the spec states that it is interchangeable:
https://github.com/opencontainers/image-spec/blob/main/media-types.md#applicationvndociimagelayerv1targzip

> skopeo inspect --raw docker://registry.local/project/image:1 | jq
{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "size": 4899,
    "digest": "sha256:17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0"
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 6191,
      "digest": "sha256:5c16b2868d93fb9e0d11be3de6610cdc53a212bdd6e0276d9c3215694fc46d12"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 29618,
      "digest": "sha256:724dc763905be41fc3f79ca1ff20e58945acd54d7b784638d28ea6b06ae9a107"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 48986733,
      "digest": "sha256:fd190c1a6475f413ee7f287018e2a345440b57a541779936e1083502d5a638b7"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 90867,
      "digest": "sha256:14163d527796b84ced95debfe05e73a545e122313c451cbf9202f8d64137e9a8"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 59097,
      "digest": "sha256:87a2d806c3528148d66209a515f032f0eb33a4482f6ff10faab1e746de204b9e"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 59947,
      "digest": "sha256:80071f9ed774d54eeaa75def6f84f08d115fed3bcd06183e24119cb7bafa86cb"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 5692,
      "digest": "sha256:863eee1bec4c3c0a6297f920b16525908c8a8f8b573985a00931cd7dc4051b9c"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 834641,
      "digest": "sha256:158dc2bb708ea44dc9f1af0e2583fd5d0a5d79044a511c44743476ce6cba1ebe"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 876746,
      "digest": "sha256:600c98360e080cdda6536309a366c1a7eed20a9ca00d23ee9ab7b4114225968f"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 44311,
      "digest": "sha256:d8fc108cfe22c9e5b29298bd0556ae60e82e5e8e57ddac32df1703ae6f1028d3"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 19327,
      "digest": "sha256:5c91ba2acd182029fc170a0c5d29da3613f1f9a916bf66808d595d9a241ea650"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 11082,
      "digest": "sha256:32f7015b94225950efeb307db93fbe30f275cdf588ce64570ea7ef018603e41b"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 14794,
      "digest": "sha256:3e74f2c81bd359addccc62f232959bdde9a8b4d42f63ee03ab110eca7dc509cd"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 173,
      "digest": "sha256:611a36c7572365584236f2e617354769600eece2597fc48b0b5284501be5714b"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 2322,
      "digest": "sha256:56b812a48e20bdc4a1e18637e445987e10673a4ad229d9ff5c4f1486ae1e1f8c"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 67331799,
      "digest": "sha256:d8a374451c7b8b69350fd51842879984d02952e6bc5cb2b4ece536b169216c47"
    }
  ]
}

[rhatdan]

@mtrmac @vrothberg PTAL

[mtrmac]

(The relevant OCI specification is the layers section of https://github.com/opencontainers/image-spec/blob/main/manifest.md , which says that only images using the appropriate OCI types can be expected to be interoperable.)

But that’s not obviously the relevant cause here: the quoted inspect output does not show anything OCI-related at all.

Please collect, and provide, a podman --log-level=debug … output of the pull and push commands.

[Timoses]

> podman pull --log-level=debug registry.local/project/image:1
INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level=debug registry.local/project/image:1)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/user/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/user/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/user/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 7
DEBU[0000] Pulling image registry.local/project/image:1 (policy: always)
DEBU[0000] Looking up image "registry.local/project/image:1" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "registry.local/project/image:1" ...
DEBU[0000] parsed reference into "[overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]@17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0"
DEBU[0000] Found image "registry.local/project/image:1" as "registry.local/project/image:1" in local containers storage
DEBU[0000] Found image "registry.local/project/image:1" as "registry.local/project/image:1" in local containers storage ([overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]@17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0)
DEBU[0000] exporting opaque data as blob "sha256:17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Attempting to pull candidate registry.local/project/image:1 for registry.local/project/image:1
DEBU[0000] parsed reference into "[overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]registry.local/project/image:1"
Trying to pull registry.local/project/image:1...
DEBU[0000] Copying source image //registry.local/project/image:1 to destination image [overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]registry.local/project/image:1
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "registry.local/project/image:1"
DEBU[0000] Found credentials for registry.local/project/image in credential helper containers-auth.json in file /run/user/1000/containers/auth.json
DEBU[0000]  No signature storage configuration found for registry.local/project/image:1, using built-in default file:///home/user/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.local
DEBU[0000] GET https://registry.local/v2/
DEBU[0000] Ping https://registry.local/v2/ status 401
DEBU[0000] GET https://registry.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull&service=harbor-registry
DEBU[0000] GET https://registry.local/v2/project/image/manifests/1
DEBU[0000] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0000] Using blob info cache at /home/user/.local/share/containers/cache/blob-info-cache-v1.boltdb
DEBU[0000] IsRunningImageAllowed for image docker:registry.local/project/image:1
DEBU[0000]  Using default policy section
DEBU[0000]  Requirement 0: allowed
DEBU[0000] Overall: allowed
DEBU[0000] Downloading /v2/project/image/blobs/sha256:17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0
DEBU[0000] GET https://registry.local/v2/project/image/blobs/sha256:17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0
Getting image source signatures
DEBU[0000] Reading /home/user/.local/share/containers/sigstore/project/image@sha256=034a5042010222fe554fd8fd3db3a3fbfd237cd09ee429b37dfa3e427131dc38/signature-1
DEBU[0000] Not looking for sigstore attachments: disabled by configuration
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0000] ... will first try using the original manifest unmodified
DEBU[0000] Checking if we can reuse blob sha256:80071f9ed774d54eeaa75def6f84f08d115fed3bcd06183e24119cb7bafa86cb: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0000] Checking if we can reuse blob sha256:724dc763905be41fc3f79ca1ff20e58945acd54d7b784638d28ea6b06ae9a107: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0000] Skipping blob sha256:80071f9ed774d54eeaa75def6f84f08d115fed3bcd06183e24119cb7bafa86cb (already present):
DEBU[0000] Checking if we can reuse blob sha256:5c16b2868d93fb9e0d11be3de6610cdc53a212bdd6e0276d9c3215694fc46d12: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0000] Skipping blob sha256:724dc763905be41fc3f79ca1ff20e58945acd54d7b784638d28ea6b06ae9a107 (already present):
DEBU[0000] Checking if we can reuse blob sha256:14163d527796b84ced95debfe05e73a545e122313c451cbf9202f8d64137e9a8: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0000] Checking if we can reuse blob sha256:863eee1bec4c3c0a6297f920b16525908c8a8f8b573985a00931cd7dc4051b9c: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0000] Skipping blob sha256:14163d527796b84ced95debfe05e73a545e122313c451cbf9202f8d64137e9a8 (already present):
DEBU[0000] Skipping blob sha256:863eee1bec4c3c0a6297f920b16525908c8a8f8b573985a00931cd7dc4051b9c (already present):
DEBU[0000] Checking if we can reuse blob sha256:fd190c1a6475f413ee7f287018e2a345440b57a541779936e1083502d5a638b7: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0000] Checking if we can reuse blob sha256:87a2d806c3528148d66209a515f032f0eb33a4482f6ff10faab1e746de204b9e: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0000] Skipping blob sha256:fd190c1a6475f413ee7f287018e2a345440b57a541779936e1083502d5a638b7 (already present):
DEBU[0000] Checking if we can reuse blob sha256:158dc2bb708ea44dc9f1af0e2583fd5d0a5d79044a511c44743476ce6cba1ebe: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0000] Skipping blob sha256:87a2d806c3528148d66209a515f032f0eb33a4482f6ff10faab1e746de204b9e (already present):
DEBU[0000] Checking if we can reuse blob sha256:600c98360e080cdda6536309a366c1a7eed20a9ca00d23ee9ab7b4114225968f: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0000] Checking if we can reuse blob sha256:d8fc108cfe22c9e5b29298bd0556ae60e82e5e8e57ddac32df1703ae6f1028d3: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0000] Checking if we can reuse blob sha256:5c91ba2acd182029fc170a0c5d29da3613f1f9a916bf66808d595d9a241ea650: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0000] Checking if we can reuse blob sha256:32f7015b94225950efeb307db93fbe30f275cdf588ce64570ea7ef018603e41b: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0000] Skipping blob sha256:5c91ba2acd182029fc170a0c5d29da3613f1f9a916bf66808d595d9a241ea650 (already present):
DEBU[0000] Skipping blob sha256:600c98360e080cdda6536309a366c1a7eed20a9ca00d23ee9ab7b4114225968f (already present):
DEBU[0000] Skipping blob sha256:158dc2bb708ea44dc9f1af0e2583fd5d0a5d79044a511c44743476ce6cba1ebe (already present):
DEBU[0000] Skipping blob sha256:32f7015b94225950efeb307db93fbe30f275cdf588ce64570ea7ef018603e41b (already present):
DEBU[0000] Skipping blob sha256:d8fc108cfe22c9e5b29298bd0556ae60e82e5e8e57ddac32df1703ae6f1028d3 (already present):
DEBU[0000] Skipping blob sha256:5c16b2868d93fb9e0d11be3de6610cdc53a212bdd6e0276d9c3215694fc46d12 (already present):
Copying blob 80071f9ed774 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 14163d527796 skipped: already exists
Copying blob 80071f9ed774 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 14163d527796 skipped: already exists
Copying blob 863eee1bec4c skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob 5c91ba2acd18 skipped: already exists
DEBU[0000] Checking if we can reuse blob sha256:56b812a48e20bdc4a1e18637e445987e10673a4ad229d9ff5c4f1486ae1e1f8c: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = false
Copying blob 80071f9ed774 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 14163d527796 skipped: already exists
Copying blob 863eee1bec4c skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob 5c91ba2acd18 skipped: already exists
Copying blob 600c98360e08 skipped: already exists
Copying blob 158dc2bb708e skipped: already exists
Copying blob 32f7015b9422 skipped: already exists
Copying blob d8fc108cfe22 skipped: already exists
Copying blob 5c16b2868d93 skipped: already exists
Copying blob 3e74f2c81bd3 skipped: already exists
Copying blob 56b812a48e20 skipped: already exists
Copying blob d8a374451c7b skipped: already exists
Copying blob 611a36c75723 skipped: already exists
Copying blob 80071f9ed774 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 14163d527796 skipped: already exists
Copying blob 863eee1bec4c skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob 5c91ba2acd18 skipped: already exists
Copying blob 600c98360e08 skipped: already exists
Copying blob 158dc2bb708e skipped: already exists
Copying blob 32f7015b9422 skipped: already exists
Copying blob d8fc108cfe22 skipped: already exists
Copying blob 5c16b2868d93 skipped: already exists
Copying blob 3e74f2c81bd3 skipped: already exists
Copying blob 56b812a48e20 skipped: already exists
Copying blob d8a374451c7b skipped: already exists
Copying blob 611a36c75723 skipped: already exists
DEBU[0000] No compression detected
DEBU[0000] Compression change for blob sha256:17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0 ("application/vnd.oci.image.config.v1+json") not supported
DEBU[0000] Using original blob without modification
Copying config 17a9ad7647 done
Writing manifest to image destination
Storing signatures
DEBU[0000] setting image creation date to 2023-09-08 10:19:37.061307605 +0000 UTC
DEBU[0000] reusing image ID "17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0"
DEBU[0000] saved image metadata "{\"signatures-sizes\":{\"sha256:034a5042010222fe554fd8fd3db3a3fbfd237cd09ee429b37dfa3e427131dc38\":[]}}"
DEBU[0000] added name "registry.local/project/image:1" to image "17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0"
DEBU[0000] Pulled candidate registry.local/project/image:1 successfully
DEBU[0000] Looking up image "17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0" in local containers storage
DEBU[0000] Trying "17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0" ...
DEBU[0000] parsed reference into "[overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]@17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0"
DEBU[0000] Found image "17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0" as "17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0" in local containers storage
DEBU[0000] Found image "17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0" as "17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0" in local containers storage ([overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]@17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0)
DEBU[0000] exporting opaque data as blob "sha256:17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0"
17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0
DEBU[0000] Called pull.PersistentPostRunE(podman pull --log-level=debug registry.local/project/image:1)
DEBU[0000] Shutting down engines

> podman push --log-level=debug registry2.local/project/image:1
INFO[0000] podman filtering at log level debug
DEBU[0000] Called push.PersistentPreRunE(podman push --log-level=debug registry2.local/project/image:1)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/user/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/user/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/user/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 7
DEBU[0000] Looking up image "registry2.local/project/image:1" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "registry2.local/project/image:1" ...
DEBU[0000] parsed reference into "[overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]@17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0"
DEBU[0000] Found image "registry2.local/project/image:1" as "registry2.local/project/image:1" in local containers storage
DEBU[0000] Found image "registry2.local/project/image:1" as "registry2.local/project/image:1" in local containers storage ([overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]@17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0)
DEBU[0000] exporting opaque data as blob "sha256:17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0"
DEBU[0000] Pushing image registry2.local/project/image:1 to registry2.local/project/image:1
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Copying source image [overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]@17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0 to destination image //registry2.local/project/image:1
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Found credentials for registry2.local/project/image in credential helper containers-auth.json in file /run/user/1000/containers/auth.json
DEBU[0000]  No signature storage configuration found for registry2.local/project/image:1, using built-in default file:///home/user/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry2.local
DEBU[0000] Using blob info cache at /home/user/.local/share/containers/cache/blob-info-cache-v1.boltdb
DEBU[0000] IsRunningImageAllowed for image containers-storage:[overlay@/home/user/.local/share/containers/storage]@17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0
DEBU[0000]  Using transport "containers-storage" policy section ""
DEBU[0000]  Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0000] ... will first try using the original manifest unmodified
DEBU[0000] Checking if we can reuse blob sha256:c9ce7c9bc47f3222d42eb90724ad442197afd339d59ee37d9a8439fe76e7a722: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar" = true
DEBU[0000] Checking /v2/project/image/blobs/sha256:c9ce7c9bc47f3222d42eb90724ad442197afd339d59ee37d9a8439fe76e7a722
DEBU[0000] GET https://registry2.local/v2/
DEBU[0000] Checking if we can reuse blob sha256:6982f59c9581025e24acfaf553f26adfa32612002c50b74252a8b87e2257c490: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar" = true
DEBU[0000] Checking /v2/project/image/blobs/sha256:6982f59c9581025e24acfaf553f26adfa32612002c50b74252a8b87e2257c490
DEBU[0000] Checking if we can reuse blob sha256:c40927c3b35c92eafda52388d51d479c3114c511e3b1bd7e0e5609ecde9f300f: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar" = true
DEBU[0000] Checking /v2/project/image/blobs/sha256:c40927c3b35c92eafda52388d51d479c3114c511e3b1bd7e0e5609ecde9f300f
DEBU[0000] Checking if we can reuse blob sha256:7690e228ab8bb83658fe2024e7dadc8d6dbc4a2af2f278b17f786ec5cd2a244d: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar" = true
DEBU[0000] Checking /v2/project/image/blobs/sha256:7690e228ab8bb83658fe2024e7dadc8d6dbc4a2af2f278b17f786ec5cd2a244d
DEBU[0000] Checking if we can reuse blob sha256:3ebefdeba72e21f85d0e006ecb9014794ecf01c1005beb63faa0cc1d879702b5: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar" = true
DEBU[0000] Checking /v2/project/image/blobs/sha256:3ebefdeba72e21f85d0e006ecb9014794ecf01c1005beb63faa0cc1d879702b5
DEBU[0000] Checking if we can reuse blob sha256:aa3c81db85a3fca8103f2f87fa9d1c72a27c6ace1d99ff5d054055391745f1e8: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar" = true
DEBU[0000] Checking /v2/project/image/blobs/sha256:aa3c81db85a3fca8103f2f87fa9d1c72a27c6ace1d99ff5d054055391745f1e8
DEBU[0000] Ping https://registry2.local/v2/ status 401
DEBU[0000] GET https://registry2.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull%2Cpush&service=harbor-registry
DEBU[0000] GET https://registry2.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull%2Cpush&service=harbor-registry
DEBU[0000] GET https://registry2.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull%2Cpush&service=harbor-registry
DEBU[0000] GET https://registry2.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull%2Cpush&service=harbor-registry
DEBU[0000] GET https://registry2.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull%2Cpush&service=harbor-registry
DEBU[0000] GET https://registry2.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull%2Cpush&service=harbor-registry
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:7690e228ab8bb83658fe2024e7dadc8d6dbc4a2af2f278b17f786ec5cd2a244d
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:6982f59c9581025e24acfaf553f26adfa32612002c50b74252a8b87e2257c490
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:c40927c3b35c92eafda52388d51d479c3114c511e3b1bd7e0e5609ecde9f300f
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:aa3c81db85a3fca8103f2f87fa9d1c72a27c6ace1d99ff5d054055391745f1e8
DEBU[0000] ... not present
DEBU[0000] Trying to reuse cached location sha256:87a2d806c3528148d66209a515f032f0eb33a4482f6ff10faab1e746de204b9e compressed with gzip in registry2.local/project/image
DEBU[0000] Checking /v2/project/image/blobs/sha256:87a2d806c3528148d66209a515f032f0eb33a4482f6ff10faab1e746de204b9e
DEBU[0000] GET https://registry2.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull%2Cpush&scope=repository%3Aproject%2Fimage%3Apull&service=harbor-registry
DEBU[0000] ... not present
DEBU[0000] Trying to reuse cached location sha256:fd190c1a6475f413ee7f287018e2a345440b57a541779936e1083502d5a638b7 compressed with gzip in registry2.local/project/image
DEBU[0000] Checking /v2/project/image/blobs/sha256:fd190c1a6475f413ee7f287018e2a345440b57a541779936e1083502d5a638b7
DEBU[0000] GET https://registry2.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull%2Cpush&scope=repository%3Aproject%2Fimage%3Apull&service=harbor-registry
DEBU[0000] ... not present
DEBU[0000] Trying to reuse cached location sha256:5c16b2868d93fb9e0d11be3de6610cdc53a212bdd6e0276d9c3215694fc46d12 compressed with gzip in registry2.local/project/image
DEBU[0000] Checking /v2/project/image/blobs/sha256:5c16b2868d93fb9e0d11be3de6610cdc53a212bdd6e0276d9c3215694fc46d12
DEBU[0000] GET https://registry2.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull%2Cpush&scope=repository%3Aproject%2Fimage%3Apull&service=harbor-registry
DEBU[0000] ... not present
DEBU[0000] Trying to reuse cached location sha256:724dc763905be41fc3f79ca1ff20e58945acd54d7b784638d28ea6b06ae9a107 compressed with gzip in registry2.local/project/image
DEBU[0000] Checking /v2/project/image/blobs/sha256:724dc763905be41fc3f79ca1ff20e58945acd54d7b784638d28ea6b06ae9a107
DEBU[0000] GET https://registry2.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull%2Cpush&scope=repository%3Aproject%2Fimage%3Apull&service=harbor-registry
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:c9ce7c9bc47f3222d42eb90724ad442197afd339d59ee37d9a8439fe76e7a722
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:3ebefdeba72e21f85d0e006ecb9014794ecf01c1005beb63faa0cc1d879702b5
DEBU[0000] ... not present
DEBU[0000] Trying to reuse cached location sha256:80071f9ed774d54eeaa75def6f84f08d115fed3bcd06183e24119cb7bafa86cb compressed with gzip in registry2.local/project/image
DEBU[0000] Checking /v2/project/image/blobs/sha256:80071f9ed774d54eeaa75def6f84f08d115fed3bcd06183e24119cb7bafa86cb
DEBU[0000] GET https://registry2.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull%2Cpush&scope=repository%3Aproject%2Fimage%3Apull&service=harbor-registry
DEBU[0000] ... not present
DEBU[0000] Trying to reuse cached location sha256:14163d527796b84ced95debfe05e73a545e122313c451cbf9202f8d64137e9a8 compressed with gzip in registry2.local/project/image
DEBU[0000] Checking /v2/project/image/blobs/sha256:14163d527796b84ced95debfe05e73a545e122313c451cbf9202f8d64137e9a8
DEBU[0000] GET https://registry2.local/service/token?account=user&scope=repository%3Aproject%2Fimage%3Apull%2Cpush&scope=repository%3Aproject%2Fimage%3Apull&service=harbor-registry
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:87a2d806c3528148d66209a515f032f0eb33a4482f6ff10faab1e746de204b9e
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:fd190c1a6475f413ee7f287018e2a345440b57a541779936e1083502d5a638b7
DEBU[0000] ... already exists
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:724dc763905be41fc3f79ca1ff20e58945acd54d7b784638d28ea6b06ae9a107
DEBU[0000] Skipping blob sha256:7690e228ab8bb83658fe2024e7dadc8d6dbc4a2af2f278b17f786ec5cd2a244d (already present):
DEBU[0000] Checking if we can reuse blob sha256:06ad6f66c9e8d20c85031348003dcdf84f89d219759a644b47728a0a3a694e3c: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar" = true
DEBU[0000] Checking /v2/project/image/blobs/sha256:06ad6f66c9e8d20c85031348003dcdf84f89d219759a644b47728a0a3a694e3c
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:06ad6f66c9e8d20c85031348003dcdf84f89d219759a644b47728a0a3a694e3c
Copying blob 87a2d806c352 skipped: already exists
DEBU[0000] ... not present
DEBU[0000] Trying to reuse cached location sha256:863eee1bec4c3c0a6297f920b16525908c8a8f8b573985a00931cd7dc4051b9c compressed with gzip in registry2.local/project/image
DEBU[0000] Checking /v2/project/image/blobs/sha256:863eee1bec4c3c0a6297f920b16525908c8a8f8b573985a00931cd7dc4051b9c
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:863eee1bec4c3c0a6297f920b16525908c8a8f8b573985a00931cd7dc4051b9c
DEBU[0000] ... already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
DEBU[0000] Checking if we can reuse blob sha256:ae6ed0164a73bb824da92e41b244b458a703de0076e6499936ca1929c39a414d: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar" = true
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
DEBU[0000] ... not present
DEBU[0000] Trying to reuse cached location sha256:158dc2bb708ea44dc9f1af0e2583fd5d0a5d79044a511c44743476ce6cba1ebe compressed with gzip in registry2.local/project/image
DEBU[0000] Checking /v2/project/image/blobs/sha256:158dc2bb708ea44dc9f1af0e2583fd5d0a5d79044a511c44743476ce6cba1ebe
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:158dc2bb708ea44dc9f1af0e2583fd5d0a5d79044a511c44743476ce6cba1ebe
DEBU[0000] ... already exists
DEBU[0000] Skipping blob sha256:c40927c3b35c92eafda52388d51d479c3114c511e3b1bd7e0e5609ecde9f300f (already present):
Copying blob 87a2d806c352 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 724dc763905b skipped: already exists
DEBU[0000] ... not present
DEBU[0000] Trying to reuse cached location sha256:600c98360e080cdda6536309a366c1a7eed20a9ca00d23ee9ab7b4114225968f compressed with gzip in registry2.local/project/image
DEBU[0000] Checking /v2/project/image/blobs/sha256:600c98360e080cdda6536309a366c1a7eed20a9ca00d23ee9ab7b4114225968f
DEBU[0000] HEAD https://registry2.local/v2/project/image/blobs/sha256:600c98360e080cdda6536309a366c1a7eed20a9ca00d23ee9ab7b4114225968f
DEBU[0000] ... already exists
DEBU[0000] Skipping blob sha256:06ad6f66c9e8d20c85031348003dcdf84f89d219759a644b47728a0a3a694e3c (already present):
DEBU[0000] Checking if we can reuse blob sha256:44898ed02a7d92ab4dc004bd286181eeef2ecdaceac5cf54622becb50bbe645a: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar" = true
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 863eee1bec4c skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 863eee1bec4c skipped: already exists
Copying blob 158dc2bb708e skipped: already exists
DEBU[0001] HEAD https://registry2.local/v2/project/image/blobs/sha256:14163d527796b84ced95debfe05e73a545e122313c451cbf9202f8d64137e9a8
DEBU[0001] ... not present
DEBU[0001] Trying to reuse cached location sha256:d8fc108cfe22c9e5b29298bd0556ae60e82e5e8e57ddac32df1703ae6f1028d3 compressed with gzip in registry2.local/project/image
DEBU[0001] Checking /v2/project/image/blobs/sha256:d8fc108cfe22c9e5b29298bd0556ae60e82e5e8e57ddac32df1703ae6f1028d3
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 863eee1bec4c skipped: already exists
Copying blob 158dc2bb708e skipped: already exists
Copying blob 600c98360e08 skipped: already exists
DEBU[0001] ... not present
DEBU[0001] Trying to reuse cached location sha256:32f7015b94225950efeb307db93fbe30f275cdf588ce64570ea7ef018603e41b compressed with gzip in registry2.local/project/image
Copying blob 87a2d806c352 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 863eee1bec4c skipped: already exists
Copying blob 158dc2bb708e skipped: already exists
Copying blob 600c98360e08 skipped: already exists
Copying blob 14163d527796 skipped: already exists
Copying blob d8fc108cfe22 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 863eee1bec4c skipped: already exists
Copying blob 158dc2bb708e skipped: already exists
Copying blob 600c98360e08 skipped: already exists
Copying blob 14163d527796 skipped: already exists
Copying blob d8fc108cfe22 skipped: already exists
Copying blob 5c91ba2acd18 skipped: already exists
DEBU[0001] ... not present
DEBU[0001] ... not present
DEBU[0001] Trying to reuse cached location sha256:611a36c7572365584236f2e617354769600eece2597fc48b0b5284501be5714b compressed with gzip in registry2.local/project/image
DEBU[0001] Checking /v2/project/image/blobs/sha256:611a36c7572365584236f2e617354769600eece2597fc48b0b5284501be5714b
DEBU[0001] HEAD https://registry2.local/v2/project/image/blobs/sha256:611a36c7572365584236f2e617354769600eece2597fc48b0b5284501be5714b
DEBU[0001] Trying to reuse cached location sha256:3e74f2c81bd359addccc62f232959bdde9a8b4d42f63ee03ab110eca7dc509cd compressed with gzip in registry2.local/project/image
DEBU[0001] Checking /v2/project/image/blobs/sha256:3e74f2c81bd359addccc62f232959bdde9a8b4d42f63ee03ab110eca7dc509cd
DEBU[0001] HEAD https://registry2.local/v2/project/image/blobs/sha256:3e74f2c81bd359addccc62f232959bdde9a8b4d42f63ee03ab110eca7dc509cd
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 863eee1bec4c skipped: already exists
Copying blob 158dc2bb708e skipped: already exists
Copying blob 600c98360e08 skipped: already exists
Copying blob 14163d527796 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob 87a2d806c352 skipped: already exists
Copying blob fd190c1a6475 skipped: already exists
Copying blob 724dc763905b skipped: already exists
Copying blob 863eee1bec4c skipped: already exists
Copying blob 158dc2bb708e skipped: already exists
Copying blob 600c98360e08 skipped: already exists
Copying blob 14163d527796 skipped: already exists
Copying blob d8fc108cfe22 skipped: already exists
Copying blob 5c91ba2acd18 skipped: already exists
Copying blob 32f7015b9422 skipped: already exists
Copying blob 5c16b2868d93 skipped: already exists
Copying blob 3e74f2c81bd3 skipped: already exists
Copying blob 611a36c75723 skipped: already exists
Copying blob 56b812a48e20 skipped: already exists
Copying blob 80071f9ed774 skipped: already exists
Copying blob d8a374451c7b skipped: already exists
DEBU[0001] Writing manifest using preferred type application/vnd.oci.image.manifest.v1+json failed: creating an updated image manifest: preparing updated manifest, layer "sha256:56b812a48e20bdc4a1e18637e445987e10673a4ad229d9ff5c4f1486ae1e1f8c": unsupported MIME type for compression: application/vnd.docker.image.rootfs.diff.tar.gzip
DEBU[0001] Looking up image "registry2.local/project/image:1" in local containers storage
DEBU[0001] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0001] Trying "registry2.local/project/image:1" ...
DEBU[0001] parsed reference into "[overlay@/home/user/.local/share/containers/storage+/run/user/1000/containers]@17a9ad7647bb61ef800f51f9109ae22fcc62754792c91ea574d12210c7fc4ab0"
DEBU[0001] Found image "registry2.local/project/image:1" as "registry2.local/project/image:1" in local containers storage
Error: creating an updated image manifest: preparing updated manifest, layer "sha256:56b812a48e20bdc4a1e18637e445987e10673a4ad229d9ff5c4f1486ae1e1f8c": unsupported MIME type for compression: application/vnd.docker.image.rootfs.diff.tar.gzip
DEBU[0001] Shutting down engines

[Timoses]

I pasted the wrong output of skopeo inspect in the bug description. Corrected!

[mtrmac]

All right, yes, that is the originally-linked problem about invalid OCI images with non-OCI layers.

[vrothberg]

    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 2322,
      "digest": "sha256:56b812a48e20bdc4a1e18637e445987e10673a4ad229d9ff5c4f1486ae1e1f8c"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 67331799,
      "digest": "sha256:d8a374451c7b8b69350fd51842879984d02952e6bc5cb2b4ece536b169216c47"
    }

I find it curious that podman pull works with that manifest as it's violating the OCI spec. I am personally in favor to continue error'ing out. Docker should as well.

[Timoses]

Could you explain why it says

application/vnd.oci.image.layer.v1.tar+gzip
Interchangeable and fully compatible mime-types:
application/vnd.docker.image.rootfs.diff.tar.gzip

Source: https://github.com/opencontainers/image-spec/blob/main/media-types.md#applicationvndociimagelayerv1targzip

This, to me, sounds like the layer media-types should be compatible with each other. Does it not?

[mtrmac]

No, that says how the formats are similar. I’m not going to interpret that as some kind of acceptance mandate.

[mtrmac]

I find it curious that podman pull works with that manifest as it's violating the OCI spec.

Neither the generic copy code nor the OCI code in general prevent that, because that code allows non-image OCI artifacts. It would have to be the c/storage destination that rejects non-image artifacts, or images with unexpected MIME types, and we just didn’t write that code (because we decompress by autodetection and don’t really consume the MIME types).

It’s very reasonable to argue that that’s a bug… OTOH starting to be more strict now could cause breakage, IIRC we have some history of that with MIME types.

---

… actually that same artifact-support code should handle unsupported MIME types:

DEBU[0000] Checking if we can reuse blob sha256:56b812a48e20bdc4a1e18637e445987e10673a4ad229d9ff5c4f1486ae1e1f8c: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = false

“compression [change supported] … false”!

I don’t see how we are triggering the compression decision here. While I still think the image is just invalid, there really might be a bug relevant for compressed artifacts.

(Note to self: The original report is with Podman 4.5.1, not the very latest.)

[Timoses]

Ok, so it should go one way or the other. Either, become more restrictive overall, or loosen up on the other end.

To my knowledge, I was able to copy the image with skopeo. So why is podman more restrictive here?

In any case: Even when becoming more strict (and therefore being more conform to spec), I personally would have to either

• ensure that we do not produce/use Images with mixed MIME types or
• use another tool (e.g. when the image at hand is built externally).

---

I'm struggling a lot with finding good sources for how to deal with this in general. There are several questions:

• How does it happen that a non-conforming image is being built in the first place? (Some tool, e.g. Docker?, seems allow this, or even is a source for this non-compliant behaviour)
• How to prevent it from happening?
• Is there a way to convert a "broken" image?

Perhaps podman is the only tool that is so restrictive? If that is the case then (on a Tools-Layer) either

• podman also has to become less restrictive or
• other tools (like Docker/Kaniko/...) have to be involved / made aware of the "broken" state, so it can be fixed overall.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mtrmac waiting for your response I believe.