Unable to find an image with digest `622c60f3d0944014d5fd45df24142eee17c73d1b9ceca6dce636fc5bf9a43a2c`

[apyrgio]

What happened?

A user has reported the following error in their Debian installation:

dangerzone.errors.ImageNotPresentException: Unable to find an image with digest 622c60f3d0944014d5fd45df24142eee17c73d1b9ceca6dce636fc5bf9a43a2c

What is the problem?

We have reproduced it internally, and the problem is that Podman may report a different digest than the expected one, if multi-arch images are involved. More specifically:

• if a user pulls the remote Dangerzone image (ghcr.io/freedomofopress/dangerzone/v1:latest), the reported image digest will be sha256:527da9012343066523c5561af64328dd13f79bf74b6c612d31fa8563b2e8ae2a. This is the digest of the multi-arch manifest.
• If they load the bundled container image for the 0.11.0 release , the image digest will be sha256:622c60f3d0944014d5fd45df24142eee17c73d1b9ceca6dce636fc5bf9a43a2c. This the digest of the linux/amd64 manifest.
• Finally, if they pull first the remote Dangerzone image, and then somehow trigger Dangerzone to load the bundled Dangerzone image, Podman will keep reporting the digest of the multi-arch image, instead of the linux/amd64 one. This is what confuses Dangerzone and throws an error. Note that this is a bug, but not a security issue.

Who is affected?

Our current understanding is that to trigger this bug, a user needs to:

1. Install Dangerzone 0.10.0, accept updates, and fetch the latest container image.
2. Remove the ~/.local/share/dangerzone directory, but not the pulled Dangerzone image.
3. Install Dangerzone 0.11.0 (the variant that bundles the container image) and run Dangerzone. You should trigger this error.

This is not a usual combination since the user needs to remove an internal Dangerzone directory. In principle though, it affects users across all OSes.

How to reproduce?

A programmatic way to reproduce this on Linux (easier if you are a dev) is:

user@host:~$ ./dev_scripts/env.py --distro debian --version trixie build --full --prod
user@host:~$ ./dev_scripts/env.py --distro debian --version trixie run bash
user@container:~$ rm -rf ~/.local/share/dangerzone/
user@container:~$ podman image prune -af
user@container:~$ podman pull ghcr.io/freedomofpress/dangerzone/v1
user@container:~$ dangerzone
<failure>

Workaround

If you remove the pulled Dangerzone image, things should work again:

podman rmi ghcr.io/freedomofpress/dangerzone/v1:latest

Linux distribution

Debian Trixie

Dangerzone version

0.11.0

Podman info

user@dangerzone:~$ podman version
Client:        Podman Engine
Version:       5.4.2
API Version:   5.4.2
Go Version:    go1.24.4
Built:         Sun Dec 21 18:42:01 2025
Build Origin:  Debian
OS/Arch:       linux/amd64

user@dangerzone:~$ podman info -f 'json'
{
  "host": {
    "arch": "amd64",
    "buildahVersion": "1.39.3",
    "cgroupManager": "cgroupfs",
    "cgroupVersion": "v2",
    "cgroupControllers": [
      "cpu",
      "memory",
      "pids"
    ],
    "conmon": {
      "package": "conmon_2.1.12-4_amd64",
      "path": "/usr/bin/conmon",
      "version": "conmon version 2.1.12, commit: unknown"
    },
    "cpus": 8,
    "cpuUtilization": {
      "userPercent": 6.84,
      "systemPercent": 1.79,
      "idlePercent": 91.37
    },
    "databaseBackend": "sqlite",
    "distribution": {
      "distribution": "debian",
      "version": "13",
      "codename": "trixie"
    },
    "eventLogger": "file",
    "freeLocks": 2048,
    "hostname": "dangerzone",
    "idMappings": {
      "gidmap": [
        {
          "container_id": 0,
          "host_id": 1000,
          "size": 1
        },
        {
          "container_id": 1,
          "host_id": 2000,
          "size": 2000
        }
      ],
      "uidmap": [
        {
          "container_id": 0,
          "host_id": 1000,
          "size": 1
        },
        {
          "container_id": 1,
          "host_id": 2000,
          "size": 2000
        }
      ]
    },
    "kernel": "7.0.14-arch1-1",
    "logDriver": "k8s-file",
    "memFree": 9679360000,
    "memTotal": 33358118912,
    "networkBackend": "netavark",
    "networkBackendInfo": {
      "backend": "netavark",
      "version": "netavark 1.14.0",
      "package": "netavark_1.14.0-2_amd64",
      "path": "/usr/lib/podman/netavark",
      "dns": {
        "version": "aardvark-dns 1.14.0",
        "package": "aardvark-dns_1.14.0-3_amd64",
        "path": "/usr/lib/podman/aardvark-dns"
      }
    },
    "ociRuntime": {
      "name": "crun",
      "package": "crun_1.21-1_amd64",
      "path": "/usr/bin/crun",
      "version": "crun version 1.21\ncommit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88\nrundir: /tmp/storage-run-1000/crun\nspec: 1.0.0\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +WASM:wasmedge +YAJL"
    },
    "os": "linux",
    "remoteSocket": {
      "path": "/tmp/storage-run-1000/podman/podman.sock",
      "exists": true
    },
    "rootlessNetworkCmd": "pasta",
    "serviceIsRemote": false,
    "security": {
      "apparmorEnabled": false,
      "capabilities": "CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT",
      "rootless": true,
      "seccompEnabled": true,
      "seccompProfilePath": "/usr/share/containers/seccomp.json",
      "selinuxEnabled": false
    },
    "slirp4netns": {
      "executable": "/usr/bin/slirp4netns",
      "package": "slirp4netns_1.2.1-1.1_amd64",
      "version": "slirp4netns version 1.2.1\ncommit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194\nlibslirp: 4.8.0\nSLIRP_CONFIG_VERSION_MAX: 5\nlibseccomp: 2.6.0"
    },
    "pasta": {
      "executable": "/usr/bin/pasta",
      "package": "passt_0.0~git20250503.587980c-2+deb13u1_amd64",
      "version": ""
    },
    "swapFree": 0,
    "swapTotal": 0,
    "uptime": "2h 58m 19.00s (Approximately 0.08 days)",
    "variant": "",
    "linkmode": "dynamic"
  },
  "store": {
    "configFile": "/home/user/.config/containers/storage.conf",
    "containerStore": {
      "number": 0,
      "paused": 0,
      "running": 0,
      "stopped": 0
    },
    "graphDriverName": "overlay",
    "graphOptions": {

    },
    "graphRoot": "/home/user/.local/share/containers/storage",
    "graphRootAllocated": 737595678720,
    "graphRootUsed": 613539483648,
    "graphStatus": {
      "Backing Filesystem": "extfs",
      "Native Overlay Diff": "true",
      "Supports d_type": "true",
      "Supports shifting": "false",
      "Supports volatile": "true",
      "Using metacopy": "false"
    },
    "imageCopyTmpDir": "/var/tmp",
    "imageStore": {
      "number": 1
    },
    "runRoot": "/tmp/storage-run-1000/containers",
    "volumePath": "/home/user/.local/share/containers/storage/volumes",
    "transientStore": false
  },
  "registries": {

  },
  "plugins": {
    "volume": [
      "local"
    ],
    "network": [
      "bridge",
      "macvlan",
      "ipvlan"
    ],
    "log": [
      "k8s-file",
      "none",
      "passthrough",
      "journald"
    ],
    "authorization": null
  },
  "version": {
    "APIVersion": "5.4.2",
    "Version": "5.4.2",
    "GoVersion": "go1.24.4",
    "GitCommit": "",
    "BuiltTime": "Sun Dec 21 18:42:01 2025",
    "Built": 1766335321,
    "BuildOrigin": "Debian",
    "OsArch": "linux/amd64",
    "Os": "linux"
  }
}

user@dangerzone:~$ podman images
REPOSITORY                            TAG         IMAGE ID      CREATED     SIZE
ghcr.io/freedomofpress/dangerzone/v1  latest      9f3234b9c53e  6 days ago  1.07 GB

Document conversion logs

N/A

Additional info

No response

[almet]

Does downloading the latest image with the dangerzone-image utility solves this?

like this:

dangerzone-image upgrade

[apyrgio]

Good question. I tried it out locally, and it doesn't solve it if you have already pulled the multi-arch Dangerzone image somehow. If you haven't updated yet and run dangerzone-image, it works even after you do rm -rf ~/.local/share/dangerzone/!

[apyrgio]

I just found another way to reproduce this issue, that does not involve remove the last log index.

1. Install a version of Dangerzone that includes the container image.
2. Disable update checks in Dangerzone at some point between 0.10.0 and 0.11.0
3. Pull Dangerzone images with podman pull ghcr.io/freedomofpress/dangerzone/v1:latest, which does not update the last_log_index.
4. Install the dangerzone-full package, which will trigger an upgrade from the bundled container image.

It's still difficult to trigger, but does not require something as intrusive as rm -rf ~/.local/share/dangerzone/.

[apyrgio]

Fixing this issue is non-trivial. The main difficulty is that in various places in the code, we need to:

• Find the image ID from a digest.
• Find the digests of all local Dangerzone images.
• Clean older images by digest, except the digest we're interested in.
• Read the stored signatures for a digest.

Each case may have subtle issues depending on the type of the image we have downloaded (multi-arch vs single-platform), and the signatures we have stored locally.

[apyrgio]

I have tried to tackle the above in the test/1516-multi-arch-digests branch. This branch is WIP (tests definitely don't pass), but while working on this issue, I realized that there is an extra issue we should fix, which is how to handle the case of an image with multiple digests, but only a single local signature.

I'm about ~500LoCs in for what originally looked like a simple issue. I may need to re-evaluate the approach of going with multiple digests, since it creates more headaches than it solves.

[apyrgio]

Note: this simple fix:

diff --git a/dangerzone/container_utils.py b/dangerzone/container_utils.py
index 329321c7..aee5efbc 100644
--- a/dangerzone/container_utils.py
+++ b/dangerzone/container_utils.py
@@ -370,6 +370,12 @@ def get_image_id_by_digest(digest: str) -> str:
     return filtered_images[0]
 
 
+def tag_image_from_ref(ref: str, tag: str) -> None:
+    """Tag a container image by its name:tag reference."""
+    podman = init_podman_command()
+    podman.run(["tag", ref, tag])
+
+
 def expected_image_name() -> str:
     image_name_path = get_resource_path("image-name.txt")
     return image_name_path.read_text().strip("\n")
diff --git a/dangerzone/updater/signatures.py b/dangerzone/updater/signatures.py
index 8938c7d0..3272b140 100644
--- a/dangerzone/updater/signatures.py
+++ b/dangerzone/updater/signatures.py
@@ -338,10 +338,10 @@ def upgrade_container_image_airgapped(
         expected_manifest["manifests"][0].get("digest").replace("sha256:", "")
     )
 
-    runtime.load_image_tarball(container_tar)
+    image_ref = runtime.load_image_tarball(container_tar)
     # Apply the tag manually here, since images downloaded with `cosign download`
     # do not come with the tags attached.
-    runtime.tag_image_by_digest(image_digest, image_name)
+    runtime.tag_image_from_ref(image_ref.removeprefix("sha256:"), image_name)
 
     try:
         verify_signatures(signatures, image_digest)

fails as well due to this:

[DEBUG] Digest to keep: 622c60f3d0944014d5fd45df24142eee17c73d1b9ceca6dce636fc5bf9a43a2c
[INFO] Running: /usr/bin/podman image list --format '{{ .Digest }}' ghcr.io/freedomofpress/dangerzone/v1
[INFO] sha256:527da9012343066523c5561af64328dd13f79bf74b6c612d31fa8563b2e8ae2a

[INFO]
[INFO] Process returncode: 0
[DEBUG] Digests installed: ['sha256:527da9012343066523c5561af64328dd13f79bf74b6c612d31fa8563b2e8ae2a']
[WARNING] Deleting container images: ghcr.io/freedomofpress/dangerzone/v1@sha256:527da9012343066523c5561af64328dd13f79bf74b6c612d31fa8563b2e8ae2a
[INFO] Running: /usr/bin/podman rmi --force ghcr.io/freedomofpress/dangerzone/v1@sha256:527da9012343066523c5561af64328dd13f79bf74b6c612d31fa8563b2e8ae2a
[INFO] Untagged: ghcr.io/freedomofpress/dangerzone/v1:latest
Deleted: 9f3234b9c53e62038bc7d0daf374a283b2aa279ea58771552ce0204abb0a47d6

[INFO]
[INFO] Process returncode: 0
[INFO] Task 'Configuring Dangerzone sandbox' completed successfully!
[INFO] Successfully finished all Dangerzone startup tasks
[DEBUG] Startup tasks have finished
[INFO] Assigning ID 'HYDHoE' to doc '/home/user/dangerzone/tests/test_docs/sample-doc-safe-safe-safe-page-3-safe-safe.pdf'
[DEBUG] Removing all documents
[DEBUG] Starting conversion
[DEBUG] Marking doc HYDHoE as 'converting'
[INFO] Running: /usr/bin/podman images ghcr.io/freedomofpress/dangerzone/v1 --format '{{.Digest}}'
[INFO]
[INFO]
[INFO] Process returncode: 0
[ERROR] An exception occurred while converting document 'HYDHoE'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/dangerzone/isolation_provider/base.py", line 146, in convert
    with self.doc_to_pixels_proc(document) as conversion_proc:
         ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/usr/lib/python3.13/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/usr/lib/python3/dist-packages/dangerzone/isolation_provider/base.py", line 430, in doc_to_pixels_proc
    p = self.start_doc_to_pixels_proc(document)
  File "/usr/lib/python3/dist-packages/dangerzone/isolation_provider/container.py", line 133, in start_doc_to_pixels_proc
    return self.exec_container(command, name=name)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dangerzone/isolation_provider/container.py", line 89, in exec_container
    image_digest = container_utils.get_local_image_digest()
  File "/usr/lib/python3/dist-packages/dangerzone/container_utils.py", line 412, in get_local_image_digest
    raise errors.ImageNotPresentException(
        f"The image {expected_image} does not exist locally"
    )
dangerzone.errors.ImageNotPresentException: The image ghcr.io/freedomofpress/dangerzone/v1 does not exist locally

Here, we see Dangerzone getting confused later on and removing the image, because the digest it found locally did not match the expected one.

[apyrgio]

Getting past this error (see test/1516-multi-arch-digests-2) means that we hit a different one down the line:

[INFO] Running: /usr/bin/podman images ghcr.io/freedomofpress/dangerzone/v1 --format '{{.Digest}}'
[INFO] sha256:527da9012343066523c5561af64328dd13f79bf74b6c612d31fa8563b2e8ae2a

[INFO]
[INFO] Process returncode: 0
[DEBUG] Image digest: 527da9012343066523c5561af64328dd13f79bf74b6c612d31fa8563b2e8ae2a
[ERROR] An exception occurred while converting document 'KH9uxU'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/dangerzone/isolation_provider/base.py", line 146, in convert
    with self.doc_to_pixels_proc(document) as conversion_proc:
         ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/usr/lib/python3.13/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/usr/lib/python3/dist-packages/dangerzone/isolation_provider/base.py", line 430, in doc_to_pixels_proc
    p = self.start_doc_to_pixels_proc(document)
  File "/usr/lib/python3/dist-packages/dangerzone/isolation_provider/container.py", line 133, in start_doc_to_pixels_proc
    return self.exec_container(command, name=name)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dangerzone/isolation_provider/container.py", line 91, in exec_container
    verify_local_image(image_digest=image_digest)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dangerzone/updater/signatures.py", line 534, in verify_local_image
    load_and_verify_signatures(image_digest, pubkey)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dangerzone/updater/signatures.py", line 450, in load_and_verify_signatures
    raise errors.LocalSignatureNotFound(msg)
dangerzone.updater.errors.LocalSignatureNotFound: Cannot find a '/home/user/.local/share/dangerzone/signatures/f8d4fdda2a40d010a1d7d4024e78cecc14d9ab48c89301a901df960a0c47a074/527da9012343066523c5561af64328dd13f79bf74b6c612d31fa8563b2e8ae2a.json' file. You might need to download the image signatures first.

That's essentially the issue I mentioned above:

[...] while working on this issue, I realized that there is an extra issue we should fix, which is how to handle the case of an image with multiple digests, but only a single local signature.

This essentially boils down to how much we trust the .RepoDigests field, which is the only place where the ghcr.io/freedomofpress/dangerzone/v1@sha256:622c60f3d0944014d5fd45df24142eee17c73d1b9ceca6dce636fc5bf9a43a2c is reported. Alternatively, we may have to bundle multiple signatures in our single-platform images; one for the platform, and one for the root manifest.

I think that's something we need to discuss with Alexis. So far, we haven't received any other user report on this issue, so I'm fine with not including it in the 0.11.1 milestone.

[almet]

When we detect a mismatch between the reported container image digest and the signatures, we could also delete the container image and reinstall it, if (and only if) we have a way to install the signatures first.

I believe that would solve this issue. That would mean:

1. In the installer, add this case:

   elif len(podman_images) > 1 or not has_signatures_for_digest(podman_images[0]):
       # installed image and signatures disagree, the case reported here.
       # This should get the image reinstalled.
       local_log_index = 0

2. In the case of an airgapped installation, do the following:

   • always verify the signatures first (from the container)
   • clear the old images
   • load the image tarball.

I believe this would help solve this case.

We could as well include the signatures of the root image in the tarball, as you proposed @apyrgio.