Currently, we have a scheduled CI workflow that rebuilds all Docker images every Saturday morning. The strange thing is that they always have a different image digest (SHA-256 hash), even though none of the dependencies have changed in a week (no new patch versions of the target language or supporting software like Ruby or curl have been released).
I wanted to know why the hash is different every time, so I pulled two versions of the kaitai-ruby-4.0-linux-x86_64 image, one a week old from 2026-03-21 and the other currently latest from 2026-03-28:
$ podman images --digests '*ruby*'
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
ghcr.io/kaitai-io/kaitai-ruby-4.0-linux-x86_64 latest sha256:626b25df60e95fd8f18dd4948400b43eab14269980dbbab8bf4d8a5af4c058dc 9093d6ba972c 19 hours ago 1.14 GB
ghcr.io/kaitai-io/kaitai-ruby-4.0-linux-x86_64 <none> sha256:0129cebe0421c368d181c21f7bb9c566ac8fbe998c88a17f46249a7f1acffeb8 66a572af896b 7 days ago 1.14 GB
I used https://github.com/reproducible-containers/diffoci to show the differences:
$ diffoci --version
diffoci version v0.1.8
$ diffoci diff --pull never \
> podman://ghcr.io/kaitai-io/kaitai-ruby-4.0-linux-x86_64@sha256:0129cebe0421c368d181c21f7bb9c566ac8fbe998c88a17f46249a7f1acffeb8 \
> podman://ghcr.io/kaitai-io/kaitai-ruby-4.0-linux-x86_64@sha256:626b25df60e95fd8f18dd4948400b43eab14269980dbbab8bf4d8a5af4c058dc \
> | ( read -r; printf '%s\n' "$REPLY"; sort )
INFO[0000] Target platforms: [linux/amd64]
INFO[0000] Loading image "ghcr.io/kaitai-io/kaitai-ruby-4.0-linux-x86_64@sha256:0129cebe0421c368d181c21f7bb9c566ac8fbe998c88a17f46249a7f1acffeb8" from "podman"
INFO[0020] No images store for sha256:43bb0737a1a6900fe4af345cbbd71dae63d70997677c4162e574feff9f5afcaa
ghcr.io/kaitai io/kaitai ruby 4.0 linux saved
Importing elapsed: 9.9 s total: 0.0 B (0.0 B/s)
INFO[0020] Loading image "ghcr.io/kaitai-io/kaitai-ruby-4.0-linux-x86_64@sha256:626b25df60e95fd8f18dd4948400b43eab14269980dbbab8bf4d8a5af4c058dc" from "podman"
INFO[0041] No images store for sha256:a9eee2d68615e2b8200035b4faad6ce0da72c727067964d2eb8d77eb0bec2d1e
ghcr.io/kaitai io/kaitai ruby 4.0 linux saved
Importing elapsed: 8.5 s total: 0.0 B (0.0 B/s)
TYPE NAME INPUT-0 INPUT-1
Cfg ctx:/manifests-0/config/config ? ?
Desc application/vnd.docker.container.image.v1+json 66a572af896b2e1aa1f041afdab7448278ef24b315da88a5c6c937375585f861 9093d6ba972c55f39d5dae294ed5c3fa2f710b99fdc2da2a169671434e1997ff
Desc application/vnd.docker.distribution.manifest.v2+json 43bb0737a1a6900fe4af345cbbd71dae63d70997677c4162e574feff9f5afcaa a9eee2d68615e2b8200035b4faad6ce0da72c727067964d2eb8d77eb0bec2d1e
Desc application/vnd.docker.image.rootfs.diff.tar 1d017d34ef8d6a9c39cf1a9376e82c0143bc7f6b98efa63f76b71ec79351c77d 6dee259273261d635597cbdcae1c6551ea79b8d3425ec97982f59bf98dbbedb7
Desc application/vnd.docker.image.rootfs.diff.tar c2f770c6c2a4c77b0b3f1a226d34d9d6ccc21534d0cbddee9c795d28b049d156 8b24c5a98f7557837a5d6cdb6ec71070a49089a4ee7c674e53f564ec48386165
Desc application/vnd.oci.image.index.v1+json 875ba682abf522183ca60e423a1e94f9c665957a5e9c80c44a9337815ea110ee ec2705d45ac21ea038a0a7c5af99fe9c90e6201223d3ec07e682177ed7dce305
File prepare 2026-03-21 05:46:43 +0100 CET 2026-03-28 06:02:55 +0100 CET
File prepare-alpine-init 2026-03-21 05:46:43 +0100 CET 2026-03-28 06:02:55 +0100 CET
File prepare-alpine-ruby 2026-03-21 05:46:43 +0100 CET 2026-03-28 06:02:55 +0100 CET
File prepare-alpine-uninit 2026-03-21 05:46:43 +0100 CET 2026-03-28 06:02:55 +0100 CET
File prepare-apt-ruby 2026-03-21 05:46:43 +0100 CET 2026-03-28 06:02:55 +0100 CET
File prepare-apt-uninit 2026-03-21 05:46:43 +0100 CET 2026-03-28 06:02:55 +0100 CET
File root/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File root/.cache/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File root/.cache/gem/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File root/.cache/gem/specs/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File root/.cache/gem/specs/index.rubygems.org%443/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File root/.cache/gem/specs/index.rubygems.org%443/quick/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File root/.cache/gem/specs/index.rubygems.org%443/quick/Marshal.4.8/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File root/.cache/gem/specs/index.rubygems.org%443/quick/Marshal.4.8/diff-lcs-1.6.2.gemspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File root/.cache/gem/specs/index.rubygems.org%443/quick/Marshal.4.8/rspec-3.13.2.gemspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File root/.cache/gem/specs/index.rubygems.org%443/quick/Marshal.4.8/rspec-core-3.13.6.gemspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File root/.cache/gem/specs/index.rubygems.org%443/quick/Marshal.4.8/rspec-expectations-3.13.5.gemspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File root/.cache/gem/specs/index.rubygems.org%443/quick/Marshal.4.8/rspec-mocks-3.13.8.gemspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File root/.cache/gem/specs/index.rubygems.org%443/quick/Marshal.4.8/rspec-support-3.13.7.gemspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/bin/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/bin/htmldiff 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/bin/ldiff 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/bin/rspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/build_info/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/cache/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/cache/diff-lcs-1.6.2.gem 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/cache/rspec-3.13.2.gem 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/cache/rspec-core-3.13.6.gem 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/cache/rspec-expectations-3.13.5.gem 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
(...)
File usr/local/bundle/gems/rspec-support-3.13.7/lib/rspec/support/spec/with_isolated_stderr.rb 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/gems/rspec-support-3.13.7/lib/rspec/support/version.rb 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/gems/rspec-support-3.13.7/lib/rspec/support/warnings.rb 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/gems/rspec-support-3.13.7/lib/rspec/support/with_keywords_when_needed.rb 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/plugins/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/specifications/ 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/specifications/diff-lcs-1.6.2.gemspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/specifications/rspec-3.13.2.gemspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/specifications/rspec-core-3.13.6.gemspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/specifications/rspec-expectations-3.13.5.gemspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/specifications/rspec-mocks-3.13.8.gemspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File usr/local/bundle/specifications/rspec-support-3.13.7.gemspec 2026-03-21 05:47:01 +0100 CET 2026-03-28 06:03:13 +0100 CET
File validate 2026-03-21 05:46:43 +0100 CET 2026-03-28 06:02:55 +0100 CET
File validate-version 2026-03-21 05:46:43 +0100 CET 2026-03-28 06:02:55 +0100 CET
Idx ctx:/index ? ?
Mani ctx:/manifests-0/manifest ? ?
As the output above suggests, the only difference between the files in the old and new Docker image is the modification time (i.e., the file system metadata). The contents of the files are identical.
This is really annoying - these changes in timestamps are just distracting noise that only makes it harder to tell when actual changes in the Docker images occur. Therefore, I suggest that we take some steps to make our building process more (bit-for-bit) reproducible. Even just the basic step of setting SOURCE_DATE_EPOCH as shown in https://docs.docker.com/build/ci/github-actions/reproducible-builds/ would likely go a long way. I think the SOURCE_DATE_EPOCH: 0 setting is fine. That will set all timestamps to 1970-01-01 00:00:00, which is a bit weird, but it shouldn't matter at all.
If setting the SOURCE_DATE_EPOCH variable proves insufficient, we can continue with some recommendations from https://dangerzone.rocks/news/2026-03-02-repro-build/.
Currently, we have a scheduled CI workflow that rebuilds all Docker images every Saturday morning. The strange thing is that they always have a different image digest (SHA-256 hash), even though none of the dependencies have changed in a week (no new patch versions of the target language or supporting software like Ruby or curl have been released).
I wanted to know why the hash is different every time, so I pulled two versions of the
kaitai-ruby-4.0-linux-x86_64image, one a week old from 2026-03-21 and the other currentlylatestfrom 2026-03-28:I used https://github.com/reproducible-containers/diffoci to show the differences:
As the output above suggests, the only difference between the files in the old and new Docker image is the modification time (i.e., the file system metadata). The contents of the files are identical.
This is really annoying - these changes in timestamps are just distracting noise that only makes it harder to tell when actual changes in the Docker images occur. Therefore, I suggest that we take some steps to make our building process more (bit-for-bit) reproducible. Even just the basic step of setting
SOURCE_DATE_EPOCHas shown in https://docs.docker.com/build/ci/github-actions/reproducible-builds/ would likely go a long way. I think theSOURCE_DATE_EPOCH: 0setting is fine. That will set all timestamps to1970-01-01 00:00:00, which is a bit weird, but it shouldn't matter at all.If setting the
SOURCE_DATE_EPOCHvariable proves insufficient, we can continue with some recommendations from https://dangerzone.rocks/news/2026-03-02-repro-build/.