Skip to content

Conversation

@djgalloway
Copy link
Contributor

This might explain a lot of the storage weirdness we've been observing. If a container image layer is create in rootless on podman on one machine, that image layer is restored to another host with different subuid or subgid, errors can occur.

@dmick
Copy link
Member

dmick commented Dec 24, 2025

mentioned in slack, but adding here for completeness:

It seems to be the case that "container images" (like you'd push) do not contain any info about the subuid/subgid mapping used to create them; they have container-internal UIDs in them that will get mapped when instantiated into a container. Do you remember what it was that broke exactly?

@djgalloway
Copy link
Contributor Author

mentioned in slack, but adding here for completeness:

It seems to be the case that "container images" (like you'd push) do not contain any info about the subuid/subgid mapping used to create them; they have container-internal UIDs in them that will get mapped when instantiated into a container. Do you remember what it was that broke exactly?

* deleting layer "c4f15df1d80592c556020df27ca825130761cc8d71c118596d2c875a2e7305ec": unlinkat /home/jenkins-build/.local/share/containers/storage/overlay/c4f15df1d80592c556020df27ca825130761cc8d71c118596d2c875a2e7305ec/diff/usr: permission denied

time="2026-02-05T18:42:15Z" level=warning msg="Found incomplete layer \"e1503a6fd02d8ca120b20f3e7c96d1cc7c4c0add197548c6f53609fa7eb48c9f\", deleting it"

time="2026-02-05T18:42:15Z" level=warning msg="Found incomplete layer \"33d17dbb13caaf24ec94c050113b5e9b1829b9325fab0d7c150d984d9d22bb05\", deleting it"

time="2026-02-05T18:42:15Z" level=warning msg="Found incomplete layer \"99dcba4964b1a542a440e089096714bb21196c137ec6e3e7cf4a6c271261701e\", deleting it"

time="2026-02-05T18:42:15Z" level=warning msg="Found incomplete layer \"3c887834c9af32ed2b7fceb06e5f4a384ba7f9341f15c124c99fe92a0c67988c\", deleting it"

time="2026-02-05T18:42:15Z" level=warning msg="Found incomplete layer \"d7c657a5f06c8687dd8feaf400d8f6149bd8f3b6b4a08633c5e0652e771deb6c\", deleting it"

time="2026-02-05T18:42:15Z" level=warning msg="Found incomplete layer \"c4f15df1d80592c556020df27ca825130761cc8d71c118596d2c875a2e7305ec\", deleting it"

real	0m1.296s

user	0m0.919s

sys	0m0.927s

script returned exit code 125

This might explain a lot of the storage weirdness we've been observing.  If a container image layer is create in rootless on podman on one machine, that image layer is restored to another host with *different* subuid or subgid, errors can occur.

Signed-off-by: David Galloway <david.galloway@ibm.com>
`chown`ing the podman storage directory is problematic when using rootless podman

Signed-off-by: David Galloway <david.galloway@ibm.com>
Signed-off-by: David Galloway <david.galloway@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants