zstd:chunked issues

[ckyrouac]

This took awhile to track down. I'm going to continue investigating but I wanted to document what I've found so far.

The failure happens when attempting a bootc install to-disk using an image built from a base image with at least one extra layer, e.g.

FROM quay.io/centos-bootc/centos-bootc-dev:stream9
RUN dnf install -y tmux

If the image is built locally bootc install to-disk works correctly. The failure happens when pushing the image to a repo (only tested with quay.io), clearing out the image from local storage via podman system prune --all, then running bootc install to-disk. Here's example output of the failure:

[test@fedora-39 ~]$ sudo podman run --pid=host --network=host --privileged --security-opt label=type:unconfined_t -v /var/lib/containers:/var/lib/containers -v .:/output -v /dev:/dev -e RUST_LOG=debug quay.io/ckyrouac/bootc-lldb bootc install to-disk --via-loopback --generic-image --skip-fetch-check /output/test.raw
Trying to pull quay.io/ckyrouac/bootc-lldb:latest...
Getting image source signatures
...
ERROR Installing to disk: Creating ostree deployment: Performing deployment: Importing: Parsing layer blob sha256:5d35bfe747b2c76a01310e69a14daa90baf352a11f990f73d4ce3e1917668719: Failed to invoke skopeo proxy method FinishPipe: remote error: corrupted blob, expecting sha256:dede69b8181937a545b87707fbe4ace2ee9936459ffd43b7ba957324861992a0

So, the OpenImage call to the skopeo proxy is failing.

The latest version of containers-common found in Fedora39/40 repos sets pull_options.enable_partial_images=true in /usr/share/containers/storage.conf. This is the change that started causing this error. Toggling enable_partial_images to false resolves the error. I'm not familiar enough with this stack to know the root cause of this yet. I'll continue digging but I'm sure someone else would be able to track this down a lot quicker if you think it's urgent.

[cgwalters]

The latest version of containers-common found in Fedora39/40 repos sets pull_options.enable_partial_images=true in /usr/share/containers/storage.conf. This is the change that started causing this error. Toggling enable_partial_images to false resolves the error.

Ugh. Fun...thanks for finding and debugging this.

[cgwalters]

It's actually really embarrassing that this wasn't caught by our CI, needs fixing

[cgwalters]

Actually wait this is the -dev image which is intentionally tracking git main, I don't think this has hit f40 or stream9 proper yet. I see a lot of activity in https://src.fedoraproject.org/rpms/containers-common/commits/rawhide and what I bet is happening here is those spec files are being pulled into the copr.

cc @rhatdan @lsm5

[cgwalters]

And yes, we need to add bootc test gating to containers-common and skopeo pretty soon.

[ckyrouac]

hmm interesting, earlier in the week this was happening regardless of which base image I used. Just went to verify that and now this bug only happens with the -dev base image.

[lsm5]

Actually wait this is the -dev image which is intentionally tracking git main, I don't think this has hit f40 or stream9 proper yet. I see a lot of activity in https://src.fedoraproject.org/rpms/containers-common/commits/rawhide and what I bet is happening here is those spec files are being pulled into the copr.

cc @rhatdan @lsm5

The last build of containers-common on the podman-next copr was an automatic rebuild of the rawhide sources from sometime back. I disabled this automatic rebuild after we got rawhide to a sane-enough state.

Let me know if you need an update to the fedora or copr rpm. I can do a one-off build.

We're currently working on a packit workflow from upstream c/common to downstream containers-common rpm, like we have for podman and the rest, with automatic builds going to podman-next right after every upstream commit to main. I'm hoping that change will land early next week.

[ckyrouac]

so this works now using any base image. I'm not sure what changed. I guess something in the base images or in quay.io?

[cgwalters]

@vrothberg

[henrywang]

Hi @cgwalters. We encountered this issue in QE CI environment many times in two days.
All bootc install to-existing-root tests failed due to this error. https://artifacts.osci.redhat.com/testing-farm/10a21cd4-b029-48fa-9c23-9848288a7065/
This issue can be found in fedora-bootc:40 image testing. But can't be found in CS9 and RHEL 9.4/9.5 bootc image testing.

ERROR: Installing to filesystem: Creating ostree deployment: Performing deployment: Importing: Parsing layer blob sha256:f34e1c1a6f0f3ac0450db999825e519b67ac7c36697ad80ecfa3672ff285dbbc: Failed to invoke skopeo proxy method FinishPipe: remote error: expected 69427364 bytes in blob, got 72333312

[cgwalters]

Ugh man, I think this is fallout from https://bodhi.fedoraproject.org/updates/FEDORA-2024-ab42dd0ffb which rolls in containers/storage@23ff5f8 which somehow breaks things...

EDIT: No, I was wrong, enable_partial_pulls = true also in containers-common-0.58.0-2.fc40.

---

And yes, the fact that there is no gating CI in any of

• containers/storage upstream
• dist-git merge requests
• bodhi

That covers the ostree-container path let this all sail right through.

[henrywang]

OH!!!. Since yesterday (Saturday), I can't run container inside quay.io/fedora/fedora:40. And reports Error: configure storage: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver error. I have to use fedora:39 image.

[cgwalters]

Hmm, at this very moment quay.io/fedora/fedora-bootc:40 with version=40.20240606.0 has containers-common-0.58.0-2.fc40.noarch which predates that change (as the date stamp implies).

So...hum, this must somehow relate to the host environment version. Ah yes, if we look at the logs from that test run I can see that inside the fedora cloud AMI we have 'Installed: containers-common-5:0.59.1-1.fc40.noarch'`.

@henrywang can you try patching the tests to do something like this as a sanity test:

$ sed -ie 's/enable_partial_images = "true"/enable_partial_images = "false"/' /usr/share/containers/storage.conf

EDIT: See above, I'm no longer confident the relevant change here was in containers-common.

[cgwalters]

I'm trying to reproduce this locally initially by hacking up my podman-machine environment, but no luck yet.

Another thing that actually changed pretty recently too is there's a new podman: https://bodhi.fedoraproject.org/updates/FEDORA-2024-ab42dd0ffb
And we're also now getting that in the host environment. Can you play with downgrading that in the host environment too?