Fix podman network backend fallback on RHEL 9 (HMS-8783)#1365
Conversation
thozza
force-pushed
the
el9-podman-network_backend-fix
branch
from
0cf9369 to
551bc50
Compare
achilleas-k
left a comment
achilleas-k
left a comment
There was a problem hiding this comment.
Really nice! Thank.
The last commit message is a bit too long. Can you make it shorter?
Will do. I hoped that GH will mark this PR as a Draft. 😇 My plan is first to see where the consistency check fails, before pushing any fix for RHEL-9. |
thozza
force-pushed
the
el9-podman-network_backend-fix
branch
2 times, most recently
from
d27bc35 to
4b9ecd5
Compare
thozza
force-pushed
the
el9-podman-network_backend-fix
branch
2 times, most recently
from
d647bc5 to
341bda0
Compare
|
This PR is stale because it has been open 30 days with no activity. Remove "Stale" label or comment or this will be closed in 7 days. |
|
Do we still want this? EDIT: Nvm, noticed you un-staled it. |
This comment was marked as outdated.
This comment was marked as outdated.
thozza
changed the title
network_backend consistency when embedding containersnetwork_backend consistency when embedding containers (HMS-8783)
This comment was marked as outdated.
This comment was marked as outdated.
|
boop |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
This PR is stale because it had no activity for the past 30 days. Remove the "Stale" label or add a comment, otherwise this PR will be closed in 7 days. |
thozza
force-pushed
the
el9-podman-network_backend-fix
branch
from
341bda0 to
219ac1e
Compare
|
I just pushed a reworked WIP I had locally, since I won't have time to finish it in the upcoming weeks. That said, I didn't want to keep it just locally... |
|
This PR is stale because it had no activity for the past 30 days. Remove the "Stale" label or add a comment, otherwise this PR will be closed in 7 days. |
|
This PR is stale because it had no activity for the past 30 days. Remove the "Stale" label or add a comment, otherwise this PR will be closed in 7 days. |
thozza
force-pushed
the
el9-podman-network_backend-fix
branch
6 times, most recently
from
8d4cf99 to
8631167
Compare
thozza
changed the title
network_backend consistency when embedding containers (HMS-8783)
Verify that containers listed in the blueprint are actually present in the booted image's podman storage. Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Verify that rootful and rootless podman report the same network backend. When containers are embedded as root into the image (the default behavior), some podman versions interpret the existing storage as a migration and fall back to 'cni' for rootful only, leaving rootless on 'netavark'. In practice, the desired behavior is that podman uses the same network backend, regardless if there is an embedded container or not. Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Certain versions of Podman (notably on RHEL 9) fall back to the legacy 'cni' network backend when they find existing container images in the system storage, assuming a migration from an older version. This is problematic for disk images that embed containers as a customization, because the legacy backend packages are not installed by default. Add a NetworkBackend type and a helper to generate the /var/lib/containers/storage/defaultNetworkBackend file, which tells Podman which backend to use and prevents the unwanted fallback. Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a new ImageConfig option that specifies the default network backend for Podman. When set and the image embeds container images, the value is written to /var/lib/containers/storage/defaultNetworkBackend during image build. This prevents Podman from falling back to the legacy 'cni' backend when it finds pre-existing container images in storage, which it interprets as a system migration. Signed-off-by: Tomáš Hozza <thozza@redhat.com>
RHEL 9 Podman falls back to the legacy 'cni' network backend when it finds pre-existing container images in storage, but the 'cni' packages are not installed by default. Force 'netavark' so that images with embedded containers work out of the box. This only affects RHEL 9 / CentOS Stream 9; newer distros (and newer podman versions) don't have this fallback logic. Regenerate test manifests. All el9 / c9s manifests that embed containers now get the podman default network backend set. Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Verify the podman default network backend behavior at multiple levels: - YAML loading: fake distro YAML with podman_default_net_backend loads correctly into ImageConfig - osCustomizations: the backend file is generated only when both containers are present and the option is set - Real distro cross-check: RHEL 9 has it set to netavark, while Fedora and RHEL 10 leave it unset Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Make sure that the newly added checks are run on all images. Signed-off-by: Tomáš Hozza <thozza@redhat.com>
thozza
force-pushed
the
el9-podman-network_backend-fix
branch
from
8631167 to
723cbd6
Compare
| }, | ||
| }, | ||
| wantErr: check.ErrCheckFailed, | ||
| }, |
There was a problem hiding this comment.
This should also test the empty backend string / unknown cases.
Also, what's the expected result if both are unknown? Should that be a pass or a fail?
|
I know this is still a draft, and I'm curious to see what Schutzbot thinks of it. It looks pretty nice to me. |
3 participants
RHEL 9 Podman falls back to the legacy 'cni' network backend when it finds pre-existing container images in storage, interpreting them as a migration from an older version. Since the 'cni' packages are not installed by default, this silently breaks networking for images with embedded containers. Fix this by writing a
defaultNetworkBackendfile into container storage during image build, forcing Podman to use 'netavark'. Add host-config checks to detect this inconsistency and verify container embedding at boot time.Architectural Changes
Introduce a
container.NetworkBackendtype and aGenDefaultNetworkBackendFilehelper inpkg/containerto generate the/var/lib/containers/storage/defaultNetworkBackendfile. This file is a Podman-native mechanism that overrides backend auto-detection. The helper accepts a custom storage path to support OSTree images, which relocate container storage to/usr/share/containers/storage. The newPodmanDefaultNetBackendfield onImageConfigdrives this behavior declaratively from the distro definition YAML.Key Changes
GenDefaultNetworkBackendFileinpkg/container/podman.goto produce the backend override file, supporting both default and relocated storage pathsPodmanDefaultNetBackendoption toImageConfigand wire it intoosCustomizations, gated on both the option being set and containers being presentpodman_default_net_backend: "netavark"in the RHEL 9 distro definition; Fedora and RHEL 10 are intentionally left unset since they don't have the fallback bugcontainer-embeddinghost-config check that verifies blueprint containers are present in booted image podman storagepodman-network-backendhost-config check that detects rootful/rootless backend mismatchesBreaking Changes
This PR is fully backward compatible.
Testing
GenDefaultNetworkBackendFilecovering netavark/cni backends and default/custom storage pathsosCustomizationstests verifying the backend file is generated only when both containers and the option are present, including the OSTree relocated-path caselocalhost/normalization), backend consistency, and error paths