Fix CRI-O image name resolution for localhost images by elasticdotventures · Pull Request #21252 · kubernetes/minikube

elasticdotventures · 2025-08-06T03:20:40Z

Summary

Fixes CRI-O's inability to resolve locally built images when using podman driver, eliminating InvalidImageName errors.

Problem

When using minikube start --driver=podman --container-runtime=cri-o, locally built images fail to deploy with:

Failed to apply default image tag "/app4dog/middleware:latest": invalid reference format

Root Cause

CRI-O's registries.conf only searched docker.io, causing unqualified names to be incorrectly resolved.

Solution

Configure CRI-O during setup to:

Search localhost first, then docker.io for unqualified names
Mark localhost as insecure registry
Add proper unqualified-search-registries configuration

Impact

✅ Fixes podman + CRI-O custom image deployment
✅ Zero impact on Docker runtime (99.99% of users)
✅ Surgical change - only modifies CRI-O registry config
✅ Backwards compatible

Related Issues

Fixes podman-env CRI-O prepends '/' to unqualified image names causing InvalidImageName errors #21251
Related to podman-env support needs to update or deprecate #21052 (separate podman-env connectivity issue)
Original report: Critical Bug: Kubernetes prepends '/' to image names causing InvalidImageName errors in podman integration elasticdotventures/minikube#1

Testing

Enables successful deployment of locally built images referenced by unqualified names in K8s manifests.

k8s-ci-robot · 2025-08-06T03:20:51Z

Hi @elasticdotventures. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

minikube-bot · 2025-08-06T03:21:03Z

Can one of the admins verify this patch?

Configure CRI-O registries.conf to resolve unqualified image names to localhost first, then docker.io. This fixes InvalidImageName errors when deploying locally built images with podman + CRI-O runtime. Fixes kubernetes#21251

k8s-ci-robot · 2025-08-06T08:58:44Z

Keywords which can automatically close issues and at(@) or hashtag(#) mentions are not allowed in commit messages.

The list of commits with invalid commit messages:

b063560 Fix CRI-O image name resolution for localhost images

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

afbjorklund

Seems reasonable to fix the broken config, not sure if the localhost workaround is actually needed - but I don't think it hurts either

afaik, it will still search the images with the fake "localhost" registry too - just like docker will name any images with the "docker.io" (even if the image is never pushed there). Maybe it doesn't work with subdirectories/slashes in the name?

k8s-ci-robot · 2025-08-06T18:13:31Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: afbjorklund, elasticdotventures
Once this PR has been reviewed and has the lgtm label, please assign spowelljr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

medyagh · 2025-08-06T18:35:40Z

/ok-to-test

minikube-pr-bot · 2025-08-06T19:26:35Z

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21252 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 50.4s    │ 50.0s                  │
│ enable ingress │ 15.0s    │ 15.1s                  │
└────────────────┴──────────┴────────────────────────┘

Details

Times for minikube start: 50.5s 47.5s 52.0s 50.2s 51.8s
Times for minikube (PR 21252) start: 49.5s 48.9s 53.6s 48.9s 49.3s

Times for minikube (PR 21252) ingress: 14.9s 15.4s 14.9s 15.5s 14.9s
Times for minikube ingress: 15.4s 15.4s 14.9s 14.4s 15.0s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21252 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 22.5s    │ 23.5s                  │
│ enable ingress │ 12.6s    │ 12.8s                  │
└────────────────┴──────────┴────────────────────────┘

Details

Times for minikube start: 21.9s 21.1s 23.2s 21.5s 24.7s
Times for minikube (PR 21252) start: 20.7s 22.4s 22.7s 26.0s 25.6s

Times for minikube ingress: 13.7s 10.7s 12.3s 13.2s 13.2s
Times for minikube (PR 21252) ingress: 13.7s 10.3s 13.7s 12.8s 13.2s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21252 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 22.7s    │ 22.4s                  │
│ enable ingress │ 26.0s    │ 24.4s                  │
└────────────────┴──────────┴────────────────────────┘

Details

Times for minikube start: 22.1s 22.2s 25.1s 22.1s 21.8s
Times for minikube (PR 21252) start: 23.6s 20.6s 20.3s 22.3s 25.6s

Times for minikube ingress: 22.7s 38.7s 22.7s 23.2s 22.8s
Times for minikube (PR 21252) ingress: 22.7s 23.3s 30.2s 23.2s 22.7s

medyagh · 2025-08-06T21:18:33Z

@elasticdotventures can you plz check the KVM_Linux_CRIO test failures? they usually fail much lower this is failing 127/259
https://storage.googleapis.com/minikube-builds/logs/21252/40841/KVM_Linux_crio.html

medyagh

this Breaks integraiton test, please check
Docker Crio
https://storage.googleapis.com/minikube-builds/logs/21252/40841/Docker_Linux_crio.html
KVM Crio
https://storage.googleapis.com/minikube-builds/logs/21252/40841/KVM_Linux_crio.html

on mastter it has only 9-10 test failing on master, but this PR 127 test fail

https://gopogh-server-tts3vkcpgq-uc.a.run.app/?env=KVM_Linux_crio

elasticdotventures · 2025-08-07T02:48:34Z

I just want to be clear - this presumes and is blocked by #21054
This is downstream & should be compartmentalized.
Unless somebody feels strongly - I'd prefer to keep them isolated.

elasticdotventures · 2025-09-28T07:21:09Z

@medyagh I lost all momentum on this. Can you please look at #21054
If that gets merged then these tests will pass.

k8s-ci-robot · 2026-01-26T10:16:08Z

@elasticdotventures: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-triage-robot · 2026-04-26T12:16:40Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-ci-robot added the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Aug 6, 2025

k8s-ci-robot requested review from nirs and prezha August 6, 2025 03:20

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 6, 2025

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 6, 2025

elasticdotventures mentioned this pull request Aug 6, 2025

podman-env CRI-O prepends '/' to unqualified image names causing InvalidImageName errors #21251

Open

Fix CRI-O image name resolution for localhost images

b063560

Configure CRI-O registries.conf to resolve unqualified image names to localhost first, then docker.io. This fixes InvalidImageName errors when deploying locally built images with podman + CRI-O runtime. Fixes kubernetes#21251

elasticdotventures force-pushed the fix/crio-image-resolution branch from a642c85 to b063560 Compare August 6, 2025 08:58

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 6, 2025

elasticdotventures mentioned this pull request Aug 6, 2025

fix: Remove podman binary detection from podman-env command #21054

Open

medyagh requested a review from afbjorklund August 6, 2025 17:48

afbjorklund approved these changes Aug 6, 2025

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 6, 2025

medyagh requested changes Aug 6, 2025

View reviewed changes

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 26, 2026

Conversation

elasticdotventures commented Aug 6, 2025

Summary

Problem

Root Cause

Solution

Impact

Related Issues

Testing

Uh oh!

k8s-ci-robot commented Aug 6, 2025

Uh oh!

minikube-bot commented Aug 6, 2025

Uh oh!

k8s-ci-robot commented Aug 6, 2025

Uh oh!

afbjorklund left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Aug 6, 2025

Uh oh!

medyagh commented Aug 6, 2025

Uh oh!

minikube-pr-bot commented Aug 6, 2025

Uh oh!

medyagh commented Aug 6, 2025

Uh oh!

medyagh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticdotventures commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticdotventures commented Sep 28, 2025

Uh oh!

k8s-ci-robot commented Jan 26, 2026

Uh oh!

k8s-triage-robot commented Apr 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

medyagh left a comment •

edited

Loading

elasticdotventures commented Aug 7, 2025 •

edited

Loading