podman-env CRI-O prepends '/' to unqualified image names causing InvalidImageName errors

[elasticdotventures]

🐛 Critical Bug Report: CRI-O Image Name Resolution Failure

Summary

When using minikube with podman driver and CRI-O runtime, Kubernetes incorrectly prepends a forward slash (/) to unqualified image names during resolution, causing InvalidImageName errors and preventing pods from starting.

Environment

• Driver: podman
• Container Runtime: CRI-O
• Issue Scope: Image name resolution in CRI-O → podman integration
• Impact: Critical - prevents custom application deployment

Problem Description

Expected Behavior: app4dog/middleware:latest → resolves correctly
Actual Behavior: app4dog/middleware:latest → becomes /app4dog/middleware:latest ❌

Error Message

Failed to apply default image tag "/app4dog/middleware:latest": couldn't parse image name "/app4dog/middleware:latest": invalid reference format

Root Cause Analysis

1. Image Storage: podman build -t app4dog/middleware:latest stores as localhost/app4dog/middleware:latest
2. K8s Reference: Deployment references app4dog/middleware:latest (unqualified)
3. CRI-O Resolution: Incorrectly prepends / instead of resolving to localhost/ prefix
4. Validation Failure: Malformed name /app4dog/middleware:latest fails parsing

Reproduction Steps

1. minikube start --driver=podman --container-runtime=cri-o
2. eval $(minikube podman-env)
3. podman build -t app4dog/middleware:latest .
4. Deploy K8s pod referencing app4dog/middleware:latest
5. Observe InvalidImageName error

Related Issues

• Original report: Critical Bug: Kubernetes prepends '/' to image names causing InvalidImageName errors in podman integration elasticdotventures/minikube#1
• Context: podman-env support needs to update or deprecate #21052 (comment)

Impact Assessment

• Severity: Critical - breaks podman + CRI-O development workflows
• Scope: Affects custom image deployment (not pre-built images)
• Regression: Long-standing issue in CRI-O integration

Technical Context

This is separate from podman-env connectivity issues. The bug occurs in the CRI-O runtime's image name resolution logic, specifically in how /etc/containers/registries.conf handles unqualified registry names.

Location: Likely in pkg/minikube/cruntime/crio.go registry configuration or CRI-O config generation.

Next Steps

Working on engineered fix that:

• ✅ Resolves CRI-O image name resolution
• ✅ Maintains Docker compatibility (99.99% of users)
• ✅ Integrates with existing podman-env improvements
• ✅ Low-impact, surgical change

/kind bug
/priority critical-urgent
/sig node

[elasticdotventures]

🔧 Fix Available

PR Created: #21252

Solution: Configures CRI-O's registries.conf to search localhost before docker.io for unqualified image names.

Impact: Surgical fix - only affects CRI-O configuration, zero impact on Docker workflows.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[elasticdotventures]

keep alive

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten