Podman 5.8.1 brew/mac m1 pro 32gb - image list slow, images -a slow

[fredcooke]

Hey, I had image list | wc returning 1700 which seems low/normal for me - and it took about 30s to complete - -a took a bit longer about 35s

I did a podman image prune and have 500 images now and image list is 0.5s

I saved the hash list from the prune and it was 2034

My internal explanation for the difference in numbers is that 2034 is layers deleted and 1700 was top level leaf layers which in many cases shared layers beneath. That's reasonable if true.

But why did the prune speed it up so much - why was it SO slow just listing what's there? Why is it acceptably fast now when there's still 500 there?

Wound up here in discussion instead of on an issue. Hopefully not radio silence :-)

I will wind up with hundreds more images over the coming weeks/months so I'll probably run into it again if diagnostics are useful in terms of improving things. Something somewhere seems to be wrong/doing unneeded work or something.

[Luap99]

First, I don't think anyone on the team would consider 1700 a normal amount of images. That is certainly well outside of the range anyone would test with.

We don't really have performance benchmarks, at least not in that size range. I have not looked at the code but it is certainly possible that there might be some O(n^2) or worse O(2^n) performance here which could explain a big increase in command time.

Without a reproducer and storage configuration there is likely not much we could tell here regardless.

[fredcooke]

Hmmmmm, for me that's normal/low - if they were big it wouldn't be a workable situation - but the majority are:

1. Single layer
2. Tiny (5-50kB)

And of those that are not the majority:

1. Share a common base
2. Have a single light layer above that base

That is to say I have in the order of 10-20 public up stream images like ubu/ubi/alpine/deb and others plus repeated local layers of my projects on top plus the from-scratch ones.

I'm pretty new to Podman having only discovered it when gh actions ubu "upgraded" docker to one without experimental mode and without --squash. But I'm Podman's biggest fan now and going all in requiring it everywhere.

I noticed this first seeing my remote tag listing happen near instantly and while local tag listing was noticeably slow even if the list was only a few.

After the prune it being fast is interesting since there's still 500 images present - the rest that got cleaned up would be orphaned top layers overwritten during subsequent local builds of the same project at the same time frame, more or less.

Happy to help and run diagnostics if/when it gets like this again - and I'm pretty sure it will within weeks.

I don't remember ever having slowness issues with docker for local use like this - but I never paid too much attention. Every time I was forced to clean up local images with docker it was storage limitation with big images and moderate disk size. I could probably devise a realistic test and do some command masking and run it against docker and podman if that's helpful - or once it does naturally recur run local diagnostic commands to figure it out.

If you're wondering how I make so many so fast:

Each project produces minimum 2 arch specific images and 1 manifest.
Many produce their additional arch images and a manifest via D file
Many produce additional 2+1 for another reason

some all 3, so on an all 3 project a single local build produces 6 images (3x amd 3x arm) and 3 manifests. - however of those at most 2 are on a heavier base and usually are light layers.

supporting arm and amd natively seems mandatory for me for most projects howadays with people preferring mac and arm clusters winning out cost, power, environment, and performance wise.

[fredcooke]

Re storage configuration - none - or default:

init
start
connection list
stop
set rootful
start

that's all I did AFAIK.

[Luap99]

Without concrete reproducer we can just guess. If you can create a simple script that creates these images someone else could test in on their system.

Otherwise you could try to podman machine ssh and run the podman commands locally there to see if this changes anything. I doubt it but worth to try to rule our any remote weirdness.

And if you use local podman you can run podman --cpu-profile podman-images.pprof images and then the cpu profile should tell us where the most time is spend in the code.

[baude]

i agree that ideally you create a strict reproducer; something that can be tested by others and more importantly, on different environments. lots of images (and that is a lot) are more problematic perf wise for podman. perhaps some pruning or squashing steps would help your specific use case out if you can work it into the workflow ?

[fredcooke]

Thanks guys - I'll see what I can cook up - may have to wait a few weeks but I SOOOOOO want Podman to win over Docker you wouldn't believe it so I won't forget this thread and fail to come back. I've got some critical stuff to knock out first but for the record:

Prior to April 5th I didn't have automatic local builds in place so it took me bang on a month of evenings/weekends to get to this point. Can I generate that faster using a script - absolutely. I will do my best to clear out every image I have (none matter, all can be repulled) and write something that performs a set of podman operations with none as a base line - and then at check points after XYZ iterations and maybe try to plot it over time/quantity.

If I can help solve this then I can push podman to everyone I know without hesitation - I probably will anyway since --squash is and will remain king to the end of time. Thank you for making it a first class citizen that can be relied on and doesn't need hoops jumping through to use. <3

PS I contest the "a lot" - we're in the AI age - multi arch age so each image is now 2+manifest = 3 sort of. How many runs over time does it take to generate this with 6 images and 3 manifests per run/ 150-200 - how many do I apparently do in an average day? 6-10 ish builds - that's not a busy workflow it's just mild usage - and yeah prune helped and yeah clean up is needed but to me clean up should be yearly or on disk full not more frequently :-D no biggy for now. Thanks!

[mg1986jp]

The slowness with 1700 images is probably because podman images has to read metadata for every image from the local storage, and on macOS that storage sits inside the VM's virtual disk. Each image query hits the overlay storage driver, and with 1700 entries that adds up.

Your observation about the prune speeding things up makes sense. The 2034 vs 1700 difference is exactly what you described — dangling/intermediate layers that share parent layers with the 1700 top-level images. When those get pruned, the storage driver has less to iterate over.

With 500 images at 0.5s, that's a healthy baseline. If you want to keep more images without the slowdown, you could look into switching the VM's storage backend — but honestly, periodic pruning is probably the easiest approach. Something like a cron/launchd job running podman image prune -f weekly would keep it snappy.

[Luap99]

That adds nothing useful to the discussion. I can just guess but this appears to be just some LLM summary without value that you seems to be adding to a ton of discussions here recently.

We have a strict policy against that so please do not post any LLM output verbatim, especially not if it just reiterates already mentioned points without any useful new information.
https://github.com/containers/podman/blob/main/LLM_POLICY.md

[fredcooke]

Ugh, I totally missed the emdash. I'm ashamed to say I did not pick that one. At work a few days ago a contractor pasted some LLM text meant for his eyes referring to me as "the reviewer" and "Would you like me to" etc as a reply to me - horrendous. LLM for comms - fail. LLM for code - mixed bag - depends on the operator. Thanks for flagging this. And I'm second guessing using the word flagged... because... :-( what has the world become.