DRAFT: Image Stream for better UX

[mariusbertram]

What type of PR is this?

feature
Which issue does this PR fix:
#1395

What does this PR do / Why do we need it:
This PR implements a better UX for on-demand Image Pulls.

This PR is in a working PoC status.

If an issue # is not available please add repro steps and logs showing the issue:

Testing done on this change:

Automation added to e2e:

Will this break upgrades or downgrades?

Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[rchincha]

We already download blobs to a temporary dir, so no need to create a new one.
We likely just need "onDemandStreaming": enable/disable and which should depend on onDemand:true

[rchincha]

@mariusbertram thanks for taking this on. This has been a long time ask.

[andaaron]

Hi @mariusbertram, thank you for the initiative.

The sync feature is designed to be an "extension". It is an optional feature, built using the "sync" go build tag, and the package is located under https://github.com/project-zot/zot/tree/main/pkg/extensions/sync.

This PR doesn't seem to integrate with it, except for the configuration file.

"sync" uses regctl as a library to copy the files locally to a temporary folder. After the entire download is done, it moves the files to storage. After that is done, it starts serving them to the user.
The implementation goal of #1395 is to refactor "sync" and make the content available to the client as it becomes available.
We don't want to duplicate logic found in regctl, we tried that before, and the maintenance overhead was too large. If regctl library changes are needed, that can also be discussed.

[andaaron]

Is this more of an update to serve content from storage concurrently for #2589?

[mariusbertram]

Is this more of an update to serve content from storage concurrently for #2589?

The idear is to implement something like a proxy cache for the pull through. With that zot downloads the blobs to the cache dir and can after download serve this blob to the requested client.
It should be the priority to serve first the client and then persist the requested image.
Because of the caching there is no need to re download the Image with regctl and just move the blobs to the final storage (s3/local directory/ etc.)
this is a relative simple approach, while duplicating Datastreams and working with them does have some pitfalls.

I will try to implement this approach with regctl.

Because of the different approach I did Create this PR in this early stage to get your feedback.

[mariusbertram]

I did some testing. The major problem is the initial pull of the image by zot is extremely slow.
I did test different registries with the registry.access.redhat.com/ubi10/go-toolset:latest image.
It is a multi-arch image with multiple layers and a size of around 1 GB uncompressed and 400 MB compressed.
https://catalog.redhat.com/en/software/containers/rhel8/go-toolset/5b9c810add19c70b45cbd666

A direct pull takes about 6s

time podman pull registry.access.redhat.com/ubi10/go-toolset:latest 
Trying to pull registry.access.redhat.com/ubi10/go-toolset:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:3d9bca65716dd05d5e806d53c5a82afabda97976c0f85458087d22a6df090b14
Copying blob sha256:3f3988782553a7df3a374b44628e79b412aee8c0f7b10c1f66406aa8697c8ebb
Copying blob sha256:357e652cbe64b5a370b576c03b11255daafc49e4e36a7e552c53b1886129a05a
Copying blob sha256:d368f80437f862830c17c154fa28fff5db32db910afe6c110c8314cafc36e3b8
Copying config sha256:8b179f3820c4677689a6760b296880a0dead63aa698272cc445326735c1267b2
Writing manifest to image destination
Storing signatures
8b179f3820c4677689a6760b296880a0dead63aa698272cc445326735c1267b2
podman pull registry.access.redhat.com/ubi10/go-toolset:latest  0,02s user 0,01s system 0% cpu 6,253 total

with ociregistry (a very simple Pull-through oct registry ) 42s

time podman pull --tls-verify=false 192.168.1.132:8080/registry.access.redhat.com/ubi10/go-toolset:latest
Trying to pull 192.168.1.132:8080/registry.access.redhat.com/ubi10/go-toolset:latest...
Getting image source signatures
Copying blob sha256:3d9bca65716dd05d5e806d53c5a82afabda97976c0f85458087d22a6df090b14
Copying blob sha256:3f3988782553a7df3a374b44628e79b412aee8c0f7b10c1f66406aa8697c8ebb
Copying blob sha256:357e652cbe64b5a370b576c03b11255daafc49e4e36a7e552c53b1886129a05a
Copying blob sha256:d368f80437f862830c17c154fa28fff5db32db910afe6c110c8314cafc36e3b8
Copying config sha256:8b179f3820c4677689a6760b296880a0dead63aa698272cc445326735c1267b2
Writing manifest to image destination
8b179f3820c4677689a6760b296880a0dead63aa698272cc445326735c1267b2
podman pull --tls-verify=false   0,02s user 0,01s system 0% cpu 42,167 total

and a current zot build with local storage 2min 32s

time podman pull --tls-verify=false 192.168.1.132:8080/ubi10/go-toolset:latest
Trying to pull 192.168.1.132:8080/ubi10/go-toolset:latest...
Getting image source signatures
Copying blob sha256:3d9bca65716dd05d5e806d53c5a82afabda97976c0f85458087d22a6df090b14
Copying blob sha256:3f3988782553a7df3a374b44628e79b412aee8c0f7b10c1f66406aa8697c8ebb
Copying blob sha256:d368f80437f862830c17c154fa28fff5db32db910afe6c110c8314cafc36e3b8
Copying blob sha256:357e652cbe64b5a370b576c03b11255daafc49e4e36a7e552c53b1886129a05a
Copying config sha256:8b179f3820c4677689a6760b296880a0dead63aa698272cc445326735c1267b2
Writing manifest to image destination
8b179f3820c4677689a6760b296880a0dead63aa698272cc445326735c1267b2
podman pull --tls-verify=false 192.168.1.132:8080/ubi10/go-toolset:latest  0.02s user 0.01s system 0% cpu 2:32.29 total

I have not found the blocking part. But I think if this problem is solved and for my environment (Internet 100mbit/s down) with an image of this size in sub 1min should be a first step to the streaming.

BTW Redhat’s Quay has a similar approach as zot https://docs.redhat.com/en/documentation/red_hat_quay/3.7/html/use_red_hat_quay/quay-as-cache-proxy

[sizowie]

I would like to see this feature in zot - as we are experiencing same performance issues with zot.