container registry as chunk store

[katexochen]

I would love to see container registries as supported chunk store backend. Container registries are widely available and often low cost. Supports read and write. The oras project already has a pretty good implementation of using container registries as content addressable storage and provides a go library that could be used by this project.

[folbricht]

The interface of that library looks quite similar to what we'd need to support registry stores. Do you happen to have an easy way to setup a local test-server that works with the oras lib?

Unfortunately, it'll be a while (a few weeks at least) before I can look into it.

[katexochen]

Thanks @folbricht, appreciate it. :)

Do you happen to have an easy way to setup a local test-server that works with the oras lib?

I've only ever used oras with ghcr.io, but I did some research and discovered a section in the oras e2e documentation about registry services for testing, looks like the main options for local testing are

• distribution/distribution
• project-zot/zot

e2e tests of oras also show how both can be setup, hope that helps.

[emilebosch]

@katexochen I was thinking the same thing. Would u like to work on it together? i have some ideas.

[folbricht]

@emilebosch Feel free to give it a try. If you need any pointers where to start, here's a few:

• You'll have to implement the Store and WriteStore interfaces here: https://github.com/folbricht/desync/blob/master/store.go#L16-L28
• Depending on library support, perhaps the S3Store may be a good template: https://github.com/folbricht/desync/blob/master/s3.go
• And then to use it from the command-line you'd need to come up with a way of specifying it, see the existing naming scheme in https://github.com/folbricht/desync/blob/master/cmd/desync/store.go#L130
  Just ask if you need any help with that.

[katexochen]

@katexochen I was thinking the same thing. Would u like to work on it together? i have some ideas.

Sorry, I don't think I'll find the cycles to work on this soon, so feel free to take it away. :)
Let me know if you hit any issues.

[Azathothas]

We at @pkgforge would find this feature incredibly useful.
We store around 10k packages on ghcr, and currently we have no delta updates for any of them.

We already tried using our http mirror at huggingface, but quickly ran into rate limits just from a single package.

If anyone would like to revisit this & support is added, desync would be used for all our packages instead of the currently supported (not all packages) zsync.

[folbricht]

@Azathothas I can try to whip something up. Biggest hurdle seems to be that I don't yet have an environment to develop against, need to first set something up, or do you happen to have something to play in? Perhaps I can setup an instance of https://github.com/project-zot/zot

Would you be able to test any draft implementations once I have something?

[Azathothas]

I can try to whip something up

Thank you

Would you be able to test any draft implementations once I have something?

Yes, if the linked PR is ready, then I can build it & try to use it as a store for a few of our packages.
I will attach any debug info that's requested.

[folbricht]

That PR is completely untested so far and likely wont work. There are still several unknowns, like what hashing algorithms would be supported (looks like only SHA256 is supported, while desync by default uses SHA512/256), and in how far the content being sent needs to match the hash.

Can you describe how you plan on using it with some examples? Would you only want to use it as chunk store for multiple blobs like -s oci://repo:5000/mystore or treat it as a store for just one image as in -s oci://repo:5000/myimage:v1?

[xplshn]

That PR is completely untested so far and likely wont work. There are still several unknowns, like what hashing algorithms would be supported (looks like only SHA256 is supported, while desync by default uses SHA512/256), and in how far the content being sent needs to match the hash.

Can you describe how you plan on using it with some examples? Would you only want to use it as chunk store for multiple blobs like -s oci://repo:5000/mystore or treat it as a store for just one image as in -s oci://repo:5000/myimage:v1?

Is there any reason why b3sum or XXHASH isn't used here? It would make far more sense to use an alg that's faster than RAM read operations

I can recommend https://github.com/lukechampine/blake3, which aims to make it into the STD and is thus a very clean and idiomatic library.

There's also https://github.com/cespare/xxhash, which I use in various projects of my own where speed is critical

[folbricht]

Yes, it's compatibility with https://github.com/systemd/casync SHA256 is supported as an alternative, and it looks like that would have to be used here.

Just tried to use plain oras with GHCR without much luck (can't even login). Looks like things are less standardized than expected and I first have to find a combo that works.

[xplshn]

Yes, it's compatibility with https://github.com/systemd/casync SHA256 is supported as an alternative, and it looks like that would have to be used here.

Just tried to use plain oras with GHCR without much luck (can't even login). Looks like things are less standardized than expected and I first have to find a combo that works.

I don't use Oras because it is Apache V2, and I prefer MIT/ISC/3BSD/2BSD/0BSD, so I wrote my own OCI/ghcr manifest parsing for dbin, it could be used as a reference if you need one. The code is small enough that it is contained in a single file.

https://github.com/xplshn/dbin/blob/master/fetch.go

Here are some excellent resources I used to get started with container registry:

• https://stackoverflow.com/questions/78164818/how-to-download-a-file-from-github-container-registry-cli-command-github-packa
• https://pkg.go.dev/github.com/google/go-containerregistry#section-readme

And if you want to upload a few random files to ghcr: https://blog.pradumnasaraf.dev/publish-image-on-ghcr

---

I'm really hoping this can be added, do tell me if there's anything I can help with :)

[Azathothas]

@folbricht

That PR is completely untested so far and likely wont work. There are still several unknowns, like what hashing algorithms would be supported (looks like only SHA256 is supported, while desync by default uses SHA512/256), and in how far the content being sent needs to match the hash.

The OCI spec itself supports sha512: https://github.com/opencontainers/image-spec/blob/main/descriptor.md#digests
With oras (the cli, I don't know about the library), I don't see an option in the cli, perhaps with the --config it is possible?
I created a discussion: oras-project/oras#1689

---

Can you describe how you plan on using it with some examples? Would you only want to use it as chunk store for multiple blobs like -s oci://repo:5000/mystore or treat it as a store for just one image as in -s oci://repo:5000/myimage:v1?

We (pkgforge) use oras a little bit differently. For instance, we make no use of platform tag, and simply append the platform name in the tag itself. See: https://github.com/pkgforge/soarpkgs/blob/main/.github/scripts/ci/helpers.sh#L1038
So not sure if my feedback will actually be helpful.
Regardless, we would use desync in this way (desync is a pkg name, it might also be curl for example):

1. We build & upload desync : ghcr.io/pkgforge/bincache/desync/official/source/desync
   
2. We then push the store to the same package but with different tag, so something like this:

• ghcr.io/pkgforge/bincache/desync/official/source/desync:desync-aarch64-linux
• ghcr.io/pkgforge/bincache/desync/official/source/desync:desync-x86_64-linux

3. We would then use the .caibx file and the desync stores (which exist as another tag in the main package)

treat it as a store for just one image as in -s oci://repo:5000/myimage:v1?

So probably this

---

Just tried to use plain oras with GHCR without much luck (can't even login). Looks like things are less standardized than expected and I first have to find a combo that works.

1. You need to generate a token with these permissions:

2. After that, simply do

oras login --username "<YOUR-USERNAME>" --password "<YOUR-TOKEN>" "ghcr.io"

Note

• By default, github makes packages private, so make sure to change the visibility to public (unless it's meant to stay private): https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility
• Also, make sure the entire oci url is lowercase, as the regestries don't allow upper/mixed case (at least github)

#Notice I used `Azathothas` (uppercase A)
$ oras push --config "/dev/null:application/vnd.oci.empty.v1+json" "ghcr.io/Azathothas/oci-test:test-tag" hi.txt 
Error: "hi.txt": invalid reference: missing registry or repository
Please make sure the provided reference is in the form of <registry>/<repo>[:tag|@digest]

#If I use `azathothas` (everything lowercase)
$ oras push --export-manifest /tmp/original-manifest.json \
  --config "/dev/null:application/vnd.oci.empty.v1+json" \
  "ghcr.io/azathothas/oci-test:test-tag" hi.txt
Pushed [registry] ghcr.io/azathothas/oci-test:test-tag
ArtifactType: application/vnd.oci.empty.v1+json
Digest: sha256:9ee18eef49252e85cadcda3f35e652ca288972b4bf572123ee8a0a6fc384f691

3. oras --debug
   This is quite useful for debugging as it shows us the http requests/responses.
   You will notice that the registries are just fancy http stores :)

[folbricht]

Thanks for all the pointers. Got a little further now.

It looks like it really needs to be SHA256 in this case as SHA512/256 (which first hashes with SHA512 and then only uses 256 bits of the result) isn't supported in the OCI spec. Should still work, though it won't be possible to combine these stores with any other store that's using the default SHA512/256 unfortunately.