conversion of image through stdin

[h0nIg]

Hi,

ctr-remote cmd requires a running containerd. I'm wondering if there is a known possibility to convert (not optimize) an image to estargz / zstd:chunked through stdin. I would like to avoid writing the image to the disk and want to convert on the fly.

Any ideas?

[mmonaco]

+1 A related use case for this: I'd to be able to build estargz images as part of our current bazel + rules_oci build system.

[mmonaco]

I've found that skopeo can convert to zstd-chuncked without a daemon running. And the output can be stdout. However it doesn't seem to support input from stdin as it expect to be able to create a scratch dir. This can all still be in memory under /run though.

That said, I wasn't able to get containerd to stream these images. While both skopeo and ctr-remote generate layers of type application/vnd.oci.image.layer.v1.tar+zstd, the annotations are slight different.

Skopeo Example:

    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:3a895b6693a68c8cbaf4c2188931e4f7b567fb08fcdd95ea621699aa54158135",
      "size": 32157363,
      "annotations": {
        "io.github.containers.zstd-chunked.manifest-checksum": "sha256:99d7dba3356de965d23738601faf66ffc293ea13b9a9e712b2a480798df98be4",
        "io.github.containers.zstd-chunked.manifest-position": "31622140:378809:1803199:1",
        "io.github.containers.zstd-chunked.tarsplit-position": "32000957:156334:3914954"
      }
    },

ctr-remote Example:

    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:f54c7d80036f7e9e4933ea4743787d5d5c0b4b87cbde0ec7b6622152d5239854",
      "size": 32807728,
      "annotations": {
        "containerd.io/snapshot/stargz/toc.digest": "sha256:30dbc142608dc7de0f969295047cf723466d9a0247c7c1eae199dd19b6d062c9",
        "io.containers.estargz.uncompressed-size": "80415744",
        "io.containers.zstd-chunked.manifest-checksum": "sha256:abe01260c18d226fcc18fcd43fbbd7310fbb1c62cf9dec88a3a24f9f9d1e6664",
        "io.containers.zstd-chunked.manifest-position": "32645241:162439:1046882:1"
      }
    },

The ctr-remote example does do the lazy streaming. The skopeo example I see errors like:

containerd-stargz-grpc[222]: {"error":"4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy:
failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","key":"moby/1/extra
ct-839748415-9cqZ sha256:8d6b7eb76b62254226e7ea4253257e827eba1e39d2ba0835a671ff47eb731bc2","level":"debug","mountpoint":"/var/li
b/containerd-stargz-grpc/snapshotter/snapshots/1/fs","msg":"failed to pre-resolve","parent":"","time":"2025-07-24T23:11:31.41853
2792Z"}

And then falls back to a normal image pull.

[mmonaco]

I wound up writing a CLI to do an image conversion from an oci-layout dir to a new oci-layout dir.

I don't believe it's feasible to do this purely through stdin without consuming a TON of memory. For each blob (the layers, the config, and the manifest) we don't know the digest until each stream has been processed.