Make estargz compression-algorithm-agnostic and support zstd (a.k.a. zstd:chunked) #293
Conversation
|
@giuseppe WDYT? I know zstd:chunked is still under discussion (containers/storage#775) and that spec may change in the future. But this implementation will help to make that discussion forward? |
ktock
force-pushed
the
genericesgz
branch
2 times, most recently
from
3dfabd0 to
3aa157d
Compare
|
@giuseppe Could you take a look? 🙏 |
|
sorry, I've missed the previous ping. @ktock sure! And we can try to make it as close as possible to the estargz TOC to simplify using it in stargz-snaphshotter. The only difference I am aware of is how the chunking is represented. Does the current metadata for files chunking work for you? |
|
@giuseppe Metadata (TOC) implemented in containers/storage#775 looks good to me.
|
is it fine if it is named |
|
Containerd currently doesn't allow snapshotter to handle annotations that are not prefixed by I also think we need standard spec for lazy-pulling-related annotations (currently discussed in opencontainers/image-spec#815). |
ktock
force-pushed
the
genericesgz
branch
5 times, most recently
from
1a75322 to
3931c4d
Compare
| ) | ||
|
|
||
| var GetTOCDigestCommand = cli.Command{ | ||
| Name: "gettocdigest", |
There was a problem hiding this comment.
Nit: “get-toc-digest” might be more readable , but no strong opinion
ktock
force-pushed
the
genericesgz
branch
2 times, most recently
from
eddb783 to
f7e1437
Compare
| return nil, err | ||
| } | ||
| if uncompressedDesc == nil { | ||
| return nil, errors.Errorf("unexpectedly got the same blob aftr compression (%s, %q)", desc.Digest, desc.MediaType) |
| newDesc := desc | ||
| if uncompress.IsUncompressedType(newDesc.MediaType) { | ||
| if images.IsDockerType(newDesc.MediaType) { | ||
| newDesc.MediaType += ".zstd" |
There was a problem hiding this comment.
Do we really want to support zstd for DockerType?
There was a problem hiding this comment.
No, we don't. Fixed it to always convert to oci type.
…zstd:chunked) Initially, eStargz is based on gzip compression. But, through zstd:chunked work, it turned out that eStargz is not limited to gzip compression and the same chunking & verifying & prefetching method can be applied to other compression algorithms as well (e.g. zstd). This commit makes `estargz` pkg configurable and agnostic about compression algorithms. For supporting non-gzip compression, the user must implement `estargz.Decompressor` and `estargz.Compressor` interfaces and must plug them to `estargz` tools (e.g. `estargz.Open` and `estargz.NewWriterWithCompression`). `estargz` also provides test suite that is usable for testing these non-gzip eStargz implementations. This commit comes with `zstdchunked` pkg that support zstd compression for eStargz (a.k.a. zstd:chunked), based on the above extensibility. `zstdchunked` pkg contains `zstdchunked.Decompressor` and `zstdchunked.Compressor` that allows `estargz` pkg to use zstd compression (i.e. zstd:chunked) instead of gzip. Layer converter and filesystem now support zstd:chunked leveraging `zstdchunked` pkg. `ctr-remote image optimize` and `ctr-remote image convert` support `--zstdchunked` option that omits zstd-based eStargz and filesystem supports zstd-based eStargz layers by default. Signed-off-by: Kohei Tokunaga <[email protected]>
This is the subset of #281 but re-designed.
Initially, eStargz is based on gzip compression. But, through zstd:chunked work, it turned out that eStargz is not limited to gzip compression and the same chunking & verifying & prefetching method can be applied to other compression algorithms as well (e.g. zstd).
This commit makes
estargzpkg configurable and agnostic about compression algorithms. For supporting non-gzip compression, the user must implementestargz.Decompressorandestargz.Compressorinterfaces and must plug them toestargztools (e.g.estargz.Openandestargz.NewWriterWithCompression).estargzalso provides test suite that is usable for testing these non-gzip eStargz implementations.This commit comes with
zstdchunkedpkg that support zstd compression for eStargz (a.k.a. zstd:chunked), based on the above extensibility.zstdchunkedpkg containszstdchunked.Decompressorandzstdchunked.Compressorthat allowsestargzpkg to use zstd compression (i.e. zstd:chunked) instead of gzip.Layer converter and filesystem now support zstd:chunked leveraging
zstdchunkedpkg.ctr-remote image optimizeandctr-remote image convertsupport--zstdchunkedoption that omits zstd-based eStargz and filesystem supports zstd-based eStargz layers by default.In the future, we can support other compression algorithms (e.g. lz4 used by nydus?) as well.
cc @giuseppe @AkihiroSuda