Skip to content

Add safe Rust zstd decoding backend#349

Open
Shnatsel wants to merge 3 commits into
image-rs:mainfrom
Shnatsel:safe-rust-zstd
Open

Add safe Rust zstd decoding backend#349
Shnatsel wants to merge 3 commits into
image-rs:mainfrom
Shnatsel:safe-rust-zstd

Conversation

@Shnatsel
Copy link
Copy Markdown
Member

@Shnatsel Shnatsel commented Mar 5, 2026

Pros: 100% safe Rust. No memory safety bugs, no C toolchain required

Cons: ~3x slower than libzstd

Future compatibility notes

The safe Rust impl has no encoding support. That's fine for now since the crate doesn't support encoding Zstd anywyay. When encoding is implemented, we'll have to limit it to the zstd feature only, which should be trivial.

Trifecta Tech Foundation is starting work on a Rust drop-in replacement for libzstd, similar to zlib-rs. When that's ready to use, I expect it to replace the zstd feature, and this option to remain available.

Copy link
Copy Markdown
Member

@197g 197g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd gladly add it optionally, similar to rustls being an option for SSL crates. @KillingSpark If you have the time to chime in and maybe some words on the integration?

Comment thread Cargo.toml Outdated
Comment thread src/decoder/image.rs Outdated
Comment on lines +662 to +666
#[cfg(all(not(feature = "zstd"), feature = "safe-rust-zstd"))]
CompressionMethod::ZSTD => Box::new(
ruzstd::decoding::StreamingDecoder::new(reader)
.map_err(|e| std::io::Error::other(e))?,
),
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find it a little odd to get an error already from construction that is related to the contents rather than the parameters. Well, a bit hypocritical, considering I'm currently fixing us having the same mistake in Decoder::new.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They return errors in the new method for the same reason we currently do: that's where header parsing happens

@fintelia
Copy link
Copy Markdown
Contributor

fintelia commented Mar 6, 2026

Another naming option would be zstd and zstd-native which parallels how we name the two AVIF features in the image crate

@Shnatsel
Copy link
Copy Markdown
Member Author

Shnatsel commented Mar 6, 2026

There is a documented caveat in the ruzstd streaming decoder: it only decodes a single frame from the stream, while Zstd streams can contain multiple frames.

But multi-frame files are relatively uncommon and the fix should be committed upstream into ruzstd instead of complicating callers, so I don't think that would change this PR in any way.

@KillingSpark
Copy link
Copy Markdown

Hi everyone,

the code in the PR looks good. A few things:

  1. Technically the ruzstd has encoding support since a few months. Its not really configurable but it does compress somewhat well. Performance wise it's also lagging behind the C implementation. Whether this is good enough for this crate I can't really say. It's tested and fuzzed and it's ready for use, if these tradeoffs are acceptable.
  2. The multi-frame issue is very annoying. I have not yet found an API that makes me happy. The format also allows skippable frames. Just providing an API that skips these without providing a possibility to interact with the contents seems bad. Always having to pass handlers and stuff for these bloats the API in when most people don't care about these frames. Anyways, yeah right now ruzstd just passes that responsibility to the user which also isn't ideal. As I said: its annoying.
  3. Ruzstd has a small portion of unsafe code because the std VecDeque doesn't have a extend_from_within which is needed for an efficient implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants