You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In our data compression projects, we use Rust where C is traditionally used. During the work, we've hit limitations in Rust itself and in the surrounding tooling. Over the years, we've become increasingly comfortable with fixing these issues ourselves.
13
13
14
14
<!-- more -->
15
15
16
-
Previously, we felt stuck at times by missing functionality in stable Rust, without a clear path forward except to wait. In practice, waiting has not been a fruitful strategy: the features we need are niche and rarely make it to the top of Rust maintainers' to-do lists.
16
+
Previously, we felt stuck at times by missing functionality in stable Rust, without a clear path forward except to wait. In practice, waiting has not been a fruitful strategy: the features we need are niche and rarely make it to the top of Rust maintainers' to-do lists.
17
17
18
18
In this post, I'll share some of the steps we took to get unstuck. It goes over some of the fixes and improvements that we've made as a part of Trifecta Tech's [Data compression initiative](/initiatives/data-compression) (`zlib-rs`, `libbzip2-rs` and `libzstd-rs-sys`) over the past year.
19
19
@@ -73,15 +73,15 @@ Along the way it turned out that the implementation of the related `ptr_offset_w
73
73
74
74
### Improved intrinsic support
75
75
76
-
We use Miri to test the unsafe code that we still have. I recently wrote about our work on [emulating avx-512 intrinsics in Miri](https://trifectatech.org/blog/emulating-avx-512-intrinsics-in-miri/). Since then we've added support for a couple additional instructions for our `avx512vnni` implementation of adler32.
76
+
We use Miri to test the unsafe code that we still have. I recently wrote about our work on [emulating avx-512 intrinsics in Miri](https://trifectatech.org/blog/emulating-avx-512-intrinsics-in-miri/). Since then we've added support for a couple additional instructions for our `avx512vnni` implementation of adler32.
77
77
78
78
My next goal is to also be able to run the zlib-rs AArch64 SIMD tests with Miri. For now that'll need some additional support in Miri itself, but with LLVM 23 we'll be able to only use portable intrinsics and the custom Miri support should no longer be needed.
79
79
80
80
I want to stress that the Miri implementation is really only half the work. The other half is testing the implementation (sometimes giving rise to improving tests in e.g. `rust-lang/stdarch` as well), to make sure that the behavior is correct and that support never regresses.
81
81
82
82
### ICE when reading from a static array of function pointers
83
83
84
-
In the first week of working on [libzstd-rs-sys](https://github.com/trifectatechfoundation/libzstd-rs-sys) we ran into `c2rust` producing some Rust code that Miri was unable to handle.
84
+
In the first week of working on [libzstd-rs-sys](https://github.com/trifectatechfoundation/libzstd-rs-sys) we ran into `c2rust` producing some Rust code that Miri was unable to handle.
@@ -144,7 +144,7 @@ Rust can call c-variadic functions (like `libc::printf`), but defining them is u
144
144
145
145
```rust
146
146
pubunsafeextern"C"fngzprintf(
147
-
file: gzFile,
147
+
file: gzFile,
148
148
format:*constc_char,
149
149
va:...) ->c_int {
150
150
unsafe { gzvprintf(file, format, va) }
@@ -167,4 +167,4 @@ When you do cursed things, problems find you.
167
167
168
168
The interplay between low-level systems work and compiler development has been very fruitful for us. It has been enormously gratifying to be able to tackle some of these problems ourselves, and we have lots of exciting things lined up for 2026.
169
169
170
-
*These contributions to the compiler resulted from Trifecta Tech Foundation's [Data compression initiative](/initiatives/data-compression). Please [contact us](/support) if you are interested in financially supporting this work.*
170
+
*These contributions to the compiler resulted from Trifecta Tech Foundation's [Data compression initiative](/initiatives/data-compression). Please [contact us](/support) if you are interested in financially supporting this work.*
Copy file name to clipboardExpand all lines: content/initiatives/data-compression.md
+9-9Lines changed: 9 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,22 +11,22 @@ summary = "<p>Almost all content sent over the Internet undergoes data compressi
11
11
12
12
projects = [
13
13
"zlib-rs",
14
-
"zstd-rs",
14
+
"libzstd-rs",
15
15
"bzip2-rs"
16
16
]
17
17
18
18
funders = [
19
-
"nlnetfoundation",
20
-
"chainguard",
19
+
"nlnetfoundation",
20
+
"chainguard",
21
21
"ngi-zero-core",
22
22
"min-bzk",
23
23
"astral"
24
24
]
25
25
26
26
supporters = [
27
-
"devolutions",
28
-
"prossimo",
29
-
"tweedegolf",
27
+
"devolutions",
28
+
"prossimo",
29
+
"tweedegolf",
30
30
"isrg"
31
31
]
32
32
@@ -68,12 +68,12 @@ In June 2025 we released a new version of the **bzip2** crate, see [the bzip2 pr
68
68
69
69
### What's Next
70
70
71
-
Development of **zstd** began in July 2025, with the first release of the decoder planned for February 2026. We're currently seeking funding to complete work on the encoder side, see [Zstandard in Rust](/projects/zstd-rs/).
71
+
Development of **zstd** began in July 2025, with the first release of the decoder planned for February 2026. We're currently seeking funding to complete work on the encoder side, see [Zstandard in Rust](/projects/libzstd-rs/).
72
72
73
73
Meanwhile, work on zlib-rs and bzip2-rs continues to improve the implementations, enhance performance, and expand adoption.
74
74
75
75
### Roadmap
76
76
77
-
See the [zlib-rs](/projects/zlib-rs/), [bzip2-rs](/projects/bzip2-rs/) and [Zstandard in Rust](/projects/zstd-rs/) project pages.
77
+
See the [zlib-rs](/projects/zlib-rs/), [bzip2-rs](/projects/bzip2-rs/) and [Zstandard in Rust](/projects/libzstd-rs/) project pages.
78
78
79
-
Please [get in touch with us](/support), if you are interested in financially supporting our data compression projects.
79
+
Please [get in touch with us](/support), if you are interested in financially supporting our data compression projects.
0 commit comments