Skip to content

Commit 7ce97a1

Browse files
committed
docs: 📚 write about results and background
1 parent 48c85f5 commit 7ce97a1

File tree

1 file changed

+52
-13
lines changed

1 file changed

+52
-13
lines changed

Diff for: README.md

+52-13
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,33 @@
1-
# Protobuf benchmarks
1+
# Protobuf Benchmarks
22

3-
This project contains benchmarks for the [ts-proto](https://www.npmjs.com/package/ts-proto), [google-protobuf](https://www.npmjs.com/package/google-protobuf), and [protobuf-ts](https://www.npmjs.com/package/protobuf-ts) libraries.
3+
This project benchmarks the performance of the following Protocol Buffers libraries for JavaScript and TypeScript:
44

5-
## Running the benchmarks
5+
- [ts-proto](https://www.npmjs.com/package/ts-proto)
6+
- [google-protobuf](https://www.npmjs.com/package/google-protobuf)
7+
- [protobuf-ts](https://www.npmjs.com/package/protobuf-ts)
68

7-
To run the benchmarks, clone this repository and run the following commands:
9+
## Getting Started
10+
11+
To run the benchmarks, clone this repository and execute the following script:
812

913
```bash
1014
./run_all.sh
1115
```
1216

13-
This script will install the dependencies and run the benchmarks for each library.
17+
This script installs the dependencies and runs the benchmarks for each library.
18+
1419

1520
## Results
1621

17-
### Read inputs
22+
### Environment
23+
24+
- **Machine**: MacBook Pro 16-inch, 2023
25+
- **Chip**: Apple M2 Max
26+
- **Memory**: 64 GB
27+
- **macOS**: 15.2
28+
- **Node.js**: 22.13.1
29+
30+
### Read Inputs
1831

1932
| Library | ops/sec | Average Time (ns) | Margin | Samples |
2033
| --------------- | ------- | ----------------- | ------ | ------- |
@@ -23,7 +36,7 @@ This script will install the dependencies and run the benchmarks for each librar
2336
| google-protobuf | 7,578 | 146,999.69 | ±0.82% | 68,028 |
2437
| protobuf-ts | 6,746 | 158,944.02 | ±0.59% | 62,916 |
2538

26-
### Update inputs
39+
### Update Inputs
2740

2841
| Library | ops/sec | Average Time (ns) | Margin | Samples |
2942
| --------------- | ------- | ----------------- | ------ | ------- |
@@ -32,10 +45,36 @@ This script will install the dependencies and run the benchmarks for each librar
3245
| google-protobuf | 6,314 | 179,756.73 | ±0.84% | 55,638 |
3346
| protobuf-ts | 4,919 | 216,651.99 | ±0.53% | 46,157 |
3447

35-
## Environment
48+
## Analysis
3649

37-
- **Machine**: MacBook Pro 16-inch, 2023
38-
- **Chip**: Apple M2 Max
39-
- **Memory**: 64 GB
40-
- **macOS**: 15.2 (24C101)
41-
- **Node.js**: 22.13.1
50+
### Background
51+
52+
Approximately one year ago, we migrated from `google-protobuf` to `ts-proto (V1)` to take advantage of its improved performance and modern TypeScript compatibility. With the recent release of `ts-proto (V2)`, we included it in our benchmarks, expecting further performance enhancements. However, the results reveal a **performance regression** in `ts-proto (V2)` compared to its predecessor.
53+
54+
### Key Insights
55+
56+
#### Read Inputs
57+
- **`ts-proto (V1)`** achieves the best performance with **8,078 ops/sec** and the lowest average latency of **131,998.64 ns**.
58+
- **`ts-proto (V2)`** shows a **13.7% decrease in throughput** compared to V1, achieving only **6,972 ops/sec**, with an average latency of **153,027.88 ns**.
59+
- **`google-protobuf`** performs competitively, achieving **7,578 ops/sec**, just slightly behind `ts-proto (V1)`.
60+
- **`protobuf-ts`** lags behind, delivering **6,746 ops/sec** and the highest latency of **158,944.02 ns**.
61+
62+
#### Update Inputs
63+
- **`ts-proto (V1)`** leads again, achieving **7,090 ops/sec** with an average latency of **148,810.31 ns**.
64+
- **`ts-proto (V2)`** suffers a significant **26.5% drop in throughput** compared to V1, with only **5,209 ops/sec** and a latency increase to **204,076.29 ns**.
65+
- **`google-protobuf`** performs better than `ts-proto (V2)` with **6,314 ops/sec**, but trails behind `ts-proto (V1)`.
66+
- **`protobuf-ts`** ranks the lowest, with **4,919 ops/sec** and the highest average latency of **216,651.99 ns**.
67+
68+
### Observations
69+
- The migration from `google-protobuf` to `ts-proto (V1)` last year proved to be a clear performance improvement.
70+
- The release of `ts-proto (V2)`, however, introduces regressions:
71+
- **Read Inputs:** Throughput drops by **13.7%**, and latency increases by **16%** compared to V1.
72+
- **Update Inputs:** Throughput drops by **26.5%**, and latency increases by **37%** compared to V1.
73+
- Despite its age, **`google-protobuf`** remains competitive, especially for read operations.
74+
- **`protobuf-ts`** shows consistently poor performance across both tasks, highlighting room for optimization.
75+
76+
## Conclusion
77+
78+
While `ts-proto (V2)` was expected to outperform its predecessor, the benchmarks reveal clear regressions in both throughput and latency. Any potential feature improvements in V2 should be carefully weighed against these performance drawbacks.
79+
80+
For now, **`ts-proto (V1)`** remains the best choice for performance in our use case.

0 commit comments

Comments
 (0)