Skip to content

Commit 3655f8f

Browse files
saenyakornsaenyakorn
committed
chore: update README.md and update version of libs
1 parent 2d117e2 commit 3655f8f

File tree

5 files changed

+180
-60
lines changed

5 files changed

+180
-60
lines changed

‎README.md

+132-25
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,156 @@
11
# Schema Validation Benchmark <!-- omit in toc -->
22

3-
This repository aims to provide a benchmark of CPU and RAM usage during Typescript-compilation. This benchmark is heavily used [Typebox](https://github.com/sinclairzx81/typebox) as a codebase for the benchmark.
3+
At [Softnetics Team](https://www.softnetics.tech/), we specialize in building software using TypeScript. Data validation is a crucial part of our work, as we need to verify and convert user input into the correct format. Initially, we chose [Zod](https://zod.dev/) as our schema validation library. However, as our applications grew larger, we noticed high CPU and memory usage during TypeScript compilation, particularly in development. Our research revealed that these performance issues stemmed from the validation library itself. This led us to conduct a benchmark study comparing CPU and memory usage across different schema validation libraries.
44

55
# Table of Contents <!-- omit in toc -->
66

77
- [Environment](#environment)
8-
- [Data Preparation](#data-preparation)
9-
- [Benchmark](#benchmark)
8+
- [System Information](#system-information)
9+
- [Library Version](#library-version)
10+
- [Test Cases Explanation](#test-cases-explanation)
11+
- [Test Case 1: Simple](#test-case-1-simple)
12+
- [Test Case 2: Extend](#test-case-2-extend)
13+
- [Test Case 3: Union](#test-case-3-union)
14+
- [Test Case 4: Complex](#test-case-4-complex)
15+
- [Additional Test Cases 1: Transform](#additional-test-cases-1-transform)
16+
- [Running the benchmark](#running-the-benchmark)
17+
- [Benchmark Result](#benchmark-result)
18+
- [Check Time](#check-time)
19+
- [Parse Time](#parse-time)
20+
- [Memory Used](#memory-used)
21+
- [Number of Types](#number-of-types)
1022
- [Summary](#summary)
11-
- [Add more data or candidates](#add-more-data-or-candidates)
23+
- [Add more testcases or candidates](#add-more-testcases-or-candidates)
24+
- [Add more testcases](#add-more-testcases)
25+
- [Add more candidates](#add-more-candidates)
1226
- [Run the benchmark yourself](#run-the-benchmark-yourself)
1327

1428
# Environment
1529

16-
- **OS**: ubuntu-22.04 vCPU 4 | RAM 16GB (via Github Action)
17-
- **Node.js**: 20.16.0
18-
- **Bun** 1.1.10
19-
- **Pnpm** 9.14.2
20-
- **Python** 3.13.0
30+
## System Information
2131

22-
| Library | Version |
23-
| -------------- | ------------ |
24-
| zod | 0.67.13 |
25-
| typebox | 0.32.31 |
26-
| arktype | 2.0.0-dev.16 |
27-
| valibot | 0.31.0-rc.5 |
28-
| yup | 1.4.0 |
29-
| @effect/schema | 0.13.1 |
32+
| Component | Version | Note |
33+
| --------- | ---------------------------- | ------------- |
34+
| OS | ubuntu-22.04 vCPU 4 RAM 16GB | GitHub Action |
35+
| Node.js | 20.16.0 | |
36+
| Bun | 1.1.10 | |
37+
| Pnpm | 9.14.2 | |
38+
| Python | 3.13.0 | |
3039

31-
# Data Preparation
40+
## Library Version
3241

33-
TODO
42+
| Library | Version |
43+
| ----------------------------------------------------- | ------------- |
44+
| [Zod](https://zod.dev/) | 3.24.1 |
45+
| [typebox](https://github.com/sinclairzx81/typebox) | 0.34.14 |
46+
| [arktype](https://arktype.io/) | 2.0.3 |
47+
| [valibot](https://valibot.dev/) | 1.0.0-beta.14 |
48+
| [yup](https://github.com/jquense/yup) | 1.6.1 |
49+
| [@effect/schema](https://github.com/Effect-TS/effect) | 0.75.5 |
3450

35-
# Benchmark
51+
# Test Cases Explanation
3652

37-
TODO
53+
For this benchmark, we generated test cases using [typebox-codegen](https://github.com/sinclairzx81/typebox-codegen). Each test case was carefully crafted with specific objectives and varying levels of complexity. To ensure a fair and meaningful comparison, we verified that all libraries produced identical TypeScript type outputs. You can find all the test cases in the [samples](./samples) directory.
54+
55+
## Test Case 1: Simple
56+
57+
The "simple" test case is a basic schema with a single object containing a string and array field.
58+
59+
## Test Case 2: Extend
60+
61+
The "extend" test case is a schema use "extend" feature of each library.
62+
63+
## Test Case 3: Union
64+
65+
The "union" test case is a schema use "union" feature of each library. Especially, discriminate union.
66+
67+
## Test Case 4: Complex
68+
69+
The "complex" test case is a schema use common Typescript type helper e.g. `Extract`, `Omit`, `Union`, `Extend`, etc. combined all of the above together.
70+
71+
## Additional Test Cases 1: Transform
72+
73+
This test cases will use `transform` feature of each library and will infer Input and Output type. As of now, only [Zod](https://zod.dev/) and [Valibot](https://valibot.dev/) support this feature.
74+
75+
For further information about the data preparation can be found in the [Samples README.md](./samples/README.md).
76+
77+
# Running the benchmark
78+
79+
The benchmark is run by [GitHub Action](https://github.com/features/actions) with the following step.
80+
81+
1. Generate the test cases for each library using [typebox-codegen](https://github.com/sinclairzx81/typebox-codegen).
82+
2. Run the benchmark using `tsc --extendedDiagnostics` each generated files to get the semantic diagnostics e.g. memory usage, compile time, etc.
83+
84+
- After this step we will get the result in the `./samples/__benchmarks__` directory. which contains the output of the `tsc --extendedDiagnostics` command. For more information about the output, you can refer to the [TypeScript documentation](https://github.com/microsoft/TypeScript/wiki/Performance).
85+
86+
3. Read the result and generate the report using [Pandas](https://pandas.pydata.org/) and [Matplotlib](https://matplotlib.org/).
87+
88+
# Benchmark Result
89+
90+
The **TypeScript** candidate performs the best across all metrics, as it leverages native TypeScript types. However, it is not a schema validation library. It can serve as a reference point for comparing the performance of other libraries.
91+
92+
## Check Time
93+
94+
Check time refers to the duration taken by the TypeScript compiler to check and infer the types of the program. A lower check time indicates better performance and an enhanced developer experience.
95+
96+
![](./reports/__assets__/check_time.png)
97+
98+
[Zod](https://zod.dev/) exhibits the highest check time, which is approximately twice as long as the second-highest check time, [Effect](https://effect.website/) , in all test cases. Both [Valibot](https://valibot.dev/) and [Effect](https://effect.website/) show similar check times, but they are about twice as long as the other libraries.
99+
100+
## Parse Time
101+
102+
Parse time is the time required by the TypeScript compiler to generate Abstract Syntax Trees (ASTs) for the program. A lower parse time generally indicates simpler code, which improves editor performance and enhances the developer experience.
103+
104+
![](./reports/__assets__/parse_time.png)
105+
106+
As shown, [Effect](https://effect.website/) consistently has the highest parse time across all test cases, suggesting that it has the most complex type system. The other libraries show no significant difference in parse time compared to TypeScript, which serves as the baseline.
107+
108+
## Memory Used
109+
110+
Memory usage refers to the amount of memory consumed by the TypeScript compiler when executing the `tsc` command. This metric is crucial for developer experience.
111+
112+
![](./reports/__assets__/memory_used.png)
113+
114+
As depicted, most libraries exhibit similar memory usage across all test cases, with the exception of [Effect](https://effect.website/), which consistently uses the most memory. The other libraries do not show a significant difference in memory usage when compared to TypeScript, the baseline.
115+
116+
## Number of Types
117+
118+
The number of types indicates how many types are generated by the schema file. A lower number of types leads to better performance, as the TypeScript compiler needs to check fewer types and generate smaller ASTs.
119+
120+
![](./reports/__assets__/types.png)
121+
122+
Interestingly, [Zod](https://zod.dev/) stands out as an outlier in this metric, while the other libraries exhibit little variation in the number of types, compared to TypeScript.
38123

39124
# Summary
40125

41-
TODO
126+
The best schema validation library is [Yup](https://github.com/jquense/yup) and [io-ts](https://github.com/gcanti/io-ts) due to its low memory usage and quick compile time. However, it's important to note that they lacks certain advanced features, such as `transform`, `refine`, and `discriminatedUnion`, which are commonly used in real-world applications. Therefore, the best schema validation library for the [Softnetics Team](https://www.softnetics.tech/) would be [Valibot](https://valibot.dev/), as it is the fastest feature-rich library available and Typescript-first schema validation like [Zod](https://zod.dev/).
127+
128+
Unfortunately, the [Valibot](https://valibot.dev/) community is not as large as that of [Zod](https://zod.dev/). If having a strong community is a priority, [Zod](https://zod.dev/) remains the best option for now.
129+
130+
Let me know if you need further refinements!
131+
132+
The best schema validation libraries in terms of performance are [Yup](https://github.com/jquense/yup) and [io-ts](https://github.com/gcanti/io-ts), owing to their low memory usage and quick compilation times. However, they lack certain advanced features, such as `transform`, `refine`, and `discriminatedUnion`, which are often required in real-world applications. Therefore, for the [Softnetics Team](https://www.softnetics.tech/), the ideal choice would be [Valibot](https://valibot.dev/), as it combines speed, rich feature support, and a TypeScript-first approach, similar to [Zod](https://zod.dev/).
133+
134+
It’s worth noting that the [Valibot](https://valibot.dev/) community is smaller than that of [Zod](https://zod.dev/). If a strong community is a priority, [Zod](https://zod.dev/) remains the best option for now.
135+
136+
Let me know if you need further refinements!
137+
138+
# Add more testcases or candidates
139+
140+
## Add more testcases
141+
142+
To add more testcases ou can follow the steps below.
143+
144+
1. Add a new test case in the [samples](./samples) directory. [common](./samples/common) directory contains common test cases that can be used in multiple libraries. [custom](./samples/custom) directory contains test cases that are specific to a library.
145+
2. Open Pull Request to this repository.
146+
147+
## Add more candidates
42148

43-
# Add more data or candidates
149+
To add more candidates you can follow the steps below.
44150

45-
TODO
151+
1. Go to [src/constants/library.ts](./src/constants/library.ts) and add a new candidate.
152+
2. Open Pull Request to this repository.
46153

47154
# Run the benchmark yourself
48155

49-
TODO
156+
To run the benchmark yourself, you can follow the step which described in [GitHub Action](./.github/workflows/benchmark.yaml) file.

‎package.json

+3-3
Original file line numberDiff line numberDiff line change
@@ -15,14 +15,14 @@
1515
"license": "ISC",
1616
"dependencies": {
1717
"@effect/schema": "^0.75.5",
18-
"@sinclair/typebox": "^0.34.13",
18+
"@sinclair/typebox": "^0.34.14",
1919
"@trivago/prettier-plugin-sort-imports": "^4.3.0",
2020
"@typescript-eslint/typescript-estree": "^7.10.0",
21-
"arktype": "2.0.0-rc.33",
21+
"arktype": "2.0.3",
2222
"io-ts": "^2.2.22",
2323
"prettier": "^3.2.5",
2424
"typescript": "^5.4.5",
25-
"valibot": "1.0.0-beta.11",
25+
"valibot": "1.0.0-beta.14",
2626
"yup": "^1.6.1",
2727
"zod": "^3.24.1"
2828
},

‎pnpm-lock.yaml

+24-24
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎reports/benchmark_plotter.py

+10-3
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,11 @@
22
import matplotlib.pyplot as plt
33
import os
44

5+
import logging
6+
7+
logger = logging.getLogger(__name__)
8+
9+
510
class BenchmarkPlotter:
611
def __init__(self, benchmarks: pd.DataFrame, dir: str):
712
cwd = os.getcwd()
@@ -14,7 +19,7 @@ def __init__(self, benchmarks: pd.DataFrame, dir: str):
1419
"typescript": "#d62728",
1520
"valibot": "#9467bd",
1621
"yup": "#8c564b",
17-
"zod": "#e377c2",
22+
"zod": "#e377c2",
1823
}
1924
self.metrics = [
2025
{"name": "Memory used", "unit": "Megabytes"},
@@ -26,9 +31,11 @@ def __init__(self, benchmarks: pd.DataFrame, dir: str):
2631
{"name": "Symbols", "unit": "Count"},
2732
{"name": "I/O Read time", "unit": "Seconds"},
2833
{"name": "Files", "unit": "Count"},
34+
{"name": "Parse time", "unit": "Seconds"},
2935
]
3036

3137
def plot(self):
38+
logger.info("Plotting benchmarks")
3239
for metric in self.metrics:
3340
self._plotMetric(metricName=metric["name"], unit=metric["unit"])
3441

@@ -37,9 +44,9 @@ def _plotMetric(self, metricName: str, unit: str):
3744
df = df[["Name", "Library", metricName]]
3845
df = df.sort_values(by=[metricName], ascending=False)
3946
grouped = df.groupby(by=["Name"])
40-
47+
4148
fig = plt.figure(figsize=(8, 12))
42-
fig.suptitle(f'{metricName} comparison', fontsize=24)
49+
fig.suptitle(f"{metricName} comparison", fontsize=24)
4350

4451
for i, (names, group) in enumerate(grouped):
4552
ax = plt.subplot(len(grouped), 1, i + 1)

0 commit comments

Comments
 (0)