Commit 92f39d4
authored
perf: Parallelize package.json loading and reduce builder allocations (#11918)
## Summary
Parallelize the package.json loading phase and reduce allocations in the
package graph builder.
### Benchmarks (`--dry` runs, `--skip-infer`, 10 runs each, 5 warmup)
| Repo | Packages | Tasks | Before | After | Delta |
|------|----------|-------|--------|-------|-------|
| Large | ~1000 | 1690 | 2.107s ± 0.033s | 2.128s ± 0.203s | ~neutral
(noisy) |
| Medium | ~120 | ~200 | 1.292s ± 0.071s | 1.230s ± 0.078s | **1.05x
faster** |
| Small | ~5 | ~5 | 821.1ms ± 18.0ms | 812.7ms ± 15.6ms | **1.01x
faster** |
The medium repo shows the clearest improvement — it has enough packages
for rayon to help but isn't dominated by git subprocess overhead like
the large repo.
### Changes
**Parallel package.json loading**: `parse_package_jsons` previously
loaded and parsed each package.json file sequentially in a loop. Each
`PackageJson::load` call does disk I/O (`read_to_string`) and CPU-bound
JSON parsing (biome). These are independent per-package, so the loop is
replaced with `rayon::par_iter` to parallelize across all available
cores. The sequential `add_json` processing that mutates the builder is
unchanged.
**Entry API in `add_json`**: The old code called
`self.workspaces.insert(name.clone(), entry)` unconditionally, cloning
the `PackageName` on every call. On the error path (duplicate
workspace), it then did a second `get` + `clone` to retrieve the path.
Now uses `HashMap::entry()` to avoid the clone on the success path (only
clones once for `add_node`) and avoids the redundant lookup on the error
path.
**Capacity pre-allocation**: `workspaces` and `node_lookup` HashMaps are
now pre-allocated with `reserve(package_jsons.len())` before the
`add_json` loop, avoiding rehashing as entries are inserted.1 parent 6ac8c28 commit 92f39d4
File tree
3 files changed
+38
-18
lines changed- crates/turborepo-repository
- src/package_graph
3 files changed
+38
-18
lines changedSome generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
| |||
Lines changed: 36 additions & 18 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
266 | 266 | | |
267 | 267 | | |
268 | 268 | | |
269 | | - | |
270 | | - | |
271 | | - | |
272 | | - | |
273 | | - | |
274 | | - | |
275 | | - | |
276 | | - | |
277 | | - | |
278 | | - | |
279 | | - | |
280 | | - | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
281 | 285 | | |
282 | | - | |
283 | | - | |
284 | 286 | | |
285 | 287 | | |
286 | 288 | | |
| |||
293 | 295 | | |
294 | 296 | | |
295 | 297 | | |
296 | | - | |
297 | | - | |
298 | | - | |
299 | | - | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
300 | 315 | | |
301 | 316 | | |
302 | 317 | | |
303 | 318 | | |
304 | 319 | | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
305 | 323 | | |
306 | 324 | | |
307 | 325 | | |
| |||
0 commit comments