Skip to content

Speedup discovery of large workspaces#18311

Open
konstin wants to merge 3 commits intomainfrom
konsti/implement-workspace-caching
Open

Speedup discovery of large workspaces#18311
konstin wants to merge 3 commits intomainfrom
konsti/implement-workspace-caching

Conversation

@konstin
Copy link
Member

@konstin konstin commented Mar 5, 2026

This change avoids reading workspace root and member pyproject.toml multiple times for the same information. It remove all duplicate reading except for cache keys.

With the whole stack merged, it's ~1.9x faster for uv run python -V in airflow (ignoring the one uncacheable wheel with --offline):

$ hyperfine "uv-branch run --offline python -V" "uv-main run --offline python -V" --warmup 3
  Benchmark 1: uv-branch run --offline python -V
    Time (mean ± σ):      84.3 ms ±   3.2 ms    [User: 58.3 ms, System: 25.8 ms]
    Range (min … max):    80.8 ms …  98.2 ms    34 runs

  Benchmark 2: uv-main run --offline python -V
    Time (mean ± σ):     166.4 ms ±   2.8 ms    [User: 137.6 ms, System: 28.3 ms]
    Range (min … max):   162.8 ms … 172.1 ms    17 runs

  Summary
    uv-branch run --offline python -V ran
      1.97 ± 0.08 times faster than uv run --offline python -V

When analyzing the spans, reading the root airflow/pyproject.toml goes from 130 times to 2 times (regular read and cache keys), while for each workspace member, it goes from 4 to 2 reads (regular read and cache keys), removing 2 full reads each.

Before

image

After

image

@konstin konstin added the performance Potential performance improvement label Mar 5, 2026
/// Collect the workspace member projects from the `members` and `excludes` entries.
async fn collect_members(
/// Collect the workspace member projects and build the workspace object.
async fn build(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I might still prefer "collect"? or just "new"? I think "build" is overloaded

Base automatically changed from konsti/remove-project-discovery to konsti/workspace-refactorings March 5, 2026 14:40
@konstin konstin force-pushed the konsti/workspace-refactorings branch 2 times, most recently from 555d801 to a17a664 Compare March 6, 2026 10:28
Base automatically changed from konsti/workspace-refactorings to main March 6, 2026 10:37
konstin added 2 commits March 8, 2026 21:48
This doesn't actually change workspace discovery (anymore), so we can error after discovery.
This will allow caching.
@konstin konstin force-pushed the konsti/implement-workspace-caching branch 3 times, most recently from b5959c2 to b9b1c55 Compare March 8, 2026 22:30
After this change, we read each `pyproject.toml` exactly once for `uv run python` in airflow.

It's ~1.9x faster for `uv run python -V` in airflow (ignoring the one uncacheable wheel with `--offline`):

```
$ hyperfine "uv-branch run --offline python -V" "uv-main run --offline python -V" --warmup 3
  Benchmark 1: uv-branch run --offline python -V
    Time (mean ± σ):      84.3 ms ±   3.2 ms    [User: 58.3 ms, System: 25.8 ms]
    Range (min … max):    80.8 ms …  98.2 ms    34 runs

  Benchmark 2: uv-main run --offline python -V
    Time (mean ± σ):     166.4 ms ±   2.8 ms    [User: 137.6 ms, System: 28.3 ms]
    Range (min … max):   162.8 ms … 172.1 ms    17 runs

  Summary
    uv-branch run --offline python -V ran
      1.97 ± 0.08 times faster than uv run --offline python -V
```
@konstin konstin force-pushed the konsti/implement-workspace-caching branch from b9b1c55 to bdab9cb Compare March 8, 2026 23:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Potential performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants