Skip to content

Threaded FS IO probably should be async now? #4159

Open
@rami3l

Description

@rami3l

Being able to use async where relevant across the codebase makes it possible to use async constructs where relevant. rustup has many places where that can be useful: downloading channel metadata, distribution content, unpacking that content to disk, are work that could usefully proceed in parallel - and async provides a good abstraction for that.

We already have a sophisticated disk IO layer that accomodates various OS latency-inducing behaviours, and adapting that to async without any regressions could be very interesting too - but for now, it co-exists nicely with an async core.
#3367

We are still using an async-unaware threadpool here, and it's known to produce problems (#3125):

/// Threaded IO model: A pool of threads is used so that syscall latencies
/// due to (nonexhaustive list) Network file systems, virus scanners, and
/// operating system design, do not cause rustup to be significantly slower
/// than desired. In particular the docs workload with 20K files requires
/// very low latency per file, which even a few ms per syscall per file
/// will cause minutes of wall clock time.

Anyway, using RUSTUP_IO_THREADS=1 to limit concurrency feels a bit off. Will migrating to async help?

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions