Support io_uring for file I/O

[mox692]

This proposal suggests adding support for io_uring in Tokio's file API. The following provides the details of this proposal.

Proposal: https://hackmd.io/@ScaowLCjTCGW9slfxHgVHw/SkRCb0zCkx

Summary

• This proposal suggests adding support for io_uring in Tokio's file API.
• Initially, the goal is to transparently replace the file API backend with io_uring from a thread pool. Advanced features such as registered fds or registered buffers will probably be addressed in separate proposal.
• The application of io_uring to network I/O is outside the scope of this proposal.
• The implementation will happen incrementally.

Current Status

• Add initial infrastructure code (Changes to the driver, add cfg, Op future, etc)
  • rt: add infrastructure code for io_uring #7320
  • rt: add check for io_uring availability at runtime #7357
• Support io_uring in various fs operations
  • fs::OpenOptions (fs: add io_uring open operation #7321)
  • fs::write ( fs: support io_uring in fs::write #7567)
  • File::open() (fs: use FileOptions inside fs::File to support uring #7617)
  • fs::read (fs: support io_uring with tokio::fs::read #7696)
  • File::create()
  • AsyncRead for File
  • AsyncWrite for File (Implement uring in AsyncWrite for fs::File #7713)
  • statx(2) (Hard to implement Metadada, see fs: support io_uring in fs::metadata (statx) #7616)
  • Directory related APIs
  • fs::set_permissions()
  • other fs APIs
• Further improvements, such as:
  • Sharding rings in the multi-threaded runtime
  • Smarter batching logic for submission
  • Exploring the possibilities of using advanced features, such as registered buffers or registered files

[carllerche]

That sounds great to me. I think we should go ahead and move forward. It can be done incrementally.

We will need to add a new cfg feature since it will require new dependencies (at least initially). --cfg tokio_unstable_uring is probably what I would use (tokio_taskdump probably should have been tokio_unstable_taskdump). I would not do tokio_unstable_uring_fs as that implies other io_uring components will be added later via cfg flags. If we do add io_uring support for other components, it probably will not need additional features.

I am unsure, but we may also want to require runtime-level opt-in to io_uring support. For this, there would be a flag fs_mode(Mode::Uring(config)) on the runtime builder. We can worry about exposing op-level options later.

To handle the multi-threaded runtime, I would start with a single global submission queue and synchronize it. You are correct that it isn't ideal, but there isn't a mechanism yet to make one per worker. We should also make one timer per worker, which would be related. I would change the timer before worrying about sharding the io_uring submission queue.

[problame]

Came here via Discord: https://discord.com/channels/500028886025895936/500336346770964480/1361654607012692090

I'm excited to see activity on this.

I implemented the epoll-based approach similar to @mox692 dsecribes in their proposal linked at the top for our use in Neon's Pageserver: https://github.com/neondatabase/tokio-epoll-uring

We've been using it for many months across the fleet, first for buffered IO, recently direct IO for reads, now direct IO for writes.

Some feedback from skimming the RFC:

We didn't want to fork tokio, so, the addition of the io_uring FD to the tokio epoll instance is done via AsyncFd from a regular tokio task.
That is definitely suboptimal in a variety of ways, most significantly wrt hiding of "runnability".
But it avoids outside-of-runtime wakeups, which I have shown in benchmarks to be advantageous.

Wrt multi-threading, we have an io_uring instance and AsyncFd task per worker thread. Since tokio doesn't have a primitive for worker-thread-locals, we just use regular thread-locals that are only used from worker threads. (We don't use block_in_place, so, no risk of worker threads being stolen).

We opted for an owned-buffers-style API, similar to what tokio_uring does.
This avoids the need for a memory-to-memory copy into IO buffers but makes the API quite unergonomic / hard to drop-in-replace std::fs or current tokio::fs.

Also, as we are wrapping up the rollout (long & slow rollout over many past months), the "what's next" wish list is dominated by a different style of API, i.e., instead of operation().await, being able to submit a bunch of operations as we traverse our index data structure, then wait for all of them to complete at the end of traversal.

I think "high-end" users will not be satisfied with an operation().await style API.

I understand this issue here aims to remain compatible with tokio::fs, which is such a style of API.
That's going to be a big leap forward over what we have today, so I'm happy about that.

If you are open to scope-creep a bit, or at least prepare the design such that a split submit+complete style API is possible to implement from external crates, please reach out!

[one-d-wide]

Excited to see progress on this.

I poked around io_uring some time ago. And IIRC there is a bit of an incompatibility in how buffer operations are treated in io_uring vs. classic io interfaces.

AFAIK io_uring was designed to be compatible with direct memory access by hardware. This usually means that, for example, a buffer involved in a read operation must not be reused until this operation resolves, since this may lead to a race condition with an in-progress DMA request. So even if an operation was asynchronously canceled, a DMA request may still be pending.

This is in contrast to a classic io interfaces like a Read trait or a read syscall, which guarantee that once they return, the buffer is free to reuse.

As I understand, that's usually only an issue, when we allow an async request to be canceled before it resolves. For example as part of a Future, this typically necessitates an awkward api with owned buffers (like tokio-rs/tokio-uring have), or some kind of unsafe to acknowledge this possibility (like some embedded implementations have).

I guess it may be possible to implement this as part of a Drop trait of a read Future, which would cancel the request and synchronously wait until io_uring operation resolves, and hoping that it won't lock for too long. Or doing something clever inside the select block to make sure all io_uring operations are resolved before returning, since this is the place, where Futures likely to get canceled.

So I'm curious how this proposal addresses this issue. I'm also a bit rusty on this, so I might have gotten some details above wrong.

[Darksonn]

I know everyone is worried about this buffer thing but it's not a big problem. After all, using spawn_blocking to read and write like tokio::fs already does has the exact same buffer problem. When switching tokio::fs over to io_uring, we will just continue to use the same solution: copy the data into a buffer controlled by Tokio.

Now, down the road there are things such as TcpStream where this solution doesn't work. Primarily this is because it breaks behavior: it makes the stream behave like a BufWriter<TcpStream>. Probably we use IORING_OP_POLL_ADD to handle this case with identical behavior to epoll. Or we just continue to use epoll with the default tokio::net types and only use io_uring for cases such as tokio::fs.

[ADD-SP]

the goal is to transparently replace the file API backend with io_uring from a thread pool

This progress may not be "transparent", as io_uring requires higher permission than thread pool. see #7691 (comment).

[ileixe]

I'm glad that tokio community is pushing io_uring again since we maintained tokio-uring fork to support many io-uring features after tokio-uring maintenance stopped.

I know everyone is worried about this buffer thing but it's not a big problem. After all, using spawn_blocking to read and write like tokio::fs already does has the exact same buffer problem. When switching tokio::fs over to io_uring, we will just continue to use the same solution: copy the data into a buffer controlled by Tokio.

I think it's valid strategy to bring io-uring into tokio, but want to ask about the very next plan "after just working". Every serious application using io-uring requires zero-copy semantic and we have long-suffering ergonomics issue with OwnedBuf approach (passing/returning Buffer) as it's not composable. (We are using io-uring link feature which composed mulitple I/Os and it makes it worse). To my mind, it's inevitable to have new API (AsyncRead|write variant) for this owned buffer approach, but not sure what it looks like. Is there any place to discuss this approach? (At least I can't see anywhere)

[Darksonn]

Zero copy semantics with io_uring as well as linking multiple io_uring ops is not possible with the existing AsyncRead / AsyncWrite traits. We have not yet decided on how to support such use-cases. It's quite likely that, at least to begin with, such support will need to occur in a downstream crate.

Note that IO not based on the traits, e.g. the tokio::fs::{read,write} methods for reading and writing an entire file will not have this limitation, and they will be able to read/write directly into the buffer.

[marsevilspirit]

@Darksonn I'm very pleased to see the progress on io_uring, and I truly appreciate all your hard work on this.

I have a few questions I hope you could help clarify:

1. What is the current status of this effort, and when do you plan to release a version with io_uring support?
2. Will file I/O with io_uring support Direct I/O?

[Darksonn]

Please see the top of this issue for the current status. The first few functions are already supported today, see the changelog.

For direct I/O, please open a separate issue to discuss that.