Skip to content

RFC: Introduce io-uring for localfile storage #554

@zuston

Description

@zuston

Motivation

According to the paper io_uring: Rethinking Asynchronous I/O for Storage Systems, io_uring can significantly improve CPU utilization and I/O latency for storage systems, especially those with a buffer manager.

Based on this observation, Riffle introduces an io_uring-based I/O handler (UringIO) to optimize local shuffle storage performance.

Prerequisites

  1. Linux kernel version >= 5.10
  2. Currently verified on Anolis OS 8

Conclusions

  1. With io_uring enabled and 16 threads per disk (still under tuning), CPU load is reduced by approximately 3× compared to the non-io_uring implementation.
    Update: Further tuning shows that 2 threads per disk are sufficient, achieving comparable overall throughput.
  2. After enabling io_uring:
  1. Write throughput reaches 5 GB/s, compared to 3.75 GB/s without io_uring (~25% improvement).
  2. Read throughput shows no significant improvement, which is expected because:
  3. Read requests are not intensive
  4. Each read operation transfers relatively large data blocks
  5. io_uring provides limited benefits for this access pattern

Benchmark

The benchmark runs a 2.2 TB TeraSort application, with a single Riffle server handling all shuffle data.

Performance Results

Type Write Time Read Time
With io_uring 3.4 min 6.3 min
Without io_uring 4.1 min 8.3 min

cpu load comparison

Image

Subtasks

Sub-issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions