Skip to content

Read ahead performance tracking #483

@zuston

Description

@zuston

To reduce the latency of localfile reading, we introduce the page cache to prefetch the data from the localfile.
Currently, I'm applying this strategy into our internal spark jobs to observe the potential performance improvement, so this issue is to track these metrics or more next improvements

Impressive improvement of current impl

Image

Bugs fix

  1. Incorrect metric of read_ahead_misses #482
  2. Hang with read ahead when reading #475
  3. fix(read-ahead): Make sequential param as false by default #474
  4. feat(read-ahead): Correct hit metric and introduce 2 metrics to indicate latency #484

Potential improvements

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions