Opening of vector files with ReadAdvice.RANDOM_PRELOAD

[viliam-durina]

Description

Vector similarity search using HNSW accesses the vectors very heavily during the search (the vec or veq files). Even more than the HNSW graph itself (the vex file). If the vector files don't fit into the page cache, the performance is reduced very significantly (around 100x in our particular case). Users typically configure their search servers to have enough RAM to fit these files.

Lucene currently uses ReadAdvice.RANDOM when opening these files. I think it would be better to use RANDOM_PRELOAD.

If you agree, I can provide a PR.

Version and environment details

No response

[DivyanshIITB]

Hi @viliam-durina,

I find this issue interesting and would like to work on it. I see that Lucene currently uses ReadAdvice.RANDOM when opening vector files (.vec and .veq). Switching to RANDOM_PRELOAD seems reasonable, especially if it helps reduce performance degradation when the files don’t fit in the page cache.

To proceed, I plan to:

Identify where ReadAdvice.RANDOM is used in Lucene’s codebase for opening vector files.
Modify it to ReadAdvice.RANDOM_PRELOAD.
Benchmark and test the performance impact to confirm the improvement.
Run the test suite to ensure no regressions.
Before making the change, I’d love to hear your thoughts:

Have you already tested the impact of RANDOM_PRELOAD, and does it consistently show improvements in performance?
Would you recommend making this change across all cases where ReadAdvice.RANDOM is used, or only for vector file access?
Looking forward to your feedback! If this is good to go, I’d be happy to submit a PR.

Thanks!

[viliam-durina]

I have the necessary change ready in my fork of Lucene, and it works for us. I wanted input from maintainers whether they think this is a good idea in general for Lucene.

[navneet1v]

@viliam-durina if you have benchmarks that shows the performance is better it will be good to raise the PR. Once PR is there maintainers can also do more tests to see if it is really the case.

ccing few maintainers @mikemccand , @benwtrent , @msokolov

[jainankitk]

Lucene currently uses ReadAdvice.RANDOM when opening these files. I think it would be better to use RANDOM_PRELOAD.

As per the documentation for RANDOM_PRELOAD:

This should only be used on very small files that can be expected to fit in RAM with very high confidence.

Hence, I am wondering if it can have adverse impact for cases having small page cache. While #13264 adds way to configure ReadAdvice, that applies to all file formats I guess. It would be great, if users can tune the ReadAdvice for specific formats, depending on the server page cache and data size.

[jpountz]

For what it's worth, it's possible to override the read advice of vectors with something like that:

Path path = ...;
Directory dir = new FilterDirectory(FSDirectory.open(path)) {
  @Override
  public IndexInput openInput(String name, IOContext context) throws IOException {
    if (name.endsWith(".vex") || name.endsWith(".vec") || name.endsWith(".veb")) {
      context = context.withReadAdvice(ReadAdvice.RANDOM_PRELOAD);
    }
    return super.openInput(name, context);
  }
};

[msokolov]

A question I have is how unused fields (at Search time) are handled. I guess their files may have been opened, and thus preloaded? But this could be bad. We often have people creating fields they plan to use for an experiment, but they sit dormant in the index for quite some time.

[viliam-durina]

I've ran into issue with this setting now. If the file doesn't actually fit into memory, this read advice hurts the performance significantly. With it, madvise is called with POSIX_MADV_NORMAL (see PosixNativeAccess.mapReadAdvice()), which on Linux preloads large blocks on each random read (16MB on my machine). This creates a huge read amplification.

I will probably end up using a custom directory that decides on its own to cache files in memory, and uses some kind of resource management, because it can potentially consume a lot of memory.