Add more information to IOContext

[thecoop]

Description

Currently Lucene tends to use ReadAdvice.RANDOM quite liberally, in various hard-coded locations. This can cause some problems with recent kernel versions, such as #14408. There are also several other aspects about opening files - such as which file access implementation to use, how to map memory, etc, that are dependent on the context of opening such files, and cannot be determined solely by the codec that is used. For example, you may want to access a flat vector file differently if it’s being used for an exhaustive scan vs being used to access specific vectors.

To solve both these issues, Directory implementations could have information on the context a particular file is being opened with, to help it decide how and with what options it should open the file. However, there is no such information provided to Directory classes in the IOContext, and there is no space to provide such information in an extensible way.

So there needs to be a way to pass additional context-specific information in an IOContext, to help Directory implementations decide on the ReadAdvice to use, and how and with what options it should open a particular file. There are two main ways I can see of doing this:

1. Make IOContext an interface. This allows custom implementations to be created, which can then be checked for specific types inside the custom Directory implementation to then cast and pull out the required information (prototype can be found here)
2. Add a Map<String, Object> or Map<String, String> payload to IOContext allowing for arbitrary information to be included with an IOContext, that can then be checked by the Directory implementation

As part of this, some standard patterns or map keys could be used for existing Lucene directories to replace the existing ReadAdvice logic, to provide more information on how files should be opened. A later piece of work could be to remove ReadAdvice from IOContext, and provide some other way Directory implementations could determine the access pattern that is likely to be used.

[rmuir]

I like this idea of changing IOContext to provide hints to the Directory, rather than specifying the implementation. Not sure it has to be "flexible" or "customizable" initially, that could always come later as a followup. But it would be good to get the codecs out of the business of specifying the exact behavior that should happen, and instead just providing the "context".

[jpountz]

@rmuir I'm curious if you can expand a bit more on what you have in mind, what you are describing sounds to me like how things are today where ReadAdvice.RANDOM is the context and the MADV_RANDOM flag to madvise is the implementation. So I'm sure I'm missing something.

[jainankitk]

Looks related to the discussion here - #14348 (comment). More general, than specifically vector files

[rmuir]

@rmuir I'm curious if you can expand a bit more on what you have in mind, what you are describing sounds to me like how things are today where ReadAdvice.RANDOM is the context and the MADV_RANDOM flag to madvise is the implementation. So I'm sure I'm missing something.

I think today the context is trying too hard to direct the implementation.
For reference, the current structure looks like this:

public enum Context {
  MERGE,
  FLUSH,
  DEFAULT
};

public enum ReadAdvice {
  NORMAL,
  RANDOM,
  SEQUENTIAL,
  RANDOM_PRELOAD
}

// dictating implementation: let the Directory decide this
public static final IOContext DEFAULT = new IOContext(Constants.DEFAULT_READADVICE);
// dictating implementation: let the Directory decide this
public static final IOContext READONCE = new IOContext(ReadAdvice.SEQUENTIAL);

// a jazillion IOContext ctors taking different types of Objects, all of which
// set various defaults sneakily instead of letting Directory decide.

If i wanted to baby-step this, i'd walk the whole thing back to an int, as a start. I definitely think a codec should be able to OR together multiple different flags, and that the directory should be able to make use of them in a generic way. Feel free to translate this into EnumSets or whatever java does :)

This is just brainstorming and not properly thought out, so its just examples:

codec should be able to specify that the file is one of, say METADATA_FILE, DATA_FILE, INDEX_FILE (these might be bad names, but think slurped-metadata vs stored-fields-data vs stored-fields-index). It allows the Directory to make decisions based upon the general category of a file (or CFS range?), without hardcoded file extensions, etc. Ideally type-safe(ish) which is an improvement over matching "*.fdt" or other hacks.

codec should be able to specify the purpose of the file, say POSTINGS, STORED_FIELDS, VECTORS. Similar reasons to the above: give the "context", file extensions are not the solution. I don't even see any reason to pass .class names of formats here, that's also limiting. We should be able to tell the directory its TERMS, even though that's sorta an impl-detail of PostingsFormat and not a first-class codec api citizen. It is just a flag, let's be practical.

separately, codec should maybe be able to specify stuff such as whether file is accessed SEQUENTIAL or RANDOM: and thats just expressing what may happen, unrelated and disconnected from read-ahead or other possible implementations. But I've given this piece no thought, I think the first problem is that the Directory doesnt even have the "basic context" such as file's general category and purpose. And Directory should be in control of all "defaults", push those out of Codec, IOContext, etc.

[jpountz]

Thank you, I had started thinking along those lines but got blocked because I hadn't thought about using multiple "dimensions" for the context, ie. metadata/index/data is one dimension, terms/postings/vectors/etc. is another one. This sounds like a good step in the right direction.