Open
Description
The intent of this issue is to expose low-level statistics to users about the memory requirements and usage of vectors in the system.
As vector data grows in the system it is imperative for search performance that certain elements of the index fit and are resident in memory, .e.g the HNSW graph and the quantized vectors. Performance can degrade significantly if either of these is paged out of memory. A separate issue is tracking using memory "better", see #125655. This issue will focus on providing statistics to help users better understand the memory requirements of vectors in their running systems.
Specifically,
- For an HNSW index, the approximate required off-heap memory to load the entire graph.
- For a quantized index, the approximate required off-heap memory to load the quantized vectors.
- Whether or not the index is resident in memory, and optionally what proportion of it is.
- The level of granularity should be at least at the shard level, if not per field.
Considerations:
- Do we need a new API for this, or can it be part of a new section of the node stats, say
dense_vector_field_stats. - How can we, or does it even make sense to try to, expose further details about the underlying page cache activity relating to the aforementioned index data.