few questions on maximum vector dimension, in memory index , on disk , flush

[ranjith502]

as per document

What is the maximum vector dimension supported in Milvus?
Milvus can manage vectors with up to 32,768 dimensions by default. You can increase the value of Proxy.maxDimension to allow for a larger dimension vector.

1. what is the maximum vector dimension it support like 100k , 50k , unlimited. it is unlimited how milvus is handling it
2. in memory index vs disk index
   as per my research, in memory index will store vectors , scalar , index in the memory. it will store everything in the memory only or in the disk also. during search both memory index and disk index will be loaded into memory
3. what is the main difference between memory index and disk index in terms of creation ,storing files , latency , accuracy ?
4. if enough data is not available index will not be created. during this case brut force method is used , brut force is nothing but comparing query with all the vectors. which brut force algorithms are used in the milvus
5. does flushing so many times will crash the milvus server
6. as i seen unloading collection is necessary because of memory sensitive i know it should be done manually. let say we did not done it. for how many hours it is loaded into memory, like 1 hour , 10 hours , as long as you unload it.

[xiaofan-luan]

What is the maximum vector dimension supported in Milvus? Milvus can manage vectors with up to 32,768 dimensions by default. You can increase the value of Proxy.maxDimension to allow for a larger dimension vector.

1. what is the maximum vector dimension it support like 100k , 50k , unlimited. it is unlimited how milvus is handling it

1. why is a vector have unlimited dimension? As far as a know there is know such model.
2. Milvus do support multiple flavor, check diskANN and mmap in our document. Using Zilliz cloud capacity tier and extended is the easiest way to utilize disk based index
3. memory index is targeting to 1000+ QPS, with latency typical under 100ms. While Disk index is targeting to 100+ QPS, with latency typically 100-200ms. You also need high performance disk to reach such performance.
4. Growing is not fully under bruteforce search, we built a special index named growing index on top of the data. It's not as fast compared to hnsw, but still 10 times faster compared to brute force search.
5. Yes, usually you don't really need to flush by yourself. Our suggestion is to rate limit flush to 1 per minute if you don't want to frequently flush.
6. Milvus team is working on a lazy load feature, which can evict unnecessary data to object storage and lazy load it back when query happened

[ranjith502]

What is the maximum vector dimension supported in Milvus? (please tell the maximum value )
can you explain growing index ?

[xiaofan-luan]

What is the maximum vector dimension supported in Milvus? (please tell the maximum value ) can you explain growing index ?

max dimension is 32768.

Growing index is a feature that automatically creates real-time indexes for streaming data to improve query performance1. This is important because Milvus handles two types of data - indexed data and streaming data.

Growing segments contain incremental data (recent updates), while sealed segments contain historical data2. Without growing index, Milvus can only search streaming data row by row, which impacts performance1.

In Milvus 2.3.4, a index was introduced for growing segments, resulting in up to 10x faster searches in growing segments. Under the hood growing index is a special ivf index or ScaNN index which can support updates