Can I use add_with_ids in HNSW index ?

[shiwanghua]

[shiwanghua]

I have a try, seems no problem:

import os
import json
import sys, datetime, time, struct
import faiss
import math
import numpy as np

start_time = datetime.datetime.now()
print(f'start_time={start_time}')

embedding_file_path = sys.argv[1]
index_file_path = sys.argv[2]


d = 768

total_embed_num = int(os.path.getsize(embedding_file_path) / (4*d))
embed_file = open(embedding_file_path, 'rb')
print(f'total_embed_num={total_embed_num}')

index = faiss.IndexFlatIP(d)
index = faiss.IndexHNSWFlat(d, 16, faiss.METRIC_INNER_PRODUCT)
index.hnsw.efConstruction = 256
index.hnsw.efSearch = 64
M=32
index.hnsw.set_default_probas(M, 1.0 / math.log(M));
index_with_ids = faiss.IndexIDMap2(index)


t1 = time.time()
for i in range(total_embed_num):
    one_emb = np.array(struct.unpack("f" * d, embed_file.read(4 * d)))
    one_emb = np.array([one_emb])
    index_with_ids.add_with_ids(one_emb, [i+10000])
    
t2 = time.time()
print(f'add {total_embed_num} embeddings cost {round(t2-t1, 1)}s, avg={round((t2-t1)*1000/total_embed_num, 4)}ms')

faiss.write_index(index_with_ids, index_file_path)

k = 1
distances, indices = index_with_ids.search(one_emb, k)
# print(f"{index_with_ids.id_map}")
print(f"The nearest neighbor of vec1 has ID: {distances} {indices}")
print(index_with_ids.reconstruct(19999)[0])


end_time = datetime.datetime.now()
time_diff = end_time - start_time
seconds_passed = round(time_diff.total_seconds(), 1)
print(f"end time: {end_time} cost {seconds_passed}s")

output:

start_time=2024-10-09 15:38:11.316468
total_embed_num=10000
add 10000 embeddings cost 8.5s, avg=0.8453ms
The nearest neighbor of vec1 has ID: [[1.]] [[**19999**]]
0.054972053
end time: 2024-10-09 15:38:19.814552 cost 8.5s

the output id of hnsw index is 19999 not 9999.

[shiwanghua]

faiss/tests/test_factory.py

Line 276 in af70c5b

# Index now works with add_with_ids, but not with add

here says index not works with add ? why ?

[shiwanghua]

By the way, not all index support this.

faiss/faiss/python/class_wrappers.py

Line 233 in af70c5b

"""Adds vectors with arbitrary ids to the index (not all indexes support this).

[ramilbakhshyiev]

faiss/tests/test_factory.py

Line 276 in af70c5b

# Index now works with add_with_ids, but not with add

here says index not works with add ? why ?

Because the index was created with IDMap and required identifiers. Would you expect a different behavior?

the output id of hnsw index is 19999 not 9999.

If I am reading your for-loop correctly then you're i+10000 will generate the index in the range of 10000-19999, no 0-9999. Please try remove +10000 and re-running your code.

[shiwanghua]

19999 is right.

[shiwanghua]

However, remove_id is not supported in index.