Similarity Search in FAISS Returning Raw, Unintelligible Data

[Rajat-2001]

Summary

When performing similarity search using FAISS (Facebook AI Similarity Search), the results are often returned as raw, low-level vector data that isn't human-readable or useful without additional processing. Instead of meaningful textual data or relevant objects, the output is composed of unintelligible characters and symbols, representing the vectorized data internally.
Example Output:
Rank: 1, Distance: 1.629706859588623, Text: M *M 4M JM pM M M N qN N N N O TO \O ]O {O O P hP ~P P P IQ lQ Q Q Q Q FR XR ~R R R =S S S T T ;T |T T T T [U \U U U +V KV UV dV uV V V W W W W $X 4X X X X
Rank: 2, Distance: 1.6545774936676025, Text: F F F F F G G H H PH RH nH I -I EI HI ZI I I I J J J J K K =L DL #M oM M M M ;N N N N sO O O LP P P P *Q 7Q TQ _Q Q Q R dR R ;S kS T KT T T T T T !U #U

This behavior is expected from FAISS, as it returns high-dimensional vectors during similarity searches. However, it’s not helpful to end users without further translation into meaningful data such as text, image references, or other objects.

Platform

OS: Linux/Ubuntu 22.04
Faiss version: 1.7.2
Faiss compilation options: Compiled with CUDA support

OS:

Faiss version:

Installed from:

Faiss compilation options:

Running on:

• GPU

Interface:

• Python

Reproduction instructions

Install FAISS:

pip install faiss-cpu (for CPU version)
pip install faiss-gpu (for GPU version, if applicable)

Create a FAISS index and add data:
import faiss
import numpy as np

Create random data to simulate a vector search

d = 512 # Dimensionality of the vectors
nb = 1000000 # Number of vectors (adjust as needed)
np.random.seed(1234)
data = np.random.random((nb, d)).astype('float32')

Create FAISS index using L2 distance

index = faiss.IndexFlatL2(d)
index.add(data)

Perform a search with a random query vector

query = np.random.random((1, d)).astype('float32')
D, I = index.search(query, k=5)

Output the results (This is where the raw data appears)

for rank, (distance, idx) in enumerate(zip(D[0], I[0])):
print(f"Rank: {rank+1}, Distance: {distance}, Text: {data[idx]}")

Expected Output: The output should ideally show human-readable data or objects that are similar to the input query.

Example Expected Output:Rank: 1, Distance: 0.923, Text: "Some relevant text or object description"
Rank: 2, Distance: 1.023, Text: "Another relevant item"

Actual Output: Instead of meaningful text or objects, the output returns raw vector data that’s not interpretable without further processing, like:Rank: 1, Distance: 1.629706859588623, Text: M *M 4M JM pM M M N qN N N N O TO \O ]O {O O P hP ~P P P IQ lQ Q Q Q Q FR XR ~R R R =S S S T T ;T |T T T T [U \U U U +V KV UV dV uV V V W W W W $X 4X X X X
Rank: 2, Distance: 1.6545774936676025, Text: F F F F F G G H H PH RH nH I -I EI HI ZI I I I J J J J K K =L DL #M oM M M M ;N N N N sO O O LP P P P *Q 7Q TQ _Q Q Q R dR R ;S kS T KT T T T T T !U #U

The output here is raw data that represents the internal vector space from FAISS, which is not directly human-readable.

[mengdilin]

I see the vector representation of the text when running the same code e.g.

Rank: 1, Distance: 69.29815673828125, Text: [7.26770699e-01 6.38433099e-01 7.17456996e-01 6.25663102e-01
 4.13274497e-01 3.17566991e-01 6.20673537e-01 8.19755375e-01
 3.92765030e-02 9.64170456e-01 9.32426333e-01 5.62602580e-01
 8.09240520e-01 3.29693556e-01 3.40192676e-01 5.08803189e-01
 9.82578814e-01 7.94248879e-01 3.18872154e-01 7.20685780e-01
 3.01172167e-01 8.53561640e-01 2.45084807e-01 5.85401535e-01
 6.81033209e-02 7.32078373e-01 3.21439683e-01 2.34193936e-01
 8.41091633e-01 2.11577445e-01 2.87554830e-01 7.54857361e-01
 8.10208976e-01 9.54944849e-01 7.03447089e-02 2.17977345e-01
 3.20670217e-01 5.99182546e-01 2.70829231e-01 9.52204287e-01
 4.71602887e-01 4.82598871e-01 3.16621184e-01 5.74625134e-01
 2.16968536e-01 8.94633532e-01 8.37666512e-01 9.14824605e-01
 5.70379198e-01 5.65513432e-01 2.84823656e-01 6.71040893e-01
 5.34722209e-01 9.03457642e-01 2.74353981e-01 9.23455656e-02
 1.70716792e-01 6.11695826e-01 6.61235988e-01 3.46089527e-02
 3.95379603e-01 5.00676215e-01 3.55197459e-01 8.34308743e-01
 4.73491848e-02 9.35619056e-01 2.92269826e-01 8.03318501e-01
 8.50504577e-01 4.10957426e-01 4.09326524e-01 2.30254039e-01
 1.20658837e-01 3.91223758e-01 9.60945368e-01 3.85778248e-01
 2.66806960e-01 8.17217350e-01 2.41987053e-02 7.48571634e-01
 9.60616767e-01 7.44464755e-01 3.94631416e-01 3.16810787e-01
 3.23021770e-01 2.15608031e-01 1.36835650e-01 1.10203534e-01
 6.57518387e-01 7.44250894e-01 5.44475734e-01 6.29573584e-01
 2.69854277e-01 8.51293743e-01 8.76383662e-01 3.33426803e-01
 8.87680888e-01 9.19269323e-02 4.69556868e-01 9.94380474e-01
 1.43440306e-01 3.07244807e-01 1.27684876e-01 6.73267305e-01
 8.19403291e-01 8.16067338e-01 3.10232639e-01 9.22483146e-01
 3.82380502e-04 7.12397814e-01 9.11591411e-01 5.54751992e-01
 1.93134546e-01 6.15529776e-01 8.87718916e-01 2.74926633e-01
 5.41873693e-01 3.13288838e-01 3.61461103e-01 4.33548123e-01
 5.25278375e-02 6.96821213e-01 2.88641274e-01 1.39397517e-01
 5.58833003e-01 4.53423709e-01 8.49995196e-01 4.58396912e-01
 7.68779516e-01 5.84596336e-01 5.72384238e-01 2.53242091e-03
 5.50106943e-01 4.05040741e-01 3.50630641e-01 6.45490289e-01
 6.29034281e-01 2.25412138e-02 8.25324774e-01 4.40076441e-01
 4.57791477e-01 8.48059714e-01 6.11849904e-01 8.29669178e-01
 6.45171583e-01 8.49531949e-01 4.05339450e-01 5.28900623e-01
 3.08069825e-01 4.22934175e-01 9.12979364e-01 5.53172410e-01
 3.98834109e-01 5.67785263e-01 4.76955958e-02 2.52866536e-01
 4.00303423e-01 3.67616594e-01 5.59711695e-01 2.94444919e-01
 5.73152959e-01 9.36590970e-01 9.51535761e-01 9.27880406e-01
 3.03476542e-01 9.63409364e-01 2.53545195e-01 7.59987116e-01
 1.80584580e-01 4.15373266e-01 8.35560918e-01 4.48630899e-01
 6.09990060e-01 7.52243817e-01 3.51553947e-01 2.57813364e-01
 3.64143372e-01 2.99699277e-01 6.29847944e-01 6.57752991e-01
 9.00020778e-01 9.97879267e-01 6.28611684e-01 1.71783879e-01
 6.45208359e-01 6.58518076e-01 2.30402216e-01 4.33801264e-01
 5.87040298e-02 1.81988433e-01 6.71486109e-02 1.79647982e-01
 5.26141152e-02 9.86787379e-01 3.34057450e-01 6.01651847e-01
 2.68848956e-01 9.66961324e-01 3.81063014e-01 8.38388979e-01
 3.72829109e-01 8.36083770e-01 5.84495366e-01 2.50578970e-01
 2.39200503e-01 4.82463926e-01 9.75472450e-01 3.36347699e-01
 4.72989798e-01 6.94433272e-01 7.78091431e-01 6.50474131e-01
 1.18017979e-01 3.49368721e-01 3.21542978e-01 9.02540088e-01
 4.02695507e-01 9.31950629e-01 7.01920867e-01 4.09651190e-01
 8.62137198e-01 4.77156080e-02 8.05161655e-01 8.22103083e-01
 3.74917269e-01 1.56639874e-01 2.85614103e-01 2.33624458e-01
 2.11461976e-01 5.87269604e-01 1.31293029e-01 6.53850198e-01
 6.15892172e-01 1.76904127e-01 6.89722121e-01 5.74428499e-01
 6.98705256e-01 8.16782176e-01 5.91668844e-01 5.03992498e-01
 6.25681758e-01 3.57546031e-01 4.57891315e-01 2.07583994e-01
 7.38872647e-01 6.00446522e-01 6.95730627e-01 8.39388490e-01
 7.49211252e-01 4.47497785e-01 6.38065875e-01 5.27650595e-01
 4.72452968e-01 1.25888791e-02 7.57930040e-01 6.48417532e-01
 8.31148922e-01 2.30553001e-01 8.11302185e-01 8.62611115e-01
 4.91125524e-01 9.95548844e-01 3.35266292e-02 8.26502621e-01
 9.05362904e-01 1.78687438e-01 5.78079879e-01 5.68980873e-01
 6.84939027e-01 5.11514902e-01 1.18847981e-01 8.48462522e-01
 2.57661521e-01 6.00325584e-01 8.50148439e-01 9.04403508e-01
 4.63459268e-02 4.52025533e-01 4.24811780e-01 1.43267974e-01
 2.60401249e-01 1.91627249e-01 3.32747966e-01 4.16020565e-02
 5.49268961e-01 1.90887943e-01 6.59948111e-01 5.77210724e-01
 9.80534732e-01 6.08775198e-01 1.12269282e-01 3.39519173e-01
 9.03885245e-01 1.23486578e-01 8.10197413e-01 5.68242431e-01
 8.18315327e-01 9.50449646e-01 6.31682098e-01 1.18670464e-01
 4.57886159e-01 5.60076475e-01 9.97387111e-01 1.59717560e-01
 1.65036023e-01 3.67213398e-01 1.46543413e-01 7.85679042e-01
 7.26890683e-01 9.72623050e-01 4.62227941e-01 7.05032200e-02
 3.38958383e-01 9.16778743e-01 6.15536451e-01 3.81415427e-01
 2.86630422e-01 1.75583974e-01 7.65221298e-01 8.88263047e-01
 9.41107571e-01 3.50607932e-01 1.49979204e-01 5.85359871e-01
 8.23389471e-01 9.30092216e-01 4.99385476e-01 7.35199034e-01
 9.45481777e-01 8.86660874e-01 8.04166019e-01 9.23725218e-02
 9.20478582e-01 2.73312420e-01 1.82770371e-01 3.56082618e-02
 5.52357793e-01 1.08847637e-02 4.87238169e-01 1.56724066e-01
 4.19165283e-01 6.10594571e-01 8.92210066e-01 4.83960539e-01
 6.86560988e-01 1.78732559e-01 7.72807240e-01 2.18834002e-02
 7.16416299e-01 5.97097814e-01 6.64276024e-03 4.95598435e-01
 3.19792569e-01 5.90073168e-01 3.47166583e-02 3.56459111e-01
 5.82975030e-01 5.08506715e-01 5.10040879e-01 1.69427976e-01
 8.71559680e-01 8.39558005e-01 1.43014312e-01 4.76977348e-01
 2.90718257e-01 4.42341983e-01 5.83186209e-01 7.74376810e-01
 7.58755088e-01 2.80820400e-01 2.70818084e-01 4.94826019e-01
 5.36142111e-01 5.64129412e-01 1.21686675e-01 7.42299438e-01
 5.55993058e-03 3.19407135e-01 6.92150235e-01 4.38720793e-01
 3.75892580e-01 5.47786236e-01 5.05825162e-01 8.74485791e-01
 2.24471003e-01 1.49439156e-01 2.61313617e-01 8.09288025e-01
 4.57509816e-01 7.69144371e-02 5.50861120e-01 4.76475090e-01
 4.88902390e-01 4.94492024e-01 5.35943985e-01 5.34795463e-01
 5.26771545e-01 5.60858727e-01 3.20662372e-02 5.49234636e-02
 5.25732875e-01 6.10492289e-01 2.95122832e-01 7.38496840e-01
 3.43655169e-01 3.18140835e-01 3.88816565e-01 9.36832666e-01
 7.87969306e-02 9.51858759e-01 7.46736705e-01 2.57803768e-01
 6.47841752e-01 6.63486779e-01 3.77054870e-01 5.67536294e-01
 5.24720550e-01 2.34724939e-01 8.19407105e-01 4.87701148e-01
 1.01412363e-01 6.75445855e-01 7.04940334e-02 9.74214435e-01
 7.08717287e-01 9.73450541e-01 5.30135274e-01 4.81482506e-01
 3.45458359e-01 6.70208931e-01 7.06467628e-01 5.22464037e-01
 8.61217618e-01 8.52096558e-01 2.76120007e-01 7.56660819e-01
 2.30608676e-02 5.71804047e-01 1.12188026e-01 2.47684628e-01
 3.44020158e-01 7.54366338e-01 1.11063108e-01 6.44747198e-01
 4.84013520e-02 2.29549482e-01 4.03388679e-01 5.00896215e-01
 5.02678454e-01 2.63110489e-01 2.97306716e-01 3.51479292e-01
 6.48983061e-01 9.40732181e-01 6.78795695e-01 8.63719523e-01
 6.27009213e-01 2.68746167e-01 4.23323810e-01 8.86932969e-01
 8.96655440e-01 5.98086774e-01 5.25593162e-01 5.06420374e-01
 9.96521235e-01 2.69854248e-01 6.63113058e-01 9.85658050e-01
 5.81501245e-01 4.06894237e-01 2.49648452e-01 7.31917560e-01
 1.32461980e-01 9.27558899e-01 6.31870508e-01 7.05488503e-01
 1.66608021e-02 3.01463336e-01 6.25481606e-01 9.29789007e-01
 1.40320629e-01 3.64858568e-01 5.33535779e-01 7.78553843e-01
 2.24361852e-01 6.49385273e-01 1.76367208e-01 6.64262176e-01
 9.27166760e-01 2.45096803e-01 8.56748283e-01 6.15876198e-01
 3.94546598e-01 5.20040393e-01 5.87387621e-01 4.34821486e-01
 2.01727420e-01 4.53531593e-01 4.51523930e-01 6.53554201e-01
 9.24825251e-01 4.06877011e-01 4.43869308e-02 2.63883203e-01
 8.20483208e-01 7.81680763e-01 9.17134702e-01 7.32072651e-01
 8.56605530e-01 8.91052961e-01 5.93107283e-01 7.56247997e-01]