Implement Hamming distance for binary strings #124
Conversation
some logicBrain dumping here so the PR has a bit of context :) why a new codec?The In the bits = scalar.to_bits();
bits < 0x8000_0000 && bits > 0x0000_0000 I benchmarked a few approaches and this one was faster than the naive check by a nontrivial amount (~550 ps vs ~750 ps on my old macbook). Might be overkill but hey 🤠 how are splits created with hamming distanceHere I took inspiration from how it was done in annoy. Rather than building a locality-sensitive hash using separating hyperplanes as in the other distance impls, bit sampling can accomplish this. The normal vector that comes out of the To make sure the same index isn't selected at various depths in a given tree we require the normal to be a valid splitting index; i.e. there's at least one element on where this deviates with the source implementation
interpretation of the marginNot geometric like with other distances, we just need to know if a particular bit is set for a target vector. To achieve this, hamming-bitwise-fastIn the original issue a great blog post on an efficient implementation of the hamming distance is linked. En gros if you structure the code right the compiler will auto-vectorize things pretty well. |
|
Hey @nnethercott, sorry for the huge delay, I was on holiday for the past weeks, and currently I'm a bit in a rush, I really need to finish #118 before the next Meilisearch release, so I won't be able to give you a proper review until some time sorry 🙈 I quickly skimmed through your code, and it looks pretty good and well-documented. Thanks a lot for that, I think we'll be able to merge it with little change! |
@irevoire no worries, take your time :) |
Using provided implementation randomly selects a side if the margin_no_header is 0. Override side impl for Hamming Using provided implementation randomly selects a side if the margin_no_header is 0.
in annoy it's [defined like](https://github.com/spotify/annoy/blob/main/src/annoylib.h#L718C5-L718C59) ``` distance - (margin != (unsigned int) child_nr); ``` where margin is a bool and child_nr=0 => Side::Left and child_nr=1 => Side::Right in arroy terms. Lets call the second term in the above the "penalty". Reading this we have 4 cases for margin x side; - margin = 0, child_nr = 0 => penalty = 0 - margin = 1, child_nr = 0 => penalty = 1 - margin = 0, child_nr = 1 => penalty = 1 - margin = 1, child_nr = 1 => penalty = 0 so its something like margin XOR child_nr. For side=Left we get penalty = margin, and for side=Right we get penalty = 1-margin.
Co-authored-by: Tamo <[email protected]>
margin_no_header is a footgun here, since signature for margin is the same its easy to confound the two. this is bad since we may build a tree not using the hyperplane bias but then configure the reader to use it (for instance)
different platforms show varying numbers of digits in f32 debug impl => failing tests on linux vs macos runners. to fix this we ensure only 4 digits are printed
performances were terrible before, but changing margin to map to {-1,1}
instead of {0,1} and keeping trait default for side and pq_distance
fixed things
|
Update: I'm keeping this PR as a draft until #132 is merged (hopefully) cause that PR would allow us to store the splitting index as a usize in the node header instead of requiring it be stored as a one-hot vector in the index -- this means we can use way less space to store the splitting node since no vector info is needed ! A branch based off #132 with the hamming distance implemented can be found here. In the meantime here's some results on the vector-store-relevancy-benchmark for 100k vectors and an index of 100 trees (recall for hamming often significantly outperforms all other quantized formats across several datasets): |
|
@irevoire hamming's back on the menu 🍝 SplitPlaneNormals now store a random index in the node header and an empty vec. Turns out hamming is pretty good on recall too (pic below on 100k vectors, didn't update the print statement :p) . Modified vector-store-relevancy-benchmark code here for hamming (used dummy for qdrant since it isn't supported).
Sorry for the 50 commits also lol, i just merged main into this. Can work some git rebase magic later or we just sqash merge this bad boy. |
|
Hey, the parallelization work is almost done, it's just missing some error handling on rayon, and then it'll be ready to review. |
Ayy this is massive ! I tried looking into this a few days back but hit a wall, I'm glad a better mind was able to crack it ahah. Gonna check out your PR this weekend cause I'm insanely curious now
woot woot ! again no rush :) I'll try to tackle #134 in the meantime ! Edit: just seeing this now but is ur pfp a cowboy bebop ref ahah |
I'm so happy we didn't really lose much performance in the process! I didn't think that was possible, ahah
That would be me if my dog were smaller |
see you space cowboy... |
Pull Request
Had some time this weekend and wanted to try to incorporate hamming into arroy !
Check comments in the PR to explain some of the logic
Related issue
Fixes #102
What does this PR do?
Hammingdistance and a new {0,1}-quantizedUnalignedVectorCodecimpl for binary stringsPR checklist
Please check if your PR fulfills the following requirements:
Thank you so much for contributing to Meilisearch!