Add recall for test script - #190
Conversation
Signed-off-by: Samuel Herman <sherman8915@gmail.com>
Signed-off-by: Samuel Herman <sherman8915@gmail.com>
Signed-off-by: Samuel Herman <sherman8915@gmail.com>
6d1cc07 to
d540a1c
Compare
There was a problem hiding this comment.
pardon my python expertise, but what is the purpose of this file? I only see some comments (which can be in the readme), and version number.
There was a problem hiding this comment.
Not much of a python expert either, I think if you want the subdirectory to be picked up as a module you have to create the init file. But I can try removing it and see what happens.
|
|
||
| ``` | ||
| scripts/ | ||
| ├── create_and_test_large_index.py # Main testing script |
There was a problem hiding this comment.
maybe move all these files under a directory? /scripts/JvectorIndexAndSearch
I'll be adding more scripts here shortly, including Jake's Parquet loader so it'd make sense to have separate directories under scripts.
| import matplotlib.pyplot as plt | ||
| import os | ||
| """ | ||
| Create and test large JVector indices in OpenSearch. |
There was a problem hiding this comment.
consider renaming this file to reflect the scope.
It's doing more that testing large index at this point :)
| dist = 1 - cosine_sim | ||
| else: | ||
| dist = 1.0 | ||
| else: |
There was a problem hiding this comment.
inner product is not supported?
There was a problem hiding this comment.
Didn't add that yet because it requires normalization, which we're not doing.
akash-shankaran
left a comment
akash-shankaran
left a comment
There was a problem hiding this comment.
Nice change.
Maybe we should consider running this (at a small scale) as part of PR validations, until our other benchmarking solutions become robust.
Good idea! |
Signed-off-by: Samuel Herman <sherman8915@gmail.com>
d540a1c to
65e7036
Compare
* add dynamic recall measurement * refactor for better maintainability, add tests --------- Signed-off-by: Samuel Herman <sherman8915@gmail.com> (cherry picked from commit e6fd81d)
* add dynamic recall measurement * refactor for better maintainability, add tests --------- (cherry picked from commit e6fd81d) Signed-off-by: Samuel Herman <sherman8915@gmail.com> Co-authored-by: sam-herman <97131656+sam-herman@users.noreply.github.com>
* add dynamic recall measurement * refactor for better maintainability, add tests --------- Signed-off-by: Samuel Herman <sherman8915@gmail.com>
3 participants
Description
This PR achieves a number of main changes in the
scripts/create_and_test_large_index.pywhich is currently used as an ad-hoc dynamic performance benchmarking tool.Main changes:
scripts/create_and_test_large_index.py- into several modules (see documentation for the new package).Testing
Per the instructions in the readme file:
(.venv) sam.herman@Sam-Herman-FP9X7HJYJX scripts % python test_recall_integration.py \ --host localhost:9200 \ --dimension 256 \ --num-vectors 5000 \ --num-queries 10 \ --k 20 ... ... === Recall Summary === Average Recall@20: 0.8350 Min Recall@20: 0.7000 Max Recall@20: 1.0000 Std Dev: 0.0776 [Cleanup] Deleting test index... ====================================================================== ✅ INTEGRATION TEST COMPLETED SUCCESSFULLY ======================================================================Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.