Support gno-imi partition strategy #612

dasurax · 2025-04-21T08:39:37Z

closed: #562

src/algorithm/ivf_parameter.cpp

src/algorithm/ivf_partition/gno_imi_partition.cpp

jiaweizone · 2025-05-03T17:26:41Z

examples/cpp/108_index_gno_imi.cpp

+    {
+        "dtype": "float32",
+        "metric_type": "l2",
+        "dim": 128,


"dim": 960 ?

examples/cpp/108_index_gno_imi.cpp

src/algorithm/ivf.cpp

src/impl/kmeans_cluster.cpp

examples/cpp/106_index_ivf.cpp

src/algorithm/ivf.cpp

src/algorithm/ivf_partition/gno_imi_partition.cpp

src/algorithm/ivf.cpp

src/algorithm/ivf_partition/ivf_partition_strategy_parameter.h

codecov · 2025-05-19T14:16:54Z

Codecov Report

Attention: Patch coverage is 96.75810% with 13 lines in your changes missing coverage. Please review.

❌ Your project status has failed because the head coverage (89.31%) is below the target coverage (90.00%). You can increase the head coverage or adjust the target coverage.

@@            Coverage Diff             @@
##             main     #612      +/-   ##
==========================================
+ Coverage   88.82%   89.31%   +0.48%     
==========================================
  Files         199      202       +3     
  Lines       12736    13103     +367     
==========================================
+ Hits        11313    11703     +390     
+ Misses       1423     1400      -23

Flag	Coverage Δ
cpp	`89.31% <96.75%> (+0.48%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
common	`92.53% <ø> (ø)`
datacell	`91.88% <ø> (ø)`
index	`89.12% <97.06%> (+0.44%)`	⬆️
simd	`100.00% <ø> (ø)`

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c42d9a...0b46590. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jiaweizone

LGTM

examples/cpp/108_index_gno_imi.cpp

src/algorithm/ivf.cpp

LHT129

LGTM

wxyucs

lgtm

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: jinjiabao.jjb <[email protected]> Signed-off-by: suguan.dx <[email protected]>

…p#720) Signed-off-by: jinjiabao.jjb <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

- add request search object - add attribute for datasets - add new interface on index search Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

- only support hgraph, bruteforce Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

* adapt update with mark delete Signed-off-by: zhongxiaoyao.zxy <[email protected]> Signed-off-by: suguan.dx <[email protected]>

* feat(compressed_graph_datacell): add graph_storage param and test, Signed-off-by: Deming Chu <[email protected]> * feat(hgraph): add compressed graph support and test Signed-off-by: Deming Chu <[email protected]> * style: minor Signed-off-by: Deming Chu <[email protected]> * style: format Signed-off-by: Deming Chu <[email protected]> * style: graph_storage -> graph_storage_type Signed-off-by: Deming Chu <[email protected]> * feat(graph_storage_type): add invalid parameter check and test case Signed-off-by: Deming Chu <[email protected]> * refactor(graph_storage_type): refine invalid parameter check Signed-off-by: Deming Chu <[email protected]> * style(graph_storage_type): remove useless checks in parameters Signed-off-by: Deming Chu <[email protected]> * style(graph_storage_type): "compress" -> "compressed" Signed-off-by: Deming Chu <[email protected]> * refactor(graph_storage_type): max_degree_ moved to graph_interface_parameter.h Signed-off-by: Deming Chu <[email protected]> * refactor(hgraph): simplify max_degree_ get Signed-off-by: Deming Chu <[email protected]> --------- Signed-off-by: Deming Chu <[email protected]> Signed-off-by: suguan.dx <[email protected]>

- add reverse codebook for pq8bit quantization - speedup some compute for pq8bit scan Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: jinjiabao.jjb <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: aozeliu.azl <[email protected]> Co-authored-by: aozeliu.azl <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: jinjiabao.jjb <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: zourunxin.zrx <[email protected]> Co-authored-by: zourunxin.zrx <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: Deming Chu <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: suguan.dx <[email protected]>

…#784) Signed-off-by: Deming Chu <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: Xiangyu Wang <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Signed-off-by: suguan.dx <[email protected]>

dasurax requested review from LHT129, ShawnShawnYou, inabao, jiaweizone and wxyucs as code owners April 21, 2025 08:39

pull-request-size bot added the size/XXL label Apr 21, 2025

dasurax added kind/feature New feature or request version/0.15 labels Apr 21, 2025

dasurax force-pushed the support_gno_ini_for_pr branch 2 times, most recently from b53867b to afd5397 Compare April 22, 2025 07:20

wxyucs assigned LHT129 Apr 28, 2025

jiaweizone reviewed May 3, 2025

View reviewed changes

LHT129 requested changes May 6, 2025

View reviewed changes

src/algorithm/ivf.cpp Outdated Show resolved Hide resolved

src/algorithm/ivf_partition/ivf_partition_strategy_parameter.h Outdated Show resolved Hide resolved

dasurax requested a review from LHT129 May 20, 2025 02:26

jiaweizone approved these changes May 20, 2025

View reviewed changes

examples/cpp/108_index_gno_imi.cpp Show resolved Hide resolved

src/algorithm/ivf.cpp Show resolved Hide resolved

LHT129 approved these changes May 27, 2025

View reviewed changes

wxyucs approved these changes May 28, 2025

View reviewed changes

mergify bot added the module/testing label Jun 3, 2025

ShawnShawnYou approved these changes Jun 9, 2025

View reviewed changes

LHT129 and others added 10 commits June 9, 2025 17:52

speed up ci compile (antgroup#703)

43a8b9c

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

kmeans train for huge K and data count (antgroup#717)

a28b5c7

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

fix double assignment for max_degree in HNSW Merge (antgroup#724)

75ebbaf

Signed-off-by: jinjiabao.jjb <[email protected]> Signed-off-by: suguan.dx <[email protected]>

resolve not properly handling invalid IDs in CalDistanceById (antgrou…

bb27dee

…p#720) Signed-off-by: jinjiabao.jjb <[email protected]> Signed-off-by: suguan.dx <[email protected]>

remove mtime change on ci test (antgroup#736)

04ef4ab

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

add simd operator to cal residual (antgroup#733)

a87e05c

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

add computable operators for bitset (antgroup#721)

779ac49

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

add new interface for attr search (antgroup#729)

05ce90d

- add request search object - add attribute for datasets - add new interface on index search Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

add fast index create for inner index (antgroup#737)

7fee097

- only support hgraph, bruteforce Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

add option for hgraph to build by base quantization (antgroup#735)

93b7ff8

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

LHT129 and others added 24 commits June 9, 2025 17:52

add logger for kmeans train (antgroup#757)

9dc7ef2

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

support gno-imi partition

11207ba

Signed-off-by: suguan.dx <[email protected]>

rebase main

4895841

Signed-off-by: suguan.dx <[email protected]>

support gno-imi partition

e31dd99

Signed-off-by: suguan.dx <[email protected]>

support gno-imi partition

7c02b73

Signed-off-by: suguan.dx <[email protected]>

add bit operator simd implement (antgroup#756)

50949d6

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

adapt update with mark delete (antgroup#762)

b74ba9c

* adapt update with mark delete Signed-off-by: zhongxiaoyao.zxy <[email protected]> Signed-off-by: suguan.dx <[email protected]>

Support arthimetic operator for fp32 simd (antgroup#761)

d07b324

- add reverse codebook for pq8bit quantization - speedup some compute for pq8bit scan Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

support build ivf index with residual (antgroup#754)

8e190dd

Signed-off-by: jinjiabao.jjb <[email protected]> Signed-off-by: suguan.dx <[email protected]>

support estimate memory in hnsw (antgroup#709)

9b7063f

Signed-off-by: jinjiabao.jjb <[email protected]> Signed-off-by: suguan.dx <[email protected]>

reduce test log level to avoid too much debug logs (antgroup#771)

0469640

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

speed up fast_bitset by use simd operators (antgroup#764)

1332312

Signed-off-by: LHT129 <[email protected]> Signed-off-by: suguan.dx <[email protected]>

fix wrong id when buckets_per_data > 1

daefc10

Signed-off-by: suguan.dx <[email protected]>

add gnoimi param tojson ut

c57317f

Signed-off-by: suguan.dx <[email protected]>

add gnoimi build ut

6df9cbd

Signed-off-by: suguan.dx <[email protected]>

compile openblas dynamic arch (antgroup#769)

c8e99c2

Signed-off-by: aozeliu.azl <[email protected]> Co-authored-by: aozeliu.azl <[email protected]> Signed-off-by: suguan.dx <[email protected]>

support remove in graph datacell (antgroup#695)

07fc182

Signed-off-by: jinjiabao.jjb <[email protected]> Signed-off-by: suguan.dx <[email protected]>

add search allocator (antgroup#716)

61b4a0f

Signed-off-by: zourunxin.zrx <[email protected]> Co-authored-by: zourunxin.zrx <[email protected]> Signed-off-by: suguan.dx <[email protected]>

print memory usage detail in eval_performance for HGraph (antgroup#770)

e2d8263

Signed-off-by: Deming Chu <[email protected]> Signed-off-by: suguan.dx <[email protected]>

add ut IVF Build With Large K

fc6bd3e

Signed-off-by: suguan.dx <[email protected]>

fix(hgraph): issues of segment fault and recall degradation (antgroup…

391264c

…#784) Signed-off-by: Deming Chu <[email protected]> Signed-off-by: suguan.dx <[email protected]>

add spdlog to the deps of mockimpl (antgroup#775)

7a291bc

Signed-off-by: Xiangyu Wang <[email protected]> Signed-off-by: suguan.dx <[email protected]>

optimize header files including (antgroup#787)

05407d0

Signed-off-by: Xiangyu Wang <[email protected]> Signed-off-by: suguan.dx <[email protected]>

dasurax force-pushed the support_gno_ini_for_pr branch from d49f19d to 05407d0 Compare June 9, 2025 10:18

mergify bot added module/simd module/tools labels Jun 9, 2025

suguan.dx added 2 commits June 9, 2025 18:25

merge

6037138

Signed-off-by: suguan.dx <[email protected]>

fix ut

0b46590

Signed-off-by: suguan.dx <[email protected]>

wxyucs merged commit 0475fd0 into antgroup:main Jun 10, 2025
20 of 21 checks passed

Support gno-imi partition strategy #612

Support gno-imi partition strategy #612

Uh oh!

Conversation

dasurax commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jiaweizone May 3, 2025

Choose a reason for hiding this comment

Uh oh!

dasurax May 9, 2025

Choose a reason for hiding this comment

Uh oh!

jiaweizone May 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jiaweizone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

LHT129 left a comment

Choose a reason for hiding this comment

Uh oh!

wxyucs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

dasurax commented Apr 21, 2025 •

edited

Loading

codecov bot commented May 19, 2025 •

edited

Loading