Skip to content

Sparse dsp opt - #1729

Open
lyang24 wants to merge 1 commit into
zilliztech:sparse_dsp_devfrom
lyang24:sparse_dsp_opt
Open

Sparse dsp opt#1729
lyang24 wants to merge 1 commit into
zilliztech:sparse_dsp_devfrom
lyang24:sparse_dsp_opt

Conversation

@lyang24

@lyang24 lyang24 commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

issue: #1728

@sre-ci-robot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lyang24
To complete the pull request process, please assign chasingegg after the PR has been reviewed.
You can assign the PR to them by writing /assign @chasingegg in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify

mergify Bot commented Jul 19, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@alexanderguzhva

Copy link
Copy Markdown
Collaborator

@lyang24 Hi, is this PR ready to be reviewed? Thanks

@lyang24

lyang24 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@lyang24 Hi, is this PR ready to be reviewed? Thanks

yes should be good to review

@alexanderguzhva

Copy link
Copy Markdown
Collaborator

@lyang24 please rebase to master branch, because it moved to conan v2

Comment thread include/knowhere/sparse_utils.h Outdated
Comment thread include/knowhere/sparse_utils.h Outdated
Comment thread include/knowhere/sparse_utils.h Outdated
Comment thread src/index/sparse/sparse_dsp_index.h Outdated
@alexanderguzhva

Copy link
Copy Markdown
Collaborator

@sparknack Could you please help reviewing this as well? Thanks!

@lyang24

lyang24 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

please rebase to master branch, because it moved to conan v2

can i get sparse_dev_dev branch rebased to master by any chance

@chasingegg

chasingegg commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

@lyang24 I've rebased sparse_dsp_dev to main, you could check

@chasingegg
chasingegg force-pushed the sparse_dsp_dev branch 2 times, most recently from 49ef4e5 to 810efce Compare July 24, 2026 10:36
@lyang24
lyang24 force-pushed the sparse_dsp_opt branch 3 times, most recently from 756fb1a to 1bd65c2 Compare July 25, 2026 05:56
Improve DSP search correctness and performance with rank-safe threshold handling, superblock-major bound accumulation, candidate masks, conditional ASC computation, and hybrid SIMD intersections.

Flatten and parallelize DSP metadata construction, persist native metadata, bound search workspace reuse, extend instrumentation, and add correctness, serialization, concurrency, and SIMD coverage.

Signed-off-by: lyang24 <lanqingy93@gmail.com>
@alexanderguzhva

alexanderguzhva commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

@lyang24

2026-07-25T07:02:40.0946960Z /home/runner/work/knowhere/knowhere/src/index/sparse/sparse_index_node.cc:1179:25: error: implicit capture of 'this' with a capture default of '=' is deprecated [clang-diagnostic-deprecated-this-capture,-warnings-as-errors]
2026-07-25T07:02:40.0949212Z  1179 |                         index_->GetAllDistances(queries[i], drop_ratio_search, bitset, computer);
2026-07-25T07:02:40.0949773Z       |                         ^
2026-07-25T07:02:40.0950546Z /home/runner/work/knowhere/knowhere/src/index/sparse/sparse_index_node.cc:1176:43: note: add an explicit capture of 'this' to capture '*this' by reference
2026-07-25T07:02:40.0951740Z  1176 |                 auto compute_dist_func = [=]() -> std::vector<DistId> {
2026-07-25T07:02:40.0952118Z       |                                           ^
2026-07-25T07:02:40.0952405Z       |                                            , this
2026-07-25T07:02:40.0953199Z /home/runner/work/knowhere/knowhere/src/index/sparse/sparse_index_node.cc:1184:56: error: C-style casts are discouraged; use static_cast [modernize-avoid-c-style-cast,-warnings-as-errors]
2026-07-25T07:02:40.0954066Z  1184 |                             distances_ids.emplace_back((int64_t)i, distances[i]);
2026-07-25T07:02:40.0954468Z       |                                                        ^~~~~~~~~ 
2026-07-25T07:02:40.0954820Z       |                                                        static_cast<int64_t>( )
2026-07-25T07:02:40.0955287Z 
2026-07-25T07:02:40.0960641Z [34/56][5.2s] clang-tidy-22 -p=./build/Release -quiet /home/runner/work/knowhere/knowhere/src/simd/sparse_simd_avx512.cc
2026-07-25T07:02:40.0961631Z /home/runner/work/knowhere/knowhere/src/simd/sparse_simd_avx512.cc:49:63: error: C-style casts are discouraged; use static_cast [modernize-avoid-c-style-cast,-warnings-as-errors]
2026-07-25T07:02:40.0962499Z    49 |                 uint32_t hi = std::min<uint64_t>(full_chunks, uint64_t(base) + step + 1);
2026-07-25T07:02:40.0962923Z       |                                                               ^~~~~~~~
2026-07-25T07:02:40.0963281Z       |                                                               static_cast<uint64_t>
2026-07-25T07:02:40.0963524Z 

@alexanderguzhva

Copy link
Copy Markdown
Collaborator

superseded by #1763

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants