Reject ID selectors when IndexShards shifts shard IDs - #5582
Conversation
|
Hi @rioyu123! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
Related to #5581.
Summary
This PR proposes a narrow fail-fast boundary for selector searches when
IndexShardsapplies nonzero offsets to shard IDs. It prevents the incorrectfiltered results in #5581 without introducing a search-parameter cloning API
or mutating caller-owned parameters across worker threads.
The check runs before any shard search. Searches without a selector,
successive_ids=false, and configurations where all shard offsets are zeroretain their existing behavior. A nonzero offset also triggers the check when
that shard is currently empty; the boundary is based on the configured ID
offsets. The public
searchdocumentation explainsthe limitation and the need for globally assigned shard IDs when using
successive_ids=falseto preserve a global ID space.Design question
This is a defensive error, not full support for translating global-ID
selectors into each shard's local ID space. This implementation is ready for review; feedback on the preferred approach
in #5581 is welcome. If per-shard selector
adaptation is preferred, that needs a separate mechanism that preserves
derived search parameters without sharing mutable state between workers.
This change covers the
IndexShardsTemplate::searchimplementation used byIndexShardsandIndexBinaryShards. It does not change the separateIndexShardsIVF::searchoverride.Validation
2ed4c106e9fb9686e7727e5daf8ad6ad1e164109with GCC 13.3 / C++20, genericCPU code and BLAS/LAPACK.
test_threaded_index.cppagainst the unmodified library:2 failed, 6 passed. Both new rejection tests fail because no exception
is raised.
threaded searches, both floating-point failure cases, binary shards,
unchanged output buffers and parameters, no sub-search before rejection,
and the no-selector / no-translation controls.
test_threaded_index.cpp,test_binary_flat.cppandtest_params_override.cpp: 28 passed.clang-format 21.1.8 --dry-run --Werroron the three changed files andgit diff --checkpassed.Tests were compiled directly with the repository-pinned GoogleTest against
the source-built library. Full repository CI, Python bindings, GPU builds
and other platforms were not run locally.