feat: add Redis Vector Set support - #356
Open
happy-v587 wants to merge 33 commits into
Open
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds standalone Redis Vector Set support across storage, command parsing, RESP2/RESP3 encoding, lifecycle handling, cluster routing, and Rust/Python tests. It introduces vector codecs, atomic mutations, point reads, FLAT similarity search, seven commands, and a Phase 1 implementation plan. ChangesRedis Vector Set
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
happy-v587
marked this pull request as ready for review
July 20, 2026 04:12
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
src/resp/src/encode.rs (1)
232-281: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winReduce duplication: delegate to existing
append_*helpers.These arms re-implement
append_null,append_boolean,append_double,append_big_number,append_bulk_error, andappend_verbatim_stringbyte-for-byte. Two copies of the same wire formatting (notably theDoubleNaN/inf handling and theVerbatimStringlength logic) can silently diverge on future edits. Consider calling the helpers directly.♻️ Example delegation
- RespData::Null => { - self.buffer.extend_from_slice(b"_"); - self.append_crlf() - } - RespData::Boolean(value) => { - self.buffer.extend_from_slice(b"#"); - self.buffer - .extend_from_slice(if *value { b"t" } else { b"f" }); - self.append_crlf() - } - RespData::Double(value) => { - if value.is_nan() { - self.buffer.extend_from_slice(b",nan"); - } else if value.is_infinite() { - self.buffer.extend_from_slice(if value.is_sign_negative() { - b",-inf" - } else { - b",inf" - }); - } else { - let _ = write!(self.buffer, ",{value}"); - } - self.append_crlf() - } + RespData::Null => self.append_null(), + RespData::Boolean(value) => self.append_boolean(*value), + RespData::Double(value) => self.append_double(*value),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/resp/src/encode.rs` around lines 232 - 281, Update the RespData encoding match arms for Null, Boolean, Double, BigNumber, BulkError, and VerbatimString to delegate directly to their existing append_null, append_boolean, append_double, append_big_number, append_bulk_error, and append_verbatim_string helpers. Remove the duplicated byte-formatting and validation logic while preserving each helper’s current output.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/storage/src/meta_compaction_filter.rs`:
- Around line 364-381: Add #[allow(clippy::unwrap_used)] to the
test_vectorset_meta_value_expired function so its existing encoded_key unwrap is
explicitly permitted under the project’s Clippy settings.
In `@src/storage/tests/redis_vector_test.rs`:
- Around line 378-384: Replace direct tempfile::tempdir() usage in
test_storage_routes_all_members_of_one_vectorset_to_one_instance,
test_expired_vectorset_reads_as_missing, and
test_vector_storage_rejects_cluster_mode with unique_test_db_path(); call
safe_cleanup_test_db(&path) before opening storage, pass &path to open(), and
call safe_cleanup_test_db(&path) at each test’s end. Apply these changes at
src/storage/tests/redis_vector_test.rs:378-384, 446-452, and 506-512.
---
Nitpick comments:
In `@src/resp/src/encode.rs`:
- Around line 232-281: Update the RespData encoding match arms for Null,
Boolean, Double, BigNumber, BulkError, and VerbatimString to delegate directly
to their existing append_null, append_boolean, append_double, append_big_number,
append_bulk_error, and append_verbatim_string helpers. Remove the duplicated
byte-formatting and validation logic while preserving each helper’s current
output.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 4b9c8226-57d2-4f25-8bf0-c96a5a0392c3
📒 Files selected for processing (25)
docs/superpowers/plans/2026-07-19-redis-vector-set.mdsrc/cmd/src/lib.rssrc/cmd/src/table.rssrc/cmd/src/vector.rssrc/conf/src/raft_type.rssrc/raft/src/lib.rssrc/resp/src/encode.rssrc/resp/src/negotiation.rssrc/resp/tests/integration_tests.rssrc/storage/src/batch.rssrc/storage/src/data_compaction_filter.rssrc/storage/src/format_base_value.rssrc/storage/src/format_vector.rssrc/storage/src/lib.rssrc/storage/src/logindex/types.rssrc/storage/src/meta_compaction_filter.rssrc/storage/src/redis.rssrc/storage/src/redis_strings.rssrc/storage/src/redis_vectors.rssrc/storage/src/storage.rssrc/storage/src/storage_impl.rssrc/storage/src/vector.rssrc/storage/tests/redis_basic_test.rssrc/storage/tests/redis_vector_test.rstests/python/test_vector_set_commands.py
16 tasks
happy-v587
force-pushed
the
feat/redis-vector
branch
from
July 25, 2026 12:59
cb68d38 to
049f9c0
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/superpowers/plans/2026-07-19-redis-vector-set.md`:
- Line 423: The VectorSet recreation flow must guarantee collision-free
generations instead of relying solely on timestamps. Update the VectorMeta
generation logic around VectorMeta::new so each expired-and-recreated VectorSet
receives a monotonic or otherwise unique version, and ensure stale VectorDataCF
rows cannot be addressed by the new generation; alternatively, add a rapid
expire/recreate test that proves timestamp-based generations cannot collide.
- Line 467: Update the documented test command to avoid passing two explicit
filters to cargo test. Run the vector and format_vector test filters as separate
commands, or replace them with one broader valid filter while preserving
coverage of both test groups.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: edc0f854-de79-48e2-af78-de6e26e6b763
📒 Files selected for processing (25)
docs/superpowers/plans/2026-07-19-redis-vector-set.mdsrc/cmd/src/lib.rssrc/cmd/src/table.rssrc/cmd/src/vector.rssrc/conf/src/raft_type.rssrc/raft/src/lib.rssrc/resp/src/encode.rssrc/resp/src/negotiation.rssrc/resp/tests/integration_tests.rssrc/storage/src/batch.rssrc/storage/src/data_compaction_filter.rssrc/storage/src/format_base_value.rssrc/storage/src/format_vector.rssrc/storage/src/lib.rssrc/storage/src/logindex/types.rssrc/storage/src/meta_compaction_filter.rssrc/storage/src/redis.rssrc/storage/src/redis_strings.rssrc/storage/src/redis_vectors.rssrc/storage/src/storage.rssrc/storage/src/storage_impl.rssrc/storage/src/vector.rssrc/storage/tests/redis_basic_test.rssrc/storage/tests/redis_vector_test.rstests/python/test_vector_set_commands.py
🚧 Files skipped from review as they are similar to previous changes (23)
- src/cmd/src/lib.rs
- src/conf/src/raft_type.rs
- src/cmd/src/table.rs
- src/storage/src/storage.rs
- src/storage/src/meta_compaction_filter.rs
- src/raft/src/lib.rs
- src/storage/src/batch.rs
- src/storage/src/logindex/types.rs
- src/storage/src/format_base_value.rs
- src/storage/src/lib.rs
- src/resp/src/negotiation.rs
- src/storage/src/redis_strings.rs
- src/storage/src/data_compaction_filter.rs
- src/storage/tests/redis_basic_test.rs
- tests/python/test_vector_set_commands.py
- src/cmd/src/vector.rs
- src/storage/src/redis.rs
- src/storage/src/storage_impl.rs
- src/resp/src/encode.rs
- src/storage/src/format_vector.rs
- src/storage/src/vector.rs
- src/storage/src/redis_vectors.rs
- src/storage/tests/redis_vector_test.rs
…ions - Remove MODULE_NO_CLUSTER guard so vector commands work in Raft Group clusters (binlog replication); keep follower redirect for writes. - Split src/cmd/src/vector.rs into one module per command. - Introduce SimilarityMetric (currently Cosine) persisted in VectorMeta. - Introduce VectorSearchEngine (currently Flat) for future HNSW. - Add RocksDB encoding layout comments for vector meta/data values. - Update the Redis Vector Set superpower plan to reflect cluster support. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
src/storage/src/vector.rs (1)
182-204: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winNon-obvious inverted
Ord— consider a clarifying comment.
Ord::cmpreverses score comparison (other.score.total_cmp(&self.score)) to makeBinaryHeap::peek()surface the worst candidate, but the tie-break (self.element.cmp(&other.element)) is not reversed. This is correct (verified againstflat_search's eviction logic and the finalsort_byat lines 257-262, which are consistent), but the asymmetry is easy to break during future edits. A short comment explaining "reversed for min-heap-of-top-k semantics" would help future maintainers avoid introducing a sign-flip bug.📝 Suggested comment
impl Ord for ScoredCandidate { + // Reversed on `score` so `BinaryHeap` (a max-heap) surfaces the *worst* + // candidate via `peek()`, enabling top-`count` retention via eviction. + // The element tie-break is intentionally NOT reversed. fn cmp(&self, other: &Self) -> Ordering {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/storage/src/vector.rs` around lines 182 - 204, Add a concise explanatory comment in ScoredCandidate::cmp documenting that the score ordering is intentionally reversed so BinaryHeap::peek() exposes the worst top-k candidate, while the element tie-break remains in its existing direction. Do not alter the comparison behavior.src/storage/src/format_vector.rs (1)
134-201: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMissing regression test for unsupported-metric rejection.
SimilarityMetric::from_u8(Line 179) is new validation logic replacing the old hardcoded cosine check, butvector_codecs_reject_malformed_bytesonly mutates theformatbyte (offset 17), not themetricbyte (offset 19). Add a case that sets an unsupported metric byte and assertsVectorMeta::decodeerrors, to directly cover the newfrom_u8rejection path.✅ Suggested additional test case
let mut bad_meta_format = encoded_meta; bad_meta_format[17] = 0; assert!(VectorMeta::decode(&bad_meta_format).is_err()); + + let mut bad_metric = bad_meta_format; + bad_metric[17] = VECTOR_META_FORMAT; + bad_metric[19] = 0xFF; + assert!(VectorMeta::decode(&bad_metric).is_err()); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/storage/src/format_vector.rs` around lines 134 - 201, Add a regression case to the vector metadata malformed-bytes test, `vector_codecs_reject_malformed_bytes`, that mutates the metric byte at offset 19 to an unsupported value and asserts `VectorMeta::decode` returns an error. Keep the existing format-byte case and target the `SimilarityMetric::from_u8` validation path.src/storage/tests/redis_vector_test.rs (1)
458-467: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDecode
VectorMetabefore mutating its stale TTL fields.The current test slices
VectorMetabytes with hard-coded index math, even thoughVectorMeta::encode/decodedefine the field layout. Decode the CF value, mutate onlyversion/is_deleted-related TTL logic via accessor methods, then re-encode before storing; this keeps the test aligned with the meta-value format and avoids magic offsets.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/storage/tests/redis_vector_test.rs` around lines 458 - 467, Update the test setup around the retrieved meta value to decode it with VectorMeta::decode, mutate the stale TTL-related fields through the type’s accessor methods rather than hard-coded byte offsets, then re-encode the VectorMeta before storing it with put_cf. Preserve the intended previous generation and expired TTL values while keeping the test aligned with the encode/decode format.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/superpowers/plans/2026-07-19-redis-vector-set.md`:
- Around line 44-51: Update Task 6 to reference the vector module layout under
src/cmd/src/vector/, using vector/mod.rs for shared parsing, helpers, and
registration tests plus separate modules for each command; remove or replace the
conflicting instruction to create src/cmd/src/vector.rs. Apply the same path
correction to the additionally affected section.
---
Nitpick comments:
In `@src/storage/src/format_vector.rs`:
- Around line 134-201: Add a regression case to the vector metadata
malformed-bytes test, `vector_codecs_reject_malformed_bytes`, that mutates the
metric byte at offset 19 to an unsupported value and asserts
`VectorMeta::decode` returns an error. Keep the existing format-byte case and
target the `SimilarityMetric::from_u8` validation path.
In `@src/storage/src/vector.rs`:
- Around line 182-204: Add a concise explanatory comment in ScoredCandidate::cmp
documenting that the score ordering is intentionally reversed so
BinaryHeap::peek() exposes the worst top-k candidate, while the element
tie-break remains in its existing direction. Do not alter the comparison
behavior.
In `@src/storage/tests/redis_vector_test.rs`:
- Around line 458-467: Update the test setup around the retrieved meta value to
decode it with VectorMeta::decode, mutate the stale TTL-related fields through
the type’s accessor methods rather than hard-coded byte offsets, then re-encode
the VectorMeta before storing it with put_cf. Preserve the intended previous
generation and expired TTL values while keeping the test aligned with the
encode/decode format.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 9c1b7e87-d210-4d07-b742-02e28caf3c58
📒 Files selected for processing (20)
docs/superpowers/plans/2026-07-19-redis-vector-set.mdsrc/cmd/src/lib.rssrc/cmd/src/vector/mod.rssrc/cmd/src/vector/vadd.rssrc/cmd/src/vector/vcard.rssrc/cmd/src/vector/vdim.rssrc/cmd/src/vector/vemb.rssrc/cmd/src/vector/vismember.rssrc/cmd/src/vector/vrem.rssrc/cmd/src/vector/vsim.rssrc/net/src/executor_ext.rssrc/net/tests/storage_command_e2e_tests.rssrc/resp/src/encode.rssrc/resp/tests/resp2_encoding.rssrc/storage/src/format_member_data_key.rssrc/storage/src/format_vector.rssrc/storage/src/lib.rssrc/storage/src/redis_vectors.rssrc/storage/src/vector.rssrc/storage/tests/redis_vector_test.rs
💤 Files with no reviewable changes (1)
- src/cmd/src/lib.rs
🚧 Files skipped from review as they are similar to previous changes (1)
- src/storage/src/lib.rs
Restore the flag constant so the bit remains reserved, even though vector commands no longer use it. Co-Authored-By: Claude <noreply@anthropic.com>
…-db#356 - Document reversed score ordering in HeapHit::cmp. - Add regression test for unsupported vector metric byte. - Expose VectorMeta accessors and use decode/encode in tests. - Delegate RESP encoding to version-aware append_* helpers. Co-Authored-By: Claude <noreply@anthropic.com>
Keep remote VectorSearchEngine / SimilarityMetric additions and resolve the outstanding CodeRabbit review fixes: - format_vector.rs: keep SimilarityMetric + metric field, keep encode/decode public for integration tests, preserve metric-byte regression test. - redis_vectors.rs: accept remote removal of HeapHit; search is now delegated to VectorSearchEngine. - vector.rs: add BinaryHeap top-k retention comment to ScoredCandidate::cmp. Co-Authored-By: Claude <noreply@anthropic.com>
- merge the private encode_resp_data_inner into the RespEncode trait method; version handling stays inline at the encoding points - translate the vector-set plan doc to Chinese and update Step 3 to match the merged encoder design - move vector value layout comments next to their struct definitions
# Conflicts: # src/net/src/executor_ext.rs
- CanonicalVector now holds quantized data (VectorData: Fp32/Binary/Int8) with to_quantized() as the single conversion entry point - score() dispatches per quantization: hamming similarity for BIN, cosine on the dequantized FP32 form otherwise - VectorDataValue format v3: quant byte, reserved flags byte (for future SETATTR attributes), per-quantization payload layouts - VectorMeta carries the set-level quantization; vadd/vsim convert members and queries to the set's quantization - unify vector meta reads via decode_vector_meta/read_vector_meta_opt
- move parse_vadd/parse_vsim/parse_vemb and their tests from vector/mod.rs into the respective command modules; mod.rs keeps only the shared helpers - parse_vadd becomes a keyword option loop matching the Redis syntax: NOQUANT/Q8/BIN are wired to quantization (default NOQUANT), while CAS/EF/SETATTR/M/REDUCE are rejected with dedicated errors - add Redis syntax comments to vector command arity definitions
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
AlexStocks
reviewed
Aug 1, 2026
AlexStocks
reviewed
Aug 1, 2026
AlexStocks
reviewed
Aug 1, 2026
AlexStocks
reviewed
Aug 1, 2026
AlexStocks
reviewed
Aug 1, 2026
# Conflicts: # src/resp/src/encode.rs # src/storage/src/redis_strings.rs
Collaborator
Author
PR 改动总结这个 PR 是一次完整的 Redis Vector Set 第一阶段实现，不是单纯增加几个命令，因此改动范围较大：共涉及 75 个文件，约 +10,590 / -290 行。 主要改动包括：
改动行数主要集中在 VectorSet 存储格式、命令实现、生命周期/compaction 处理，以及约 3,000 行以上的新增测试代码；整体是跨命令层、协议层、存储层和集群层的完整数据类型落地。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
变更说明
基于 Discussion #331 为 Kiwi 增加 Redis Vector Set 的 standalone Phase 1 实现。
本次实现包括：
VectorSet数据类型和独立的vector_data_cf，持久化 FP32 向量及其元数据。VADD、VSIM、VREM、VCARD、VDIM、VEMB、VISMEMBER七个命令。实现边界
NOQUANT和 cosine；尚未实现 Q8、BIN、HNSW/IVF、VINFO、INFO VECTOR及VEMB RAW。VSIM和VSIM ... TRUTH当前均使用同一套精确 FLAT 搜索。用户影响
用户可以在 standalone Kiwi 上持久化、查询、删除 Vector Set 成员，并通过 Redis RESP2 或 RESP3 客户端执行精确相似度搜索。
验证
make fmtmake lintmake buildmake testKIWI_PORT=7389 pytest -q tests/python/test_vector_set_commands.py（14 passed）Summary by CodeRabbit
New Features
VADD,VREM,VCARD,VDIM,VEMB,VISMEMBER, andVSIM.WITHSCORES.Bug Fixes
Tests