feat(iterator): add DocIterator for full collection traversal by YongqiYin · Pull Request #597 · alibaba/zvec

YongqiYin · 2026-07-16T12:53:07Z

Add streaming full-collection traversal across C++/C/Python:

C++: Collection::CreateIterator + DocIterator (isolated Flush+snapshot scan, ConcatenatingReader across segments, FilteringReader for deletes, batch-prefetched vectors)
C API: zvec_collection_create_iterator/next/close + iterator options
Python: collection.iter_docs() generator (constant memory)
Tests: C++ (unit/integration/concurrency/perf), C API, Python

Relates to #380

Copilot

Pull request overview

This PR introduces a full-collection document iterator (“DocIterator”) across the C++ core, C API, and Python bindings to enable streaming traversal/export without relying on large topk queries (relates to #380).

Changes:

Add C++ Collection::CreateIterator() and DocIterator with snapshot isolation, segment concatenation, and delete filtering.
Expose the iterator via the C API (zvec_collection_create_iterator/next/close) and Python (Collection.iter_docs() generator).
Add C++/C/Python tests covering basic iteration, delete filtering, field selection, and concurrency/isolation.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tests/db/iterator_test.cc	New C++ unit/integration/concurrency/perf tests for full traversal.
tests/c/c_api_test.c	Adds C API iterator tests and option handling coverage.
src/include/zvec/db/options.h	Introduces `IteratorOptions` (output_fields/include_vector).
src/include/zvec/db/doc_iterator.h	Public C++ `DocIterator` interface.
src/include/zvec/db/collection.h	Adds `Collection::CreateIterator()` API.
src/include/zvec/c_api.h	Adds public C iterator and iterator-options API.
src/db/index/segment/filtering_reader.h	New Arrow reader wrapper to filter deleted docs.
src/db/index/segment/concatenating_reader.h	New Arrow reader to concatenate readers across segments.
src/db/doc_iterator.cc	Implements row-by-row Doc materialization + optional vector prefetch.
src/db/doc_iterator_internal.h	Internal `DocIterator::Impl` definition (lifetime ordering, caches).
src/db/collection.cc	Implements iterator creation, snapshot scan, and reader chain construction.
src/binding/python/model/python_collection.cc	Exposes `_DocIterator` + `Collection.CreateIterator()` to Python.
src/binding/python/include/python_collection.h	Declares `bind_iterator()` hook.
src/binding/c/c_api.cc	Implements iterator options + iterator handles for the C API.
python/zvec/model/collection.py	Adds `Collection.iter_docs()` streaming generator.
python/tests/test_iter_docs.py	New Python tests for iterator behavior and isolation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

YongqiYin · 2026-07-16T13:01:36Z

@copilot resolve the merge conflicts in this pull request

CLAassistant · 2026-07-16T13:13:01Z

All committers have signed the CLA.

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

egolearner · 2026-07-20T11:57:53Z

+    // 3. create new writing segment (collection.cc:1596-1608)
+    // Skip for read-only collections: they cannot flush, and a read-only
+    // collection has no unpersisted writes (the writing segment is empty).
+    if (!options_.read_only_ && writing_segment_->doc_count() != 0) {


这里不能用 doc_count 来判断。只有删除的场景也需要调用 switch？

// Count documents visible to an optional global-doc-ID filter. virtual uint64_t doc_count(const IndexFilter::Ptr filter = nullptr) = 0;

egolearner · 2026-07-20T12:02:54Z

+  std::vector<std::shared_ptr<arrow::RecordBatchReader>> readers;
+  for (const auto &seg : segments) {
+    auto scalar_reader = seg->scan(scan_columns);
+    if (scalar_reader) {


需要做下错误处理。

已补错误日志

错误日志不够，需要返回错误给用户

egolearner · 2026-07-20T12:20:05Z

+              uint64_t min_doc_id = seg->meta()->min_doc_id();
+              for (const auto &field : impl_->schema->vector_fields()) {
+                auto indexer = seg->get_combined_vector_indexer(field->name());
+                if (!indexer) continue;


这里需要做错误处理?

已做错误处理

代码推出来了，再回复吧，要不然没意义

egolearner · 2026-07-20T12:22:27Z

+          break;
+        }
+        case DataType::INT32: {
+          auto a = std::dynamic_pointer_cast<arrow::Int32Array>(array);


a 命名太随意了

另外，这段 Arrow Array -> Doc field 转换和现有 sqlengine::fill_doc_field / SegmentImpl::Fetch 里的标量字段转换重复了。建议把 row 级别的转换抽成内部公共 helper，让 iterator、Fetch 和 sqlengine 共用，否则新增字段类型或调整 null/list 语义时需要多处同步。

egolearner · 2026-07-20T12:32:15Z

+// Same logic as SegmentImpl::ConvertVectorDataBufferToDocField
+// (segment.cc:1024-1093) but as a free function to avoid modifying Segment's
+// interface.
+static Status SetVectorFieldFromBuffer(


复用SegmentImpl::ConvertVectorDataBufferToDocField 的实现吧，可以放到公共类中

Add streaming full-collection traversal across C++/C/Python: - C++: Collection::CreateIterator + DocIterator (isolated Flush+snapshot scan, ConcatenatingReader across segments, FilteringReader for deletes, batch-prefetched vectors) - C API: zvec_collection_create_iterator/next/close + iterator options - Python: collection.iter_docs() generator (constant memory) - Tests: C++ (unit/integration/concurrency/perf), C API, Python Relates to alibaba#380

Copilot AI review requested due to automatic review settings July 16, 2026 12:53

YongqiYin requested review from Cuiyus, chinaux, feihongxu0824 and zhourrr as code owners July 16, 2026 12:53

github-actions Bot assigned YongqiYin Jul 16, 2026

Copilot started reviewing on behalf of YongqiYin July 16, 2026 12:53 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

YongqiYin force-pushed the feat/doc-iterator branch from b596564 to 4aca7cb Compare July 16, 2026 13:01

YongqiYin force-pushed the feat/doc-iterator branch from 4aca7cb to feef1d7 Compare July 16, 2026 16:53

YongqiYin requested a review from Copilot July 17, 2026 03:15

Copilot started reviewing on behalf of YongqiYin July 17, 2026 03:15 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread src/db/collection.cc

Comment thread src/db/doc_iterator.cc

Comment thread src/db/doc_iterator.cc

YongqiYin force-pushed the feat/doc-iterator branch 2 times, most recently from 96273fc to 4f542e0 Compare July 17, 2026 06:04

egolearner requested changes Jul 20, 2026

View reviewed changes

YongqiYin force-pushed the feat/doc-iterator branch from 4f542e0 to 16c0bd2 Compare July 21, 2026 09:15

Merge branch 'main' into feat/doc-iterator

fff802a

Uh oh!

Conversation

YongqiYin commented Jul 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

YongqiYin commented Jul 16, 2026

Uh oh!

CLAassistant commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CLAassistant commented Jul 16, 2026 •

edited

Loading