fix: external collection query_iterator fails with fieldID(1) not found#50386
fix: external collection query_iterator fails with fieldID(1) not found#50386xiaocai2333 wants to merge 1 commit into
Conversation
The proxy appends the reserved system Timestamp field (fieldID=1) to OutputFieldsId for the iterator MVCC dedup. On the QueryNode side, FillRetrieveResultIfEmpty is invoked when a query produces a fully empty result and iterates OutputFieldsId to synthesize empty columns, resolving each field id through the collection SchemaHelper. System fields are never part of the user/collection schema, so for external collections (whose schema has no system fields and whose segcore synthesizes the timestamp as an Int64 column) the lookup fails with 'fieldID(1) not found'. This surfaces when a query_iterator paginates past the last matching row and the next batch matches zero rows, and reproduces deterministically when the iterator is rebuilt from the last cursor. Fix: special-case reserved system fields (RowID/Timestamp) in FillRetrieveResultIfEmpty and emit them as empty Int64 columns, matching the non-empty retrieve path, instead of resolving them through the schema. The proxy already strips system fields via filterSystemFields, so the user-facing result is unchanged. issue: milvus-io#50188 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: cai.zhang <cai.zhang@zilliz.com> (cherry picked from commit 38d1c1c)
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: xiaocai2333 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
mergify
Bot
added
dco-passed
|
[INFO] PR Label Summary by Default
[WARNING] Milestone not set
You can set milestone by commenting: Use /refresh-label to update related check and label manually |
|
[ci-v2-notice] To rerun ci-v2 checks, comment with:
If you have any questions or requests, please contact @zhikunyao. |
✅ CI Loop Results
|
| Stage | Result | Duration | Tests |
|---|---|---|---|
| ✅ Build | SUCCESS | 24.6min | - |
| ✅ Code-Check | SUCCESS | 12.7min | - |
| ✅ UT-GO | SUCCESS | 23.2min | 1012 passed |
| ✅ UT-Integration | SUCCESS | 21.7min | 46 passed |
| ✅ UT-CPP-Cov | SUCCESS | 79.6min | 7625 passed |
Total: 111min | Pipeline | Artifacts
Overall Coverage: 69.3%
Diff Coverage: CPP 85.7% (2515 hit, 419 miss, 2934 measurable lines, 5416 unmeasured) | Go 100.0% (24 hit, 0 miss, 24 measurable lines, 10 unmeasured)
Diff Coverage HTML: view changed lines
Total Patch Coverage: 85.8% (2539/2958 measurable lines, 5426 unmeasured)
2 participants
Cherry-pick
pr: #50203
Cherry-pick of #50203 to the 3.0 branch.
Issue
issue: #50188
Problem
For external collections,
query_iteratorreturns hundreds of thousands of rows and then fails withMilvusException: (code=65535, message=fieldID(1) not found). Rebuilding the iterator from the last primary-key cursor reproduces the error immediately.Root Cause
The proxy unconditionally appends the reserved system Timestamp field (
common.TimeStampField, fieldID=1) toOutputFieldsIdfor the iterator MVCC dedup. On the QueryNode side,FillRetrieveResultIfEmptyis invoked when a query produces a fully empty result and resolves eachOutputFieldsIdentry through the collectionSchemaHelper. System fields are never part of the external-collection schema (whose segcore synthesizes the timestamp as a constant Int64 column viaSynthesizeExternalSystemFields), soGetFieldFromID(1)fails withfieldID(1) not found. Internal collections do not hit this because segcore returns empty-but-present columns for zero-match queries.Fix
Special-case reserved system fields (RowID / Timestamp) in
FillRetrieveResultIfEmptyand emit them as empty Int64 columns — matching the non-empty retrieve path — instead of resolving them through the schema. The proxy already strips system fields viafilterSystemFields, so the user-facing result is unchanged.Test
go test -tags dynamic,test -gcflags="all=-N -l" ./internal/util/typeutil/ -run TestFillIfEmptypasses, including the added external-collection and RowID+Timestamp cases.🤖 Generated with Claude Code