[Bug]: A doc without a vector reads back a fake all-zero vector — missing values are indistinguishable from genuine zeros

[YongqiYin]

Description

Vector fields can be declared nullable through the C-family APIs (zvec_field_schema_create(name, data_type, nullable, dimension); C++ FieldSchema(name, VECTOR_*, dimension, nullable, index_params)). The Python SDK hardcodes nullable=false and rejects vector-less docs at doc validation, so the scenario is created via C/C++ — but such a collection can be read from any API. zvec forces with-id mode for every vector index (engine_helper.hpp calls with_use_id_map(false) on the shared param builder), so key == physical slot id. To keep slots aligned with row ids, the write path fills a missing row with a zero-vector placeholder registered under kInvalidKey (FlatStreamerEntity::add_vector_with_id). Therefore nothing fails, nothing is misaligned, and search never recalls the placeholder — it is simply returned as data. The problem: a missing value is indistinguishable from a genuine all-zero vector, even though the kInvalidKey marker is already stored in the flat index's keys array; the read path just never checks it.

Steps to Reproduce

Append to tests/db/iterator_test.cc, then ninja iterator_test && ./bin/iterator_test --gtest_filter='*ReproIssue691*':

TEST_F(IteratorTest, ReproIssue691) {
  auto schema = std::make_shared<CollectionSchema>("demo");
  schema->add_field(std::make_shared<FieldSchema>(
      "vec", DataType::VECTOR_FP32, 4, true /*nullable*/,
      std::make_shared<FlatIndexParams>(MetricType::IP)));
  CollectionOptions options;
  options.read_only_ = false;
  auto coll =
      Collection::CreateAndOpen(iter_test_path, *schema, options).value();
Doc d1, d2;

std::vector<float> zeros{0.0f, 0.0f, 0.0f, 0.0f};

d1.set_pk("pk_1");                         // no vector: field left out

d2.set_pk("pk_2");

d2.set<std::vector<float>>("vec", zeros);  // genuine all-zero vector

std::vector<Doc> docs{d1, d2};

ASSERT_TRUE(coll->insert(docs).has_value());

coll->flush();
auto show = [](const char *how, const std::string &pk, const auto &v) {

std::cout << how << " " << pk << ": ";

if (!v.has_value()) {

std::cout << "vec unset" << std::endl;

return;

}

std::cout << "vec =";

for (float x : v.value()) std::cout << " " << x;

std::cout << std::endl;

};
auto fetched = coll->fetch({"pk_1", "pk_2"}, std::nullopt, true).value();

for (const char *pk : {"pk_1", "pk_2"})

show("fetch  ", pk, fetched[pk]->get<std::vector<float>>("vec"));
auto it = coll->create_iterator().value();

while (auto doc = it->next().value())

show("iterate", doc->pk(), doc->get<std::vector<float>>("vec"));

it->close();

coll->destroy();

}

Actual output on main — pk_1 (no vector) and pk_2 (a real all-zero vector) are indistinguishable on both read paths:

fetch   pk_1: vec = 0 0 0 0
fetch   pk_2: vec = 0 0 0 0
iterate pk_1: vec = 0 0 0 0
iterate pk_2: vec = 0 0 0 0

Expected

fetch   pk_1: vec unset
fetch   pk_2: vec = 0 0 0 0
iterate pk_1: vec unset
iterate pk_2: vec = 0 0 0 0

Suggested fix

Propagate the stored marker on read — no storage-format and no write-path change (~110 lines across 9 files, plus tests): flat get_vector_by_key returns IndexError_NoExist for a kInvalidKey slot → Index::_dense_fetch / _sparse_fetch pass it through (real failures stay RuntimeError) → VectorColumnIndexer::Fetch maps it to Status::NotFound (block-lookup failures stay InternalError, so corruption is never read as missingness) → fetch(), create_iterator()/next() and the SQL vector fetch leave the field unset / emit NULL. Coverage, measured with that patch (300 docs, one vector-less; every other doc reads back correctly in all cells). The declared index type is only built by optimize() / create_index(), while writing segments and flushed blocks always use flat (segment.cc forces MakeDefaultVectorIndexParams):

stage	declared Flat	declared HNSW	declared IVF
before flush, and after flush()	unset	unset	unset
after optimize()	unset	0 0 0 0	0 0 0 0

So the patch fixes the default index (flat, dense and sparse) and every not-yet-optimized segment, but the same doc flips back to all zeros once the declared HNSW/IVF/Vamana index is built. Reason: on optimize(), HNSW/IVF are rebuilt via merge and the zero-vector placeholder is carried over as a genuine entry (the missing row's id is re-added with a zero vector), so get_vector_by_id(id) returns that zero vector — the kInvalidKey marker that flat keeps in its keys array is not represented in the rebuilt index, so the read side has nothing to test. A complete fix needs an index-agnostic presence record at the DB layer (e.g. a per-block bitmap of vector-less rows, remapped on merge like the delete bitmap). IVF additionally needs IVFEntity::key_to_id to stop returning one sentinel for both "key not found" and "keys-segment read failure". A complete fix therefore needs either an index-agnostic presence record at the DB layer (e.g. a per-block bitmap of vector-less rows, remapped on merge like the delete bitmap), or per-algorithm existence checks — or, if optional vectors are not meant to be supported, rejecting vector-less docs at insert, as the Python/Node SDKs already do.

Severity

Medium-low: no crash, no failure, no misalignment, no search pollution. The impact is silent data fabrication on the read path — exports, statistics and ETL checks see a "vector" that was never written — for explicitly nullable vector fields created through the C-family APIs.

[egolearner]

Additional context:

1. The Python and Node.js SDKs do not expose the nullable option for vector fields and treat all vector fields as required. Therefore, documents missing a vector field are rejected by these SDKs. The C/C++ SDKs, as well as C-based SDKs such as Go and Rust, allow vector fields to be configured as nullable/optional, so this scenario is reachable through those APIs.

2. The Flat streamer preserves row-ID alignment by inserting a zero-filled placeholder when a vector is missing. Vector indexes that are not built incrementally in the DB write path, such as HNSW, are first written as Flat indexes and rebuilt during optimize(). Therefore, vectors after a missing row are not compacted into the wrong physical slots.

3. When using SDKs that allow nullable vector fields, missing vectors do not cause fetch() to fail for Flat or HNSW indexes. The Flat implementation fills missing row IDs with zero-vector placeholders. HNSW indexes are initially written as Flat indexes and later rebuilt during optimize(), preserving those placeholders. As a result, fetching a document with a missing vector succeeds and returns an all-zero vector, both before and after HNSW optimization, rather than returning NotFound or an internal error.

Therefore, for Flat and HNSW, the actual issue is not a fetch failure or physical-slot misalignment. It is the silent conversion of a missing vector into a zero vector, making a missing value indistinguishable from a genuine all-zero vector.

There are also two problems with the Python reproduction code in the original issue:

• FieldSchema("id", DataType.STRING) creates a required scalar field because nullable defaults to False. Doc(id="id0", ...) only sets the document primary key; it does not populate the scalar field named id. Therefore, the first insert already fails with:

  Invalid doc[id0]: field[id] is required but not provided
  

• Even if the scalar id field is populated or removed, VectorSchema("vec", ...) is also non-nullable in the Python SDK. The vector-less document is rejected with:

  Invalid doc[id1]: field[vec] is required but not provided
  

Consequently, the original Python example cannot reach flush(), fetch(), or iter_docs() through the public Python API and does not reproduce the reported read-path behavior.

[YongqiYin]

修复了不严谨描述，对issue进行了更新