feat: support external collection schema evolution#50394
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: weiliu1031 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
[ci-v2-notice] To rerun ci-v2 checks, comment with:
If you have any questions or requests, please contact @zhikunyao. |
80f3ef5 to
cc3325b
Compare
❌ CI Loop Results
|
| Stage | Result | Duration | Tests |
|---|---|---|---|
| ✅ Build | SUCCESS | 14.2min | - |
| ❌ Code-Check | FAILURE | 2.0min | - |
| ✅ UT-CPP-Cov | SUCCESS | 52.5min | 7838 passed |
Total: 70min | Pipeline | Artifacts
Diff Coverage: CPP 95.5% (42 hit, 2 miss, 44 measurable lines, 30 unmeasured)
Total Patch Coverage: 95.5% (42/44 measurable lines, 30 unmeasured)
❌ CI Loop Results
|
| Stage | Result | Duration | Tests |
|---|---|---|---|
| ✅ Build | SUCCESS | 14.0min | - |
| ❌ Code-Check | FAILURE | 2.0min | - |
| ✅ UT-CPP-Cov | SUCCESS | 53.2min | 7838 passed |
Total: 72min | Pipeline | Artifacts
Diff Coverage: CPP 95.5% (42 hit, 2 miss, 44 measurable lines, 30 unmeasured)
Total Patch Coverage: 95.5% (42/44 measurable lines, 30 unmeasured)
Codecov Report❌ Patch coverage is Please upload reports for the commit 6622ace to get more accurate results.
Additional details and impacted files@@ Coverage Diff @@
## master #50394 +/- ##
===========================================
+ Coverage 78.95% 85.21% +6.26%
===========================================
Files 2239 671 -1568
Lines 397069 118431 -278638
===========================================
- Hits 313503 100923 -212580
+ Misses 73963 17456 -56507
+ Partials 9603 52 -9551
🚀 New features to boost your workflow:
|
❌ CI Loop Results
|
| Stage | Result | Duration | Tests |
|---|---|---|---|
| ✅ Build | SUCCESS | 14.1min | - |
| ❌ Code-Check | FAILURE | 4.8min | - |
| ✅ UT-CPP-Cov | SUCCESS | 52.8min | 7838 passed |
Total: 69min | Pipeline | Artifacts
Diff Coverage: CPP 95.5% (42 hit, 2 miss, 44 measurable lines, 30 unmeasured)
Total Patch Coverage: 95.5% (42/44 measurable lines, 30 unmeasured)
cc3325b to
a2b4384
Compare
❌ CI Loop Results
|
| Stage | Result | Duration | Tests |
|---|---|---|---|
| ✅ Build | SUCCESS | 9.3min | - |
| ❌ Code-Check | FAILURE | 3.8min | - |
| ✅ UT-CPP-Cov | SUCCESS | 36.4min | 7838 passed |
Total: 48min | Pipeline | Artifacts
Diff Coverage: CPP 95.5% (42 hit, 2 miss, 44 measurable lines, 30 unmeasured)
Total Patch Coverage: 95.5% (42/44 measurable lines, 30 unmeasured)
a2b4384 to
514503d
Compare
514503d to
4452a94
Compare
❌ CI Loop Results
|
| Stage | Result | Duration | Tests |
|---|---|---|---|
| ✅ Build | SUCCESS | 9.5min | - |
| ✅ Code-Check | SUCCESS | 5.9min | - |
| ❌ UT-GO | FAILURE | 26.8min | 1027 run, failed (details unavailable) |
| ✅ UT-Integration | SUCCESS | 24.3min | 46 passed |
| ✅ UT-CPP-Cov | SUCCESS | 36.8min | 7839 passed |
Total: 113min | Pipeline | Artifacts
Diff Coverage: CPP 95.5% (42 hit, 2 miss, 44 measurable lines, 30 unmeasured)
Total Patch Coverage: 95.5% (42/44 measurable lines, 30 unmeasured)
Failed Test Logs:
- UT-GO: view log
issue: milvus-io#45881 External collections need schema metadata changes and refresh tasks to advance while the collection is loaded. This change lets schema alter reach the DDL path, validates external source-backed and function-output fields, and records the refresh schema version so stale refresh results are rejected. It also adds SDK helpers, refresh configuration, generated DataCoord proto updates, and the design document for the schema evolution flow. Validation: - git diff --cached --check Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: milvus-io#45881 External collections can expose schema-visible fields before refreshed manifests materialize them. Serving, search, and index paths must reject those fields until refresh and reopen catch QueryNode up. This change adds the coverage resolver, QueryNode validation, index gating, manifest reopen handling, BM25 stats refresh, and regression coverage for missing external fields. Snapshot restore also preserves packed child-field coverage so restored external collections keep the materialization guarantees captured by their snapshots. Validation: - git diff --check - source ~/.profile && source scripts/setenv.sh && make milvus - go test internal/datacoord targeted snapshot tests - go test internal/querynodev2/delegator TestLoadGrowingWithBM25 Signed-off-by: Wei Liu <wei.liu@zilliz.com>
4452a94 to
6622ace
Compare
issue: #45881
design doc: docs/design-docs/design_docs/20260605-external-table-schema-evolution.md
What changed
This PR supports schema evolution for external collections.
It allows
AlterCollectionSchemato update external collection schemas, validates source-backed fields and function-output fields, adds SDK helpers, and records refresh schema versions so stale refresh results are rejected.It also ensures schema-visible external fields are not served until their backing manifests are materialized. Query/search/index paths now reject unmaterialized fields, QueryCoord schedules loaded segment reopen when manifests are stale, and QueryNode refreshes BM25/function state after schema changes.
Validation