Skip to content

feat: support external collection schema evolution#50394

Open
weiliu1031 wants to merge 2 commits into
milvus-io:masterfrom
weiliu1031:external-schema-evolution-pr
Open

feat: support external collection schema evolution#50394
weiliu1031 wants to merge 2 commits into
milvus-io:masterfrom
weiliu1031:external-schema-evolution-pr

Conversation

@weiliu1031

Copy link
Copy Markdown
Contributor

issue: #45881

design doc: docs/design-docs/design_docs/20260605-external-table-schema-evolution.md

What changed

This PR supports schema evolution for external collections.

It allows AlterCollectionSchema to update external collection schemas, validates source-backed fields and function-output fields, adds SDK helpers, and records refresh schema versions so stale refresh results are rejected.

It also ensures schema-visible external fields are not served until their backing manifests are materialized. Query/search/index paths now reject unmaterialized fields, QueryCoord schedules loaded segment reopen when manifests are stale, and QueryNode refreshes BM25/function state after schema changes.

Validation

  • git diff --cached --check

@sre-ci-robot sre-ci-robot requested review from bigsheeper and czs007 June 9, 2026 03:07
@sre-ci-robot sre-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines. label Jun 9, 2026
@mergify mergify Bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Jun 9, 2026
@sre-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: weiliu1031
To complete the pull request process, please assign jiaoew1991 after the PR has been reviewed.
You can assign the PR to them by writing /assign @jiaoew1991 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot

Copy link
Copy Markdown
Contributor

[ci-v2-notice]
Notice: New ci-v2 system is enabled for this PR.

To rerun ci-v2 checks, comment with:

  • /ci-rerun-code-check // for ci-v2/code-check
  • /ci-rerun-code-check-macos // for Code Checker MacOS (GitHub Actions)
  • /ci-rerun-build // for ci-v2/build
  • /ci-rerun-build-all // for ci-v2/build-all (multi-arch builds)
  • /ci-rerun-buildenv // for ci-v2/build-env (build milvus-env builder images)
  • /ci-rerun-ut-integration // for ci-v2/ut-integration, will rerun ci-v2/build
  • /ci-rerun-ut-go // for ci-v2/ut-go, will rerun ci-v2/build
  • /ci-rerun-ut-cpp // for ci-v2/ut-cpp
  • /ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp, will rerun ci-v2/build
  • /ci-rerun-e2e-default // for ci-v2/e2e-default
  • /ci-rerun-e2e-amd // for ci-v2/e2e-amd (e2e pool dispatcher)
  • /ci-rerun-build-ut-cov // for ci-v2/build-ut-cov (build + unit tests in one pipeline)
  • /ci-rerun-gosdk // for ci-v2/go-sdk (Go SDK E2E tests, ARM)

If you have any questions or requests, please contact @zhikunyao.

@weiliu1031 weiliu1031 force-pushed the external-schema-evolution-pr branch from 80f3ef5 to cc3325b Compare June 9, 2026 03:53
@sre-ci-robot

Copy link
Copy Markdown
Contributor

❌ CI Loop Results 80f3ef5

Stage Result Duration Tests
✅ Build SUCCESS 14.2min -
❌ Code-Check FAILURE 2.0min -
✅ UT-CPP-Cov SUCCESS 52.5min 7838 passed

Total: 70min | Pipeline | Artifacts
Diff Coverage: CPP 95.5% (42 hit, 2 miss, 44 measurable lines, 30 unmeasured)
Total Patch Coverage: 95.5% (42/44 measurable lines, 30 unmeasured)

@sre-ci-robot

Copy link
Copy Markdown
Contributor

❌ CI Loop Results 80f3ef5

Stage Result Duration Tests
✅ Build SUCCESS 14.0min -
❌ Code-Check FAILURE 2.0min -
✅ UT-CPP-Cov SUCCESS 53.2min 7838 passed

Total: 72min | Pipeline | Artifacts
Diff Coverage: CPP 95.5% (42 hit, 2 miss, 44 measurable lines, 30 unmeasured)
Total Patch Coverage: 95.5% (42/44 measurable lines, 30 unmeasured)

@sre-ci-robot sre-ci-robot added the low-code-coverage add test-label from zhikun, diff coverage > 80% label Jun 9, 2026
@codecov

codecov Bot commented Jun 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.87500% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 85.21%. Comparing base (1853a14) to head (4452a94).
⚠️ Report is 3 commits behind head on master.

⚠️ Current head 4452a94 differs from pull request most recent head 6622ace

Please upload reports for the commit 6622ace to get more accurate results.

Files with missing lines Patch % Lines
...rnal/core/src/segcore/ChunkedSegmentSealedTest.cpp 95.23% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           master   #50394       +/-   ##
===========================================
+ Coverage   78.95%   85.21%    +6.26%     
===========================================
  Files        2239      671     -1568     
  Lines      397069   118431   -278638     
===========================================
- Hits       313503   100923   -212580     
+ Misses      73963    17456    -56507     
+ Partials     9603       52     -9551     
Components Coverage Δ
Client ∅ <ø> (∅)
Core 85.94% <96.87%> (+<0.01%) ⬆️
Go ∅ <ø> (∅)
Files with missing lines Coverage Δ
internal/core/src/segcore/SegmentLoadInfoTest.cpp 99.00% <100.00%> (+<0.01%) ⬆️
...rnal/core/src/segcore/ChunkedSegmentSealedTest.cpp 99.72% <95.23%> (-0.14%) ⬇️

... and 1570 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sre-ci-robot

Copy link
Copy Markdown
Contributor

❌ CI Loop Results cc3325b

Stage Result Duration Tests
✅ Build SUCCESS 14.1min -
❌ Code-Check FAILURE 4.8min -
✅ UT-CPP-Cov SUCCESS 52.8min 7838 passed

Total: 69min | Pipeline | Artifacts
Diff Coverage: CPP 95.5% (42 hit, 2 miss, 44 measurable lines, 30 unmeasured)
Total Patch Coverage: 95.5% (42/44 measurable lines, 30 unmeasured)

@weiliu1031 weiliu1031 force-pushed the external-schema-evolution-pr branch from cc3325b to a2b4384 Compare June 9, 2026 06:29
@sre-ci-robot

Copy link
Copy Markdown
Contributor

❌ CI Loop Results a2b4384

Stage Result Duration Tests
✅ Build SUCCESS 9.3min -
❌ Code-Check FAILURE 3.8min -
✅ UT-CPP-Cov SUCCESS 36.4min 7838 passed

Total: 48min | Pipeline | Artifacts
Diff Coverage: CPP 95.5% (42 hit, 2 miss, 44 measurable lines, 30 unmeasured)
Total Patch Coverage: 95.5% (42/44 measurable lines, 30 unmeasured)

@sre-ci-robot sre-ci-robot added low-code-coverage add test-label from zhikun, diff coverage > 80% and removed low-code-coverage add test-label from zhikun, diff coverage > 80% labels Jun 9, 2026
@weiliu1031 weiliu1031 force-pushed the external-schema-evolution-pr branch from a2b4384 to 514503d Compare June 9, 2026 08:08
@sre-ci-robot sre-ci-robot removed the low-code-coverage add test-label from zhikun, diff coverage > 80% label Jun 9, 2026
@weiliu1031 weiliu1031 force-pushed the external-schema-evolution-pr branch from 514503d to 4452a94 Compare June 9, 2026 09:32
@sre-ci-robot

Copy link
Copy Markdown
Contributor

❌ CI Loop Results 4452a94

Stage Result Duration Tests
✅ Build SUCCESS 9.5min -
✅ Code-Check SUCCESS 5.9min -
❌ UT-GO FAILURE 26.8min 1027 run, failed (details unavailable)
✅ UT-Integration SUCCESS 24.3min 46 passed
✅ UT-CPP-Cov SUCCESS 36.8min 7839 passed

Total: 113min | Pipeline | Artifacts
Diff Coverage: CPP 95.5% (42 hit, 2 miss, 44 measurable lines, 30 unmeasured)
Total Patch Coverage: 95.5% (42/44 measurable lines, 30 unmeasured)

Failed Test Logs:

issue: milvus-io#45881

External collections need schema metadata changes and refresh tasks
to advance while the collection is loaded. This change lets schema alter
reach the DDL path, validates external source-backed and function-output
fields, and records the refresh schema version so stale refresh results
are rejected.

It also adds SDK helpers, refresh configuration, generated DataCoord
proto updates, and the design document for the schema evolution flow.

Validation:
- git diff --cached --check

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: milvus-io#45881

External collections can expose schema-visible fields before refreshed
manifests materialize them. Serving, search, and index paths must
reject those fields until refresh and reopen catch QueryNode up.

This change adds the coverage resolver, QueryNode validation, index
gating, manifest reopen handling, BM25 stats refresh, and regression
coverage for missing external fields. Snapshot restore also preserves
packed child-field coverage so restored external collections keep the
materialization guarantees captured by their snapshots.

Validation:
- git diff --check
- source ~/.profile && source scripts/setenv.sh && make milvus
- go test internal/datacoord targeted snapshot tests
- go test internal/querynodev2/delegator TestLoadGrowingWithBM25

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
@weiliu1031 weiliu1031 force-pushed the external-schema-evolution-pr branch from 4452a94 to 6622ace Compare June 10, 2026 02:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/test dco-passed DCO check passed. kind/feature Issues related to feature request from users sig/testing size/XXL Denotes a PR that changes 1000+ lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants