ci: Add cluster test for vector search #61009

EricZequan · 2025-05-08T03:25:37Z

What problem does this PR solve?

Issue Number: close #61017

Problem Summary:

What changed and how does it work?

Run some e2e tests for each PR to test vector search.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: “EricZequan” <[email protected]>

tiprow · 2025-05-08T03:25:55Z

Hi @EricZequan. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

EricZequan · 2025-05-08T03:29:38Z

/cc @breezewish Since the latest release of tiflash does not include pingcap/tiflash#10103, some tests of this PR still fail, but other tests have been verified locally. I will continue to follow up on this issue.

Signed-off-by: “EricZequan” <[email protected]>

wuhuizuo · 2025-05-08T03:41:17Z

Ref: PingCAP-QE/ci#3502

tests/clusterintegrationtest/README.md

tests/clusterintegrationtest/run1.sh

wuhuizuo · 2025-05-08T03:53:15Z

tests/clusterintegrationtest/_include.sh

+  cd - || exit 1
+
+  echo "Starting TiUP Playground in the background..."
+  tiup playground nightly --db=1 --kv=1 --tiflash=1 --db.binpath=../../bin/tidb-server &


how to use it in release branches in feature?

how to use it in CI jobs in pull requests of tiflash repo?

How to test against a possible command like tiflash=pr/xxxx?

codecov · 2025-05-08T03:55:10Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.4256%. Comparing base (e72c97a) to head (00354bc).
Report is 39 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #61009        +/-   ##
================================================
+ Coverage   73.1624%   74.4256%   +1.2631%     
================================================
  Files          1726       1729         +3     
  Lines        479299     490202     +10903     
================================================
+ Hits         350667     364836     +14169     
+ Misses       107134     104159      -2975     
+ Partials      21498      21207       -291

Flag	Coverage Δ
integration	`43.2078% <ø> (?)`
unit	`73.3841% <ø> (+0.9572%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7804% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`48.4391% <ø> (+0.8505%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: “EricZequan” <[email protected]>

purelind · 2025-05-08T08:05:01Z

/test ?

ti-chi-bot · 2025-05-08T08:05:04Z

@purelind: The following commands are available to trigger required jobs:

/test build

/test check-dev

/test check-dev2

/test mysql-test

/test pull-br-integration-test

/test pull-integration-ddl-test

/test pull-integration-e2e-test

/test pull-lightning-integration-test

/test pull-mysql-client-test

/test pull-unit-test-ddlv1

/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test

/test pull-common-test

/test pull-e2e-test

/test pull-integration-common-test

/test pull-integration-copr-test

/test pull-integration-jdbc-test

/test pull-integration-mysql-test

/test pull-integration-nodejs-test

/test pull-integration-python-orm-test

/test pull-next-gen-real-tikv-test

/test pull-sqllogic-test

/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build

pingcap/tidb/ghpr_check

pingcap/tidb/ghpr_check2

pingcap/tidb/ghpr_mysql_test

pingcap/tidb/ghpr_unit_test

pingcap/tidb/pull_integration_ddl_test

pingcap/tidb/pull_integration_e2e_test

pingcap/tidb/pull_mysql_client_test

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2025-05-08T08:05:24Z

@purelind: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2025-05-08T08:05:27Z

@ti-chi-bot[bot]: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

@purelind: The following commands are available to trigger required jobs:
/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-br-integration-test
/test pull-integration-ddl-test
/test pull-integration-e2e-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test pull-unit-test-ddlv1
/test unit-test
The following commands are available to trigger optional jobs:
/test pingcap/tidb/canary_ghpr_unit_test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-integration-python-orm-test
/test pull-next-gen-real-tikv-test
/test pull-sqllogic-test
/test pull-tiflash-test
Use /test all to run the following jobs that were automatically triggered:
pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_integration_e2e_test
pingcap/tidb/pull_mysql_client_test
In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

purelind · 2025-05-08T08:00:41Z

tests/clusterintegrationtest/_include.sh

+
+function start_tidb_latest() {
+  echo "Starting TiUP Playground in the background..."
+  tiup playground nightly --db=1 --kv=1 --tiflash=1 &


Why do we need to use nightly to start the cluster instead of using the previously built binary?

Only run_upgrade_test.sh will use this func. The test performed by this script is to build vector data under a fixed version of tidb and tiflash, and then upgrade the version to test compatibility. Here we first use the newest release for testing, and then will change it to a certain version (such as 8.5.0) later.

I also think this is wrong. @EricZequan The upgrade test should test whether old version can be upgraded to the PR version, because PR version is going to be "latest". So you should start the cluster using v8.5.1 first, then start using PR version.

I add the way of starting v8.5.1 and nightly version, PTAL~

EricZequan · 2025-05-09T02:33:37Z

/hold
Waiting for QE to configure pipeline test pr correctness

tests/clusterintegrationtest/README.md

Signed-off-by: “EricZequan” <[email protected]>

tests/clusterintegrationtest/run_python_testers.sh

Signed-off-by: “EricZequan” <[email protected]>

breezewish · 2025-05-12T09:47:24Z

tests/clusterintegrationtest/_include.sh

+  ../../bin/tidb-server -V
+  echo
+  echo "+ TiKV Version"
+  ~/.tiup/components/tikv/v9.0.0-beta.1.pre-nightly/tikv-server --version


This will be broken after TiDB switch the tag?

tikv are fixed and not be broken after change the tag. But tidb and tiflash exist this problem, I will try to fix it after @purelind giving some dependency

tests/clusterintegrationtest/_include.sh

tests/clusterintegrationtest/run_mysql_tester.sh

Relate to pingcap/tidb#61009 Add new vector search test presubmit job. The test is currently in the testing phase and will not be triggered automatically; it can only be triggered manually.

purelind · 2025-05-14T09:25:31Z

/test ?

tiprow · 2025-05-28T06:56:00Z

@EricZequan: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

/test pull-vector-search-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

wuhuizuo · 2025-05-28T06:59:18Z

tests/clusterintegrationtest/cluster.sh

+
+cd - || exit 1
+
+echo "Starting TiUP Playground in the background..."


the nightly tiup package has hours latency, how to test it with refreshed tikv and tiflash binaries?

and how to to test it with next-gen tidb binaries?

As long as it is nightly, it is fine, because we are mainly testing the impact of tidb commit on vector, and we only need to keep tidb up to date.
In addition, how to use next-gen tidb is being discussed with @purelind

wuhuizuo · 2025-05-28T07:00:12Z

tests/clusterintegrationtest/_include.sh

+    tiup tidb:v8.5.1 -V
+    echo
+    echo "+ TiKV Version"
+    tiup tikv:v8.5.1 --version
+    echo


why it should be hard coded?

In the upgrade test, we will use the v8.5.1 cluster. This cluster has been cached, so we can directly output its version.

EricZequan · 2025-05-28T07:38:07Z

/retest

tests/clusterintegrationtest/cluster.sh

EricZequan · 2025-05-28T08:37:07Z

/retest

hawkingrei · 2025-05-28T08:47:09Z

/retest

purelind · 2025-05-28T09:00:36Z

/test pull-tiflash-integration-test

tiprow · 2025-05-28T09:00:58Z

@purelind: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

/test pull-tiflash-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

EricZequan · 2025-05-28T09:25:04Z

/retest

EricZequan · 2025-05-28T09:46:37Z

/retest

Signed-off-by: “EricZequan” <[email protected]>

EricZequan · 2025-05-28T10:20:53Z

/test pull-tiflash-integration-test

tiprow · 2025-05-28T10:21:22Z

@EricZequan: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

/test pull-tiflash-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: “EricZequan” <[email protected]>

EricZequan · 2025-05-28T10:56:46Z

/test pull-tiflash-integration-test

tiprow · 2025-05-28T10:57:08Z

@EricZequan: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

/test pull-tiflash-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2025-05-28T11:03:59Z

[LGTM Timeline notifier]

Timeline:

2025-05-28 06:46:24.769382034 +0000 UTC m=+1405.831399654: ☑️ agreed by breezewish.
2025-05-28 11:03:58.715274954 +0000 UTC m=+16859.777292574: ☑️ agreed by winoros.

winoros · 2025-05-28T11:23:47Z

/hold for a while to see whether others have more comments

EricZequan · 2025-05-28T12:56:54Z

/retest

ti-chi-bot · 2025-05-30T02:33:32Z

@wuhuizuo: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

bb7133

LGTM

ti-chi-bot · 2025-06-03T10:21:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bb7133, breezewish, winoros, wuhuizuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [bb7133,breezewish,winoros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

EricZequan · 2025-06-04T02:29:46Z

/unhold

add cluster test for vector search

eae926f

Signed-off-by: “EricZequan” <[email protected]>

ti-chi-bot bot added do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 8, 2025

update

5210f2f

Signed-off-by: “EricZequan” <[email protected]>

wuhuizuo reviewed May 8, 2025

View reviewed changes

tests/clusterintegrationtest/README.md Show resolved Hide resolved

tests/clusterintegrationtest/run1.sh Outdated Show resolved Hide resolved

tests/clusterintegrationtest/run1.sh Outdated Show resolved Hide resolved

wuhuizuo reviewed May 8, 2025

View reviewed changes

update

53ac07e

Signed-off-by: “EricZequan” <[email protected]>

purelind reviewed May 9, 2025

View reviewed changes

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 9, 2025

breezewish reviewed May 10, 2025

View reviewed changes

tests/clusterintegrationtest/README.md Outdated Show resolved Hide resolved

update

b225211

Signed-off-by: “EricZequan” <[email protected]>

breezewish reviewed May 12, 2025

View reviewed changes

tests/clusterintegrationtest/run_python_testers.sh Outdated Show resolved Hide resolved

update

6a5f858

Signed-off-by: “EricZequan” <[email protected]>

breezewish reviewed May 12, 2025

View reviewed changes

purelind reviewed May 14, 2025

View reviewed changes

tests/clusterintegrationtest/_include.sh Outdated Show resolved Hide resolved

purelind reviewed May 14, 2025

View reviewed changes

tests/clusterintegrationtest/_include.sh Outdated Show resolved Hide resolved

purelind reviewed May 14, 2025

View reviewed changes

tests/clusterintegrationtest/run_mysql_tester.sh Outdated Show resolved Hide resolved

purelind mentioned this pull request May 14, 2025

feat(pingcap/tidb): add new vector search test presubmit job PingCAP-QE/ci#3519

Merged

wuhuizuo reviewed May 28, 2025

View reviewed changes

purelind reviewed May 28, 2025

View reviewed changes

tests/clusterintegrationtest/cluster.sh Show resolved Hide resolved

update

276e18c

Signed-off-by: “EricZequan” <[email protected]>

ti-chi-bot bot removed the do-not-merge/needs-tests-checked label May 28, 2025

update readme

00354bc

Signed-off-by: “EricZequan” <[email protected]>

winoros approved these changes May 28, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels May 28, 2025

wuhuizuo approved these changes May 30, 2025

View reviewed changes

bb7133 approved these changes Jun 3, 2025

View reviewed changes

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 4, 2025

ti-chi-bot bot merged commit ca423cf into pingcap:master Jun 4, 2025
25 checks passed


		cd - \|\| exit 1

		echo "Starting TiUP Playground in the background..."

ci: Add cluster test for vector search #61009

ci: Add cluster test for vector search #61009

Uh oh!

Conversation

EricZequan commented May 8, 2025 • edited by winoros Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented May 8, 2025

Uh oh!

EricZequan commented May 8, 2025

Uh oh!

wuhuizuo commented May 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

purelind commented May 8, 2025

Uh oh!

ti-chi-bot bot commented May 8, 2025

Uh oh!

tiprow bot commented May 8, 2025

Uh oh!

tiprow bot commented May 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

breezewish May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EricZequan commented May 9, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

purelind commented May 14, 2025

Uh oh!

tiprow bot commented May 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EricZequan commented May 28, 2025

Uh oh!

Uh oh!

EricZequan commented May 28, 2025

Uh oh!

hawkingrei commented May 28, 2025

Uh oh!

purelind commented May 28, 2025

Uh oh!

tiprow bot commented May 28, 2025

Uh oh!

EricZequan commented May 8, 2025 •

edited by winoros

Loading

codecov bot commented May 8, 2025 •

edited

Loading

breezewish May 12, 2025 •

edited

Loading