Skip to content

Commit 0610dc4

Browse files
takaidohigasitakaidohigasi
committed
Merge remote-tracking branch 'upstream/master' into improve-string-key-handling
# Conflicts: # dumpling/export/ir.go # dumpling/export/util_for_test.go
2 parents 8e5eb75 + aaa5331 commit 0610dc4

667 files changed

Lines changed: 20190 additions & 11811 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/tidb-test-guidelines/references/lightning-case-map.md‎

Lines changed: 18 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -11,24 +11,24 @@
1111
### Tests
1212
- `pkg/lightning/backend/backend_test.go` - lightning/backend: Tests open/close import cleanup engine.
1313

14-
## pkg/lightning/backend/external
15-
16-
### Tests
17-
- `pkg/lightning/backend/external/bench_test.go` - lightning/backend/external: Tests compare writer.
18-
- `pkg/lightning/backend/external/byte_reader_test.go` - lightning/backend/external: Tests byte reader.
19-
- `pkg/lightning/backend/external/codec_test.go` - lightning/backend/external: Tests range property codec.
20-
- `pkg/lightning/backend/external/concurrent_reader_test.go` - lightning/backend/external: Tests concurrent read.
21-
- `pkg/lightning/backend/external/engine_test.go` - lightning/backend/external: Tests memory ingest data.
22-
- `pkg/lightning/backend/external/file_test.go` - lightning/backend/external: Tests add key-value maintain range property.
23-
- `pkg/lightning/backend/external/iter_test.go` - lightning/backend/external: Tests merge KV iter.
24-
- `pkg/lightning/backend/external/merge_test.go` - lightning/backend/external: Tests split data files.
25-
- `pkg/lightning/backend/external/misc_bench_test.go` - lightning/backend/external: Tests misc bench.
26-
- `pkg/lightning/backend/external/onefile_writer_test.go` - lightning/backend/external: Tests onefile writer basic.
27-
- `pkg/lightning/backend/external/reader_test.go` - lightning/backend/external: Tests read all data basic.
28-
- `pkg/lightning/backend/external/sort_test.go` - lightning/backend/external: Tests global sort local basic.
29-
- `pkg/lightning/backend/external/split_test.go` - lightning/backend/external: Tests general properties.
30-
- `pkg/lightning/backend/external/util_test.go` - lightning/backend/external: Tests seek props offsets.
31-
- `pkg/lightning/backend/external/writer_test.go` - lightning/backend/external: Tests writer.
14+
## pkg/ingestor/globalsort
15+
16+
### Tests
17+
- `pkg/ingestor/globalsort/bench_test.go` - ingestor/globalsort: Tests compare writer.
18+
- `pkg/ingestor/globalsort/byte_reader_test.go` - ingestor/globalsort: Tests byte reader.
19+
- `pkg/ingestor/globalsort/codec_test.go` - ingestor/globalsort: Tests range property codec.
20+
- `pkg/ingestor/globalsort/concurrent_reader_test.go` - ingestor/globalsort: Tests concurrent read.
21+
- `pkg/ingestor/globalsort/engine_test.go` - ingestor/globalsort: Tests memory ingest data.
22+
- `pkg/ingestor/globalsort/file_test.go` - ingestor/globalsort: Tests add key-value maintain range property.
23+
- `pkg/ingestor/globalsort/iter_test.go` - ingestor/globalsort: Tests merge KV iter.
24+
- `pkg/ingestor/globalsort/merge_test.go` - ingestor/globalsort: Tests split data files.
25+
- `pkg/ingestor/globalsort/misc_bench_test.go` - ingestor/globalsort: Tests misc bench.
26+
- `pkg/ingestor/globalsort/onefile_writer_test.go` - ingestor/globalsort: Tests onefile writer basic.
27+
- `pkg/ingestor/globalsort/reader_test.go` - ingestor/globalsort: Tests read all data basic.
28+
- `pkg/ingestor/globalsort/sort_test.go` - ingestor/globalsort: Tests global sort local basic.
29+
- `pkg/ingestor/globalsort/split_test.go` - ingestor/globalsort: Tests general properties.
30+
- `pkg/ingestor/globalsort/util_test.go` - ingestor/globalsort: Tests seek props offsets.
31+
- `pkg/ingestor/globalsort/writer_test.go` - ingestor/globalsort: Tests writer.
3232

3333
## pkg/lightning/backend/kv
3434

‎.github/licenserc.yml‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -57,10 +57,6 @@ header:
5757
- "pkg/objstore/mockobjstore/*_mock.go"
5858
- "pkg/extension/enterprise/"
5959
- "lightning/pkg/**/mock/*_mock.go"
60-
- "lightning/pkg/web/res_vfsdata.go"
61-
- "lightning/web/docs/api.yaml"
62-
- "lightning/web/public/index.html"
63-
- "lightning/web/webpack.config.js"
6460
- "lightning/pkg/checkpoints/checkpointspb/file_checkpoints.pb.go"
6561
- "pkg/lightning/manual/manual.go"
6662
- "pkg/lightning/manual/manual_nocgo.go"

‎.github/workflows/check-bazel-prepare.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: Check Bazel Prepare
2+
on:
3+
pull_request:
4+
branches:
5+
- master
6+
7+
permissions:
8+
contents: read
9+
10+
concurrency:
11+
group: check-bazel-prepare-${{ github.event.pull_request.number }}
12+
cancel-in-progress: true
13+
14+
jobs:
15+
check:
16+
name: Check Bazel Prepare
17+
runs-on: ubuntu-latest
18+
steps:
19+
- uses: actions/checkout@v6
20+
with:
21+
repository: ${{ github.event.pull_request.head.repo.full_name }}
22+
ref: ${{ github.event.pull_request.head.sha }}
23+
fetch-depth: 0
24+
persist-credentials: false
25+
- name: Set up Go
26+
uses: actions/setup-go@v6
27+
with:
28+
go-version: 1.25
29+
cache: false
30+
- name: Set up Bazelisk
31+
uses: bazel-contrib/setup-bazel@0.16.0
32+
with:
33+
bazelisk-cache: false
34+
repository-cache: false
35+
external-cache: false
36+
- name: Check Bazel Prepare
37+
shell: bash
38+
run: |
39+
set -euo pipefail
40+
# Avoid the TiDB CI path that assumes Jenkins-owned temp directories.
41+
unset CI
42+
make check-bazel-prepare

‎.github/workflows/generate-bazel-files.yml‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,16 @@ jobs:
4242
sed -i '/ats.apps.svc/d' DEPS.bzl
4343
sed -i '/bazel-cache/d' WORKSPACE
4444
sed -i '/ats.apps.svc/d' WORKSPACE
45-
make bazel_prepare
45+
bazel run //:gazelle
46+
bazel run //:gazelle -- update-repos -from_file=go.mod -to_macro DEPS.bzl%go_deps -build_file_proto_mode=disable -prune
47+
bazel run \
48+
--run_under="cd ${PWD} && " \
49+
//tools/tazel:tazel
50+
51+
tmp_out="$(mktemp -d -t tidbbzl.XXXXXX)"
52+
trap 'rm -rf "${tmp_out}"' EXIT
53+
bazel run //cmd/mirror -- --mirror > "${tmp_out}/DEPS.bzl"
54+
cp "${tmp_out}/DEPS.bzl" DEPS.bzl
4655
- name: Restore non-generated files
4756
shell: bash
4857
run: |
@@ -62,10 +71,16 @@ jobs:
6271
fi
6372
6473
case "${path}" in
74+
build/BUILD.bazel|build/*/BUILD.bazel)
75+
;;
76+
WORKSPACE|WORKSPACE.bazel|build|build/*)
77+
echo "Unsafe generated file path: ${path}" >&2
78+
exit 1
79+
;;
6580
DEPS.bzl|*.bazel|*.bzl)
6681
;;
6782
*)
68-
echo "Unexpected file changed by bazel_prepare: ${path}" >&2
83+
echo "Unexpected generated file path: ${path}" >&2
6984
exit 1
7085
;;
7186
esac

‎.github/workflows/integration-test-dumpling.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,23 @@ jobs:
9090
run: make build_lightning
9191
- name: "integration test"
9292
run: make dumpling_integration_test VERBOSE="true"
93+
- name: "dump tidb log on failure"
94+
if: ${{ failure() }}
95+
run: |
96+
if [ -f /tmp/dumpling_test_result/tidb.log ]; then
97+
echo "=== /tmp/dumpling_test_result/tidb.log (tail -n 500) ==="
98+
tail -n 500 /tmp/dumpling_test_result/tidb.log
99+
else
100+
echo "tidb.log not found at /tmp/dumpling_test_result/tidb.log"
101+
fi
102+
- name: "upload tidb log artifact"
103+
if: ${{ failure() }}
104+
uses: actions/upload-artifact@v4
105+
with:
106+
name: dumpling-tidb-log-${{ matrix.mysql_version }}
107+
path: /tmp/dumpling_test_result/tidb.log
108+
if-no-files-found: ignore
109+
retention-days: 7
93110
- name: "set up tmate session if necessary"
94111
if: ${{ failure() && github.event_name == 'workflow_dispatch' && github.event.inputs.debug }}
95112
uses: mxschmitt/action-tmate@v3

‎.github/workflows/update-bazel-files.yml‎

Lines changed: 46 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -59,11 +59,10 @@ jobs:
5959
6060
if [[ "${head_repo}" != "${REPOSITORY}" ]]; then
6161
echo "same_repo=false" >> "${GITHUB_OUTPUT}"
62-
exit 0
62+
else
63+
echo "same_repo=true" >> "${GITHUB_OUTPUT}"
6364
fi
6465
65-
echo "same_repo=true" >> "${GITHUB_OUTPUT}"
66-
6766
if [[ "${pr_head_sha}" != "${RUN_HEAD_SHA}" ]]; then
6867
echo "stale=true" >> "${GITHUB_OUTPUT}"
6968
exit 0
@@ -75,6 +74,42 @@ jobs:
7574
shell: bash
7675
run: |
7776
echo "Skip stale bazel update for PR #${{ steps.pr.outputs.pr_number }}."
77+
- name: Download Fork Bazel Files Artifact
78+
if: steps.pr.outputs.same_repo != 'true' && steps.pr.outputs.stale != 'true'
79+
uses: actions/download-artifact@v4
80+
with:
81+
name: bazel-files
82+
path: bazel-artifact
83+
repository: ${{ github.repository }}
84+
run-id: ${{ github.event.workflow_run.id }}
85+
github-token: ${{ github.token }}
86+
- name: Report fork PR Bazel changes
87+
if: steps.pr.outputs.same_repo != 'true' && steps.pr.outputs.stale != 'true'
88+
shell: bash
89+
env:
90+
PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
91+
REPOSITORY: ${{ github.repository }}
92+
RUN_ID: ${{ github.event.workflow_run.id }}
93+
run: |
94+
set -euo pipefail
95+
96+
patch_path="${PWD}/bazel-artifact/bazel.patch"
97+
if [[ ! -f "${patch_path}" ]]; then
98+
echo "Missing bazel.patch artifact for fork PR #${PR_NUMBER}." >&2
99+
exit 1
100+
fi
101+
102+
if [[ ! -s "${patch_path}" ]]; then
103+
echo "No generated Bazel changes for fork PR #${PR_NUMBER}."
104+
exit 0
105+
fi
106+
107+
{
108+
echo "Generated Bazel changes cannot be committed automatically for fork PR #${PR_NUMBER}."
109+
echo "Run: gh run download ${RUN_ID} --repo ${REPOSITORY} --name bazel-files --dir bazel-artifact && git apply bazel-artifact/bazel.patch"
110+
} | tee -a "${GITHUB_STEP_SUMMARY}"
111+
112+
exit 1
78113
- name: Checkout PR branch
79114
if: steps.pr.outputs.same_repo == 'true' && steps.pr.outputs.stale != 'true'
80115
uses: actions/checkout@v6
@@ -101,7 +136,7 @@ jobs:
101136
set -euo pipefail
102137
103138
patch_path="${PWD}/bazel-artifact/bazel.patch"
104-
summary="$(git apply --summary "${patch_path}")"
139+
summary="$(git -C pr apply --summary "${patch_path}")"
105140
106141
while IFS= read -r line; do
107142
case "${line}" in
@@ -126,6 +161,12 @@ jobs:
126161
echo "Unexpected patch path: ${path}" >&2
127162
exit 1
128163
;;
164+
build/BUILD.bazel|build/*/BUILD.bazel)
165+
;;
166+
WORKSPACE|WORKSPACE.bazel|build|build/*)
167+
echo "Unsafe patch path: ${path}" >&2
168+
exit 1
169+
;;
129170
DEPS.bzl|*.bazel|*.bzl)
130171
;;
131172
*)
@@ -139,7 +180,7 @@ jobs:
139180
exit 1
140181
fi
141182
seen_paths["${path}"]=1
142-
done < <(git apply --numstat -z "${patch_path}")
183+
done < <(git -C pr apply --numstat -z "${patch_path}")
143184
144185
git -C pr apply --check --index "${patch_path}"
145186
git -C pr apply --index "${patch_path}"

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,3 +80,7 @@ var
8080
# Personal config files
8181
/*config.toml
8282
.cache
83+
84+
# Claude Code runtime state (per-user, not part of repo)
85+
.claude/scheduled_tasks.lock
86+
.claude/settings.local.json

‎DEPS.bzl‎

Lines changed: 18 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -6478,13 +6478,13 @@ def go_deps():
64786478
name = "com_github_pingcap_errors",
64796479
build_file_proto_mode = "disable_global",
64806480
importpath = "github.com/pingcap/errors",
6481-
sha256 = "e90414b6322256cb88c6c39569070e23c074a95a803df4d2cedb10136e1d121c",
6482-
strip_prefix = "github.com/pingcap/errors@v0.11.5-0.20260310054046-9c8b3586e4b2",
6481+
sha256 = "8843ed815adc22e020095250e3d0ac3e2ffd974260dbb58c050491bb3ba2f926",
6482+
strip_prefix = "github.com/pingcap/errors@v0.11.5-0.20260508054701-306e305bcf41",
64836483
urls = [
6484-
"http://bazel-cache.pingcap.net:8080/gomod/github.com/pingcap/errors/com_github_pingcap_errors-v0.11.5-0.20260310054046-9c8b3586e4b2.zip",
6485-
"http://ats.apps.svc/gomod/github.com/pingcap/errors/com_github_pingcap_errors-v0.11.5-0.20260310054046-9c8b3586e4b2.zip",
6486-
"https://cache.hawkingrei.com/gomod/github.com/pingcap/errors/com_github_pingcap_errors-v0.11.5-0.20260310054046-9c8b3586e4b2.zip",
6487-
"https://storage.googleapis.com/pingcapmirror/gomod/github.com/pingcap/errors/com_github_pingcap_errors-v0.11.5-0.20260310054046-9c8b3586e4b2.zip",
6484+
"http://bazel-cache.pingcap.net:8080/gomod/github.com/pingcap/errors/com_github_pingcap_errors-v0.11.5-0.20260508054701-306e305bcf41.zip",
6485+
"http://ats.apps.svc/gomod/github.com/pingcap/errors/com_github_pingcap_errors-v0.11.5-0.20260508054701-306e305bcf41.zip",
6486+
"https://cache.hawkingrei.com/gomod/github.com/pingcap/errors/com_github_pingcap_errors-v0.11.5-0.20260508054701-306e305bcf41.zip",
6487+
"https://storage.googleapis.com/pingcapmirror/gomod/github.com/pingcap/errors/com_github_pingcap_errors-v0.11.5-0.20260508054701-306e305bcf41.zip",
64886488
],
64896489
)
64906490
go_repository(
@@ -6582,13 +6582,13 @@ def go_deps():
65826582
name = "com_github_pingcap_tipb",
65836583
build_file_proto_mode = "disable_global",
65846584
importpath = "github.com/pingcap/tipb",
6585-
sha256 = "2d6aaef873e175599c39f6fe3cf85cabce88a69c157ff7ec4c22fbcbc97648dd",
6586-
strip_prefix = "github.com/pingcap/tipb@v0.0.0-20260210113932-1447c9d7e9fe",
6585+
sha256 = "a47ec816b2fa1924a4db5c2270a3bfb70f7c5bcc790b59287b5e9680b71bfbcd",
6586+
strip_prefix = "github.com/pingcap/tipb@v0.0.0-20260515142222-a4d204a193b4",
65876587
urls = [
6588-
"http://bazel-cache.pingcap.net:8080/gomod/github.com/pingcap/tipb/com_github_pingcap_tipb-v0.0.0-20260210113932-1447c9d7e9fe.zip",
6589-
"http://ats.apps.svc/gomod/github.com/pingcap/tipb/com_github_pingcap_tipb-v0.0.0-20260210113932-1447c9d7e9fe.zip",
6590-
"https://cache.hawkingrei.com/gomod/github.com/pingcap/tipb/com_github_pingcap_tipb-v0.0.0-20260210113932-1447c9d7e9fe.zip",
6591-
"https://storage.googleapis.com/pingcapmirror/gomod/github.com/pingcap/tipb/com_github_pingcap_tipb-v0.0.0-20260210113932-1447c9d7e9fe.zip",
6588+
"http://bazel-cache.pingcap.net:8080/gomod/github.com/pingcap/tipb/com_github_pingcap_tipb-v0.0.0-20260515142222-a4d204a193b4.zip",
6589+
"http://ats.apps.svc/gomod/github.com/pingcap/tipb/com_github_pingcap_tipb-v0.0.0-20260515142222-a4d204a193b4.zip",
6590+
"https://cache.hawkingrei.com/gomod/github.com/pingcap/tipb/com_github_pingcap_tipb-v0.0.0-20260515142222-a4d204a193b4.zip",
6591+
"https://storage.googleapis.com/pingcapmirror/gomod/github.com/pingcap/tipb/com_github_pingcap_tipb-v0.0.0-20260515142222-a4d204a193b4.zip",
65926592
],
65936593
)
65946594
go_repository(
@@ -7319,19 +7319,6 @@ def go_deps():
73197319
"https://storage.googleapis.com/pingcapmirror/gomod/github.com/shurcooL/httpfs/com_github_shurcool_httpfs-v0.0.0-20230704072500-f1e31cf0ba5c.zip",
73207320
],
73217321
)
7322-
go_repository(
7323-
name = "com_github_shurcool_httpgzip",
7324-
build_file_proto_mode = "disable_global",
7325-
importpath = "github.com/shurcooL/httpgzip",
7326-
sha256 = "70ef73fce2f89d622f828cb439fd6c7b48a7fe63600410a8c0a936042c0e4631",
7327-
strip_prefix = "github.com/shurcooL/httpgzip@v0.0.0-20190720172056-320755c1c1b0",
7328-
urls = [
7329-
"http://bazel-cache.pingcap.net:8080/gomod/github.com/shurcooL/httpgzip/com_github_shurcool_httpgzip-v0.0.0-20190720172056-320755c1c1b0.zip",
7330-
"http://ats.apps.svc/gomod/github.com/shurcooL/httpgzip/com_github_shurcool_httpgzip-v0.0.0-20190720172056-320755c1c1b0.zip",
7331-
"https://cache.hawkingrei.com/gomod/github.com/shurcooL/httpgzip/com_github_shurcool_httpgzip-v0.0.0-20190720172056-320755c1c1b0.zip",
7332-
"https://storage.googleapis.com/pingcapmirror/gomod/github.com/shurcooL/httpgzip/com_github_shurcool_httpgzip-v0.0.0-20190720172056-320755c1c1b0.zip",
7333-
],
7334-
)
73357322
go_repository(
73367323
name = "com_github_shurcool_vfsgen",
73377324
build_file_proto_mode = "disable_global",
@@ -7805,13 +7792,13 @@ def go_deps():
78057792
build_tags = ["nextgen", "intest"],
78067793
build_file_proto_mode = "disable_global",
78077794
importpath = "github.com/tikv/client-go/v2",
7808-
sha256 = "a5e70899561f437abf34016a1fd907ab6394ab4771f75ba509b61dd5bb806351",
7809-
strip_prefix = "github.com/tikv/client-go/v2@v2.0.8-0.20260414033830-1adc54c38a51",
7795+
sha256 = "239c5c831ab7ca4f27c0e2b3e73b2df5d828c38fb30ce2700c9d7f52b0ae4008",
7796+
strip_prefix = "github.com/tikv/client-go/v2@v2.0.8-0.20260429090231-839228f8c022",
78107797
urls = [
7811-
"http://bazel-cache.pingcap.net:8080/gomod/github.com/tikv/client-go/v2/com_github_tikv_client_go_v2-v2.0.8-0.20260414033830-1adc54c38a51.zip",
7812-
"http://ats.apps.svc/gomod/github.com/tikv/client-go/v2/com_github_tikv_client_go_v2-v2.0.8-0.20260414033830-1adc54c38a51.zip",
7813-
"https://cache.hawkingrei.com/gomod/github.com/tikv/client-go/v2/com_github_tikv_client_go_v2-v2.0.8-0.20260414033830-1adc54c38a51.zip",
7814-
"https://storage.googleapis.com/pingcapmirror/gomod/github.com/tikv/client-go/v2/com_github_tikv_client_go_v2-v2.0.8-0.20260414033830-1adc54c38a51.zip",
7798+
"http://bazel-cache.pingcap.net:8080/gomod/github.com/tikv/client-go/v2/com_github_tikv_client_go_v2-v2.0.8-0.20260429090231-839228f8c022.zip",
7799+
"http://ats.apps.svc/gomod/github.com/tikv/client-go/v2/com_github_tikv_client_go_v2-v2.0.8-0.20260429090231-839228f8c022.zip",
7800+
"https://cache.hawkingrei.com/gomod/github.com/tikv/client-go/v2/com_github_tikv_client_go_v2-v2.0.8-0.20260429090231-839228f8c022.zip",
7801+
"https://storage.googleapis.com/pingcapmirror/gomod/github.com/tikv/client-go/v2/com_github_tikv_client_go_v2-v2.0.8-0.20260429090231-839228f8c022.zip",
78157802
],
78167803
)
78177804
go_repository(

0 commit comments

Comments
 (0)