Fix kafka image migration

[lmzzzzz1]

fix: #47107

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lmzzzzz1
To complete the pull request process, please assign tedxu after the PR has been reviewed.
You can assign the PR to them by writing /assign @tedxu in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sre-ci-robot]

[ci-v2-notice]
Notice: New ci-v2 system is enabled for this PR.

To rerun ci-v2 checks, comment with:

• /ci-rerun-code-check // for ci-v2/code-check
• /ci-rerun-build // for ci-v2/build
• /ci-rerun-ut-integration // for ci-v2/ut-integration, will rerun ci-v2/build
• /ci-rerun-ut-go // for ci-v2/ut-go, will rerun ci-v2/build
• /ci-rerun-ut-cpp // for ci-v2/ut-cpp
• /ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp, will rerun ci-v2/build
• /ci-rerun-e2e-arm // for ci-v2/e2e-arm
• /ci-rerun-e2e-default // for ci-v2/e2e-default

If you have any questions or requests, please contact @zhikunyao.

[mergify]

@lmzzzzz1 Thanks for your contribution. Please submit with DCO, see the contributing guide https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md#developer-certificate-of-origin-dco.

[mergify]

@lmzzzzz1

Invalid PR Title Format Detected

Your PR submission does not adhere to our required standards. To ensure clarity and consistency, please meet the following criteria:

1. Title Format: The PR title must begin with one of these prefixes:

• feat: for introducing a new feature.
• fix: for bug fixes.
• enhance: for improvements to existing functionality.
• test: for add tests to existing functionality.
• doc: for modifying documentation.
• auto: for the pull request from bot.
• build(deps): for dependency updates from Dependabot.

2. Description Requirement: The PR must include a non-empty description, detailing the changes and their impact.

Required Title Structure:

[Type]: [Description of the PR]

Where Type is one of feat, fix, enhance, test or doc.

Example:

enhance: improve search performance significantly 

Please review and update your PR to comply with these guidelines.

[codecov]

Codecov Report

❌ Patch coverage is 72.29350% with 1231 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.71%. Comparing base (4f080bd) to head (9fa30b3).
⚠️ Report is 346 commits behind head on master.

Files with missing lines	Patch %	Lines
...rnal/core/src/segcore/ChunkedSegmentSealedImpl.cpp	19.23%	294 Missing ⚠️
...re/storagev2translator/ManifestGroupTranslator.cpp	0.00%	150 Missing ⚠️
internal/core/src/segcore/SegmentLoadInfo.cpp	0.00%	142 Missing ⚠️
internal/core/src/segcore/SegmentGrowingImpl.cpp	4.13%	116 Missing ⚠️
internal/core/src/storage/Util.cpp	23.88%	51 Missing ⚠️
internal/core/src/segcore/segment_c.cpp	37.68%	43 Missing ⚠️
client/milvusclient/iterator.go	72.66%	29 Missing and 12 partials ⚠️
internal/core/src/monitor/jemalloc_stats_c.cpp	0.00%	35 Missing ⚠️
internal/core/src/storage/MemFileManagerImpl.cpp	27.50%	29 Missing ⚠️
internal/coordinator/mix_coord.go	45.00%	19 Missing and 3 partials ⚠️
... and 51 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #47108      +/-   ##
==========================================
+ Coverage   76.09%   76.71%   +0.61%     
==========================================
  Files        1884     1920      +36     
  Lines      294531   300333    +5802     
==========================================
+ Hits       224120   230387    +6267     
+ Misses      63009    62511     -498     
- Partials     7402     7435      +33     

Components	Coverage Δ
Client	78.17% <78.78%> (+<0.01%)	⬆️
Core	82.58% <82.25%> (-0.15%)	⬇️
Go	75.50% <50.28%> (+1.32%)	⬆️

Files with missing lines	Coverage Δ
client/entity/field.go	58.88% <100.00%> (ø)
client/entity/schema.go	100.00% <100.00%> (ø)
client/milvusclient/read_options.go	77.82% <100.00%> (+0.20%)	⬆️
...replication/replicatemanager/channel_replicator.go	76.42% <100.00%> (+0.19%)	⬆️
...nternal/cdc/replication/replicatestream/metrics.go	82.43% <100.00%> (+4.93%)	⬆️
internal/compaction/params.go	94.44% <100.00%> (-0.39%)	⬇️
internal/core/src/common/Chunk.h	75.86% <100.00%> (ø)
internal/core/src/common/ChunkTarget.cpp	100.00% <100.00%> (ø)
internal/core/src/common/ChunkTarget.h	100.00% <100.00%> (ø)
internal/core/src/common/ChunkWriter.h	100.00% <100.00%> (ø)
... and 195 more

... and 600 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[gemini-code-assist]

Summary of Changes

Hello @lmzzzzz1, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request delivers a comprehensive set of updates aimed at improving the system's foundation, build processes, and client-facing capabilities. It ensures compatibility with newer Go versions, refines CI/CD pipelines for greater stability, and introduces new data handling features in the client SDK. The underlying C++ core benefits from optimized memory management, more precise JSON query processing, and a robust cancellation framework, all contributing to a more efficient and reliable system.

Highlights

• Go Version Update: The Go version used across various Dockerfiles, .env configurations, and client SDK files has been updated from 1.24.4 to 1.24.11.
• GitHub Actions Cache Improvement: The caching logic for macOS GitHub Actions workflows has been refined. Instead of relying on hashFiles, a more robust method using find and shasum is now employed to calculate hashes for ccache, go mod, and conan dependencies, ensuring better compatibility and reliability.
• Mergify Configuration Enhancement: Mergify workflows have been updated to incorporate a new WHEN_CODE_CHECK_MACOS_SUCCESS condition, ensuring that macOS code checks are properly considered in pull request merge requirements.
• Makefile and Build Process Refinements: The Makefile now includes conditional logic to disable use_disk_index on macOS due to aio limitations and sets planparser_rpath_flag accordingly. golangci-lint auto-fix targets have been made more explicit, and a new plan-parser-so target has been introduced to build a shared library for the plan parser, integrating it into C++ build targets. Mock generation for proxy has also been streamlined to use a dedicated YAML configuration.
• Docker Image and Dependency Updates: Various Dockerfiles now include tzdata for consistent timezone handling. The Kafka image has been switched to bitnamilegacy/kafka:3.1.0, and Milvus and Etcd image versions have been updated across deployment configurations.
• Client SDK Feature Expansion (Go): The Go client SDK now supports FieldTypeStruct and new MaxSim metric types. A QueryIterator has been implemented to enable paginated query results, including primary key-based pagination logic. The SDK version has been updated to 2.6.2.
• C++ Core Logic Enhancements: Significant improvements have been made to the C++ core, including better memory management for mmap regions using ChunkMmapGuard, enhanced JSON field querying with improved type handling and Exists() checks, and robust cancellation mechanisms integrated into query execution operators using folly::CancellationToken.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/mac.yaml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request primarily focuses on updating Go and Etcd versions across various Dockerfiles and Jenkins configurations, along with significant enhancements to the Go client SDK to support StructArrayField and introduce a new QueryIterator for paginated query results. It also refines GitHub Actions caching for macOS builds, updates Mergify rules to incorporate new macOS code checks, and modifies Makefile targets for C++ builds to include a new plan-parser-so dependency. Logging configurations are adjusted, and several Milvus configuration parameters are updated, including new MaxSim metric types. A notable change in the C++ core involves refactoring ChunkWriter to support ChunkMmapGuard for memory-mapped files and fixing a bug in VectorArrayChunkWriter related to sliced Arrow ListArrays. Additionally, the RTreeIndex now supports building and loading from growing segments, and the JsonFlatIndex has improved handling for JSON path parsing and mixed numeric types in range queries. The CDC module's controller logic has been refactored to use an event-driven model for replication management, and the GetReplicateInfo API now takes a request object instead of a simple string.

[gemini-code-assist]

The plan-parser-so target is quite complex, involving multiple steps to build a C++ shared library from Go source code. While the commands themselves seem correct, adding comments to explain the purpose of each step would greatly improve its readability and maintainability for future developers. For example, explaining why libmilvus-planparser.so is built from Go first, and then a C++ wrapper libmilvus-planparser-cpp.so is built linking against it.

plan-parser-so:
	@echo "Building plan parser shared library ..."
	@source $(PWD)/scripts/setenv.sh && \
		# Create output directories for the shared library and headers
		mkdir -p $(PWD)/internal/core/output/lib $(PWD)/internal/core/output/include && \
		# Enable CGO for building shared library from Go
		go env -w CGO_ENABLED="1" && \
		# Build the Go code as a C-style shared library (.so)
		GO111MODULE=on $(GO) build -buildmode=c-shared -o $(PWD)/internal/core/output/lib/libmilvus-planparser.so $(PWD)/internal/parser/planparserv2/cwrapper/wrapper.go && \
		# Move the auto-generated C header to the include directory
		mv $(PWD)/internal/core/output/lib/libmilvus-planparser.h $(PWD)/internal/core/output/include/libmilvus-planparser.h && \
		# Copy the manually-written C++ wrapper header
		cp $(PWD)/internal/parser/planparserv2/cwrapper/milvus_plan_parser.h $(PWD)/internal/core/output/include/ && \
		# Compile the C++ wrapper, linking against the Go-generated shared library
		g++ -std=c++17 -shared -fPIC -o $(PWD)/internal/core/output/lib/libmilvus-planparser-cpp.so $(PWD)/internal/parser/planparserv2/cwrapper/milvus_plan_parser.cpp \
			-I$(PWD)/internal/core/output/include \
			-L$(PWD)/internal/core/output/lib -lmilvus-planparser \
			$(planparser_rpath_flag)