Ldz/improve event store1015

[lidezhu]

What problem does this PR solve?

Issue Number: close #xxx

What is changed and how it works?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

[ti-chi-bot]

[FORMAT CHECKER NOTIFICATION]

Notice: To remove the do-not-merge/needs-linked-issue label, please provide the linked issue number on one line in the PR body, for example: Issue Number: close #123 or Issue Number: ref #456.

_{📖 For more info, you can check the "Contribute Code" section in the development guide.}

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign sdojjy for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gemini-code-assist]

Summary of Changes

Hello @lidezhu, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request primarily focuses on enhancing the performance and efficiency of the event store. It achieves this by significantly increasing the concurrency of write operations and implementing a more sophisticated event batching strategy that considers both the count and total size of events. Additionally, it augments event metadata with key and value lengths, which could be beneficial for monitoring or further optimizations.

Highlights

• Increased Write Concurrency: The number of write workers per database in the event store has been quadrupled from 2 to 8, aiming to significantly improve write throughput and parallel processing of events.
• Enhanced Event Batching Strategy: New constants batchEventMaxCount and batchEventMaxSize have been introduced to define more granular limits for event batching. The event retrieval mechanism now considers both the maximum number of events and the maximum total size of events when fetching from the channel, leading to more efficient and controlled batch processing.
• Added Key/Value Length Metadata: The RawKVEntry structure now includes KeyLen, ValueLen, and OldValueLen fields. These new fields provide explicit size information for the key, value, and old value of KV events, which can be useful for monitoring, debugging, or further optimizations.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces performance improvements to the event store. Key changes include increasing the number of write workers per database and adding batch size limits (batchEventMaxCount and batchEventMaxSize) to optimize write operations. Additionally, KeyLen, ValueLen, and OldValueLen fields are now populated in RawKVEntry to potentially improve downstream processing efficiency. The changes are well-reasoned and should enhance performance. I have a couple of minor suggestions to improve code style and consistency.

[gemini-code-assist]

In Go, it's idiomatic to declare constants without an explicit type when the type can be inferred from the value. This makes them untyped constants, which are more flexible. Consider removing the explicit int type.

Suggested change

	batchEventMaxCount int = 128
	batchEventMaxSize int = 64 * 1024
	batchEventMaxCount = 128
	batchEventMaxSize = 64 * 1024

[gemini-code-assist]

To improve consistency and avoid a redundant method call, you can store entry.Key in a variable and use it for both calculating the length and assigning the value. This aligns with how value and oldValue are handled.

key := entry.Key
value := entry.GetValue()
oldValue := entry.GetOldValue()
return common.RawKVEntry{
	OpType:      opType,
	KeyLen:      uint32(len(key)),
	ValueLen:    uint32(len(value)),
	OldValueLen: uint32(len(oldValue)),
	Key:         key,
	Value:       value,
	StartTs:     entry.StartTs,
	CRTs:        entry.CommitTs,
	RegionID:    regionID,
	OldValue:    oldValue,
}