codec(ticdc): support header line for CSV protocol (#12183) #12433
Conversation
ti-chi-bot
added
affect-ticdc-config-docs
|
This cherry pick PR is for a release branch and has not yet been approved by triage owners. To merge this cherry pick:
DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Summary of ChangesHello @ti-chi-bot, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances TiCDC's CSV protocol by adding support for an optional header line in the generated CSV files. This feature allows users to include column names as the first row, improving the readability and interoperability of the CSV output with various data processing tools. The changes involve exposing a new configuration parameter, implementing the logic to generate and write the header, and updating the CSV decoder to handle header-present files. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for header lines in the CSV protocol, which is a valuable feature. The implementation is mostly well-structured, propagating the new OutputFieldHeader configuration and adding logic to both encode and decode the headers. However, I've identified a few issues that should be addressed:
- The CSV decoder uses
log.Panicon a header mismatch, which could crash the TiCDC process. This should be changed to return an error for graceful failure. - The header encoding logic does not account for the existing
OutputHandleKeyconfiguration, which will cause a mismatch between header and data columns when it's enabled. - The integration test added to verify the header has a check that is not strict enough and will not fail the test upon header mismatch.
I've left specific comments with suggestions for each of these points. Once these are addressed, the PR should be in good shape.
pkg/sink/codec/csv/csv_decoder.go
Outdated
| if col.Name.O != header[idx+i] { | ||
| log.Panic("check column name order failed", zap.Any("col", col.Name.O), zap.Any("header", header[idx+i])) | ||
| } |
There was a problem hiding this comment.
Using log.Panic for a data validation error like a CSV header mismatch is too aggressive, as it will crash the entire capture process. It's better to return an error to allow for graceful error handling, such as putting the changefeed into an error state.
if col.Name.O != header[idx+i] {
return nil, cerror.WrapError(cerror.ErrCSVDecodeFailed,
errors.Errorf("csv header column name mismatch, expected: %s, got: %s", col.Name.O, header[idx+i]))
}| } | ||
| for _, name := range colNames { |
There was a problem hiding this comment.
The encodeHeader function does not account for the OutputHandleKey configuration. When OutputHandleKey is true, encodeMeta adds the handle key as a column, but there is no corresponding header column. This will cause a mismatch between the header and the data rows. A header for the handle key should be added, for instance ticdc-meta$handle-key.
}
if config.OutputHandleKey {
strBuilder.WriteString(config.Delimiter)
strBuilder.WriteString("ticdc-meta$handle-key")
}
for _, name := range colNames {| buf := bytes.NewBuffer(make([]byte, 0, task.size)) | ||
| rowsCnt := 0 | ||
| bytesCnt := int64(0) | ||
| // There is always only one message here in task.msgs |
There was a problem hiding this comment.
The comment // There is always only one message here in task.msgs is an important assumption. While it might be true for the current implementation, it could become a source of bugs if the logic for batching messages into tasks changes in the future. If a task could contain multiple messages, this logic would only write the header for the very first message in the task, potentially missing headers for subsequent messages if they were intended to start new files. Consider adding a more detailed explanation or a link to where this assumption is guaranteed.
| if [[ "$first_line" != ticdc-meta* ]]; then | ||
| echo "check CSV header failed. header: $first_line" | ||
| fi |
There was a problem hiding this comment.
The check for the CSV header is not strict enough. It only echoes a message on failure and does not cause the script to exit with an error. This could lead to test failures being missed. The check should be made stricter to exit on failure. Also, it's good practice to quote file paths (e.g., "$file" in head) to handle potential spaces or special characters.
| if [[ "$first_line" != ticdc-meta* ]]; then | |
| echo "check CSV header failed. header: $first_line" | |
| fi | |
| if [[ "$first_line" != ticdc-meta* ]]; then | |
| echo "check CSV header failed. header: $first_line" | |
| exit 1 | |
| fi |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: wk989898 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
Flags with carried forward coverage won't be shown. Click here to find out more. @@ Coverage Diff @@
## release-8.5 #12433 +/- ##
================================================
Coverage ? 53.2054%
================================================
Files ? 213
Lines ? 17720
Branches ? 0
================================================
Hits ? 9428
Misses ? 7689
Partials ? 603 🚀 New features to boost your workflow:
|
|
@ti-chi-bot: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
This is an automated cherry-pick of #12183
What problem does this PR solve?
Issue Number: close #12179
What is changed and how it works?
OutputFieldHeader.message.KeywhenOutputFieldHeader=truesincemessage.Keyis never used before.Header
Check List
Tests
Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note