Skip to content

[Fix] Fix clickhouse local generate file has <0x00> (#6089) #6136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from

Conversation

zhanggougou
Copy link

Purpose of this pull request

fix clickhouse local generate file has <0x00>,it will cause

Does this PR introduce any user-facing change?

no

How was this patch tested?

after fix,i success transfer 300KW records from hive to ck

Check list

@zhanggougou zhanggougou closed this Jan 5, 2024
@zhanggougou zhanggougou reopened this Jan 5, 2024
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @zhanggougou , thanks for contribute this. Could you explain the reason of change?

@zhanggougou
Copy link
Author

Hi @zhanggougou , thanks for contribute this. Could you explain the reason of change?

you can see this issue #6089
when 【buffer.position() + byteData.length > buffer.capacity()】,a new MappedByteBuffer will generate,but last MappedByteBuffer also has some space ,because the bufferSize is fixed

so i compute fileChannel.position,when buffer is not enough,the next will continue previous position,so the local_data will not contains unnecessary <0x00>

Is my problem?but when i fixed it,i succesed transfer data 【OR higher version ck local can ignore <0x00>?】

Looking forward to your reply,thanks

@EricJoy2048
Copy link
Member

Can you add e2e to test this pr?

@zhanggougou
Copy link
Author

Can you add e2e to test this pr?

sorry,i am a newcomer in this project,e2e means?

now,i suspect if this is a bug,give me some time,i will transfer data again to test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants