Skip to content

feat(query): sink fuse table process in multiple threads #9141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

feat(query): sink fuse table process in multiple threads #9141

wants to merge 1 commit into from

Conversation

b41sh
Copy link
Member

@b41sh b41sh commented Dec 8, 2022

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Build BloomIndexState and BlockStatistics in separate threads. After optimization, the time of inserting 100M values will be reduced by about 6s.

Closes #9089

@b41sh b41sh requested review from sundy-li and dantengsky December 8, 2022 03:31
@vercel
Copy link

vercel bot commented Dec 8, 2022 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Updated
databend ⬜️ Ignored (Inspect) Dec 8, 2022 at 3:31AM (UTC)

@mergify
Copy link
Contributor

mergify bot commented Dec 8, 2022

This pull request's title is not fulfill the requirements. @b41sh please update it 🙏.

Valid format:

fix(query): fix group by string bug
  ^         ^---------------------^
  |         |
  |         +-> Summary in present tense.
  |
  +-------> Type: rfc, feat, fix, refactor, ci, docs, chore

Valid types:

  • rfc: this PR proposes a new RFC
  • feat: this PR introduces a new feature to the codebase
  • fix: this PR patches a bug in codebase
  • refactor: this PR changes the code base without new features or bugfix
  • ci: this PR changes build/testing/ci steps
  • docs: this PR changes the documents or websites
  • chore: this PR only has small changes that no need to record

@b41sh b41sh changed the title performance(query): sink fuse table process in multiple threads chore(query): sink fuse table process in multiple threads Dec 8, 2022
@mergify mergify bot added the pr-chore this PR only has small changes that no need to record, like coding styles. label Dec 8, 2022
@BohuTANG BohuTANG changed the title chore(query): sink fuse table process in multiple threads performance(query): sink fuse table process in multiple threads Dec 8, 2022
@mergify
Copy link
Contributor

mergify bot commented Dec 8, 2022

This pull request's title is not fulfill the requirements. @b41sh please update it 🙏.

Valid format:

fix(query): fix group by string bug
  ^         ^---------------------^
  |         |
  |         +-> Summary in present tense.
  |
  +-------> Type: rfc, feat, fix, refactor, ci, docs, chore

Valid types:

  • rfc: this PR proposes a new RFC
  • feat: this PR introduces a new feature to the codebase
  • fix: this PR patches a bug in codebase
  • refactor: this PR changes the code base without new features or bugfix
  • ci: this PR changes build/testing/ci steps
  • docs: this PR changes the documents or websites
  • chore: this PR only has small changes that no need to record

@BohuTANG BohuTANG changed the title performance(query): sink fuse table process in multiple threads feat(query): sink fuse table process in multiple threads Dec 8, 2022
@BohuTANG
Copy link
Member

BohuTANG commented Dec 8, 2022

I have updated the title, avoid to use chore, it will not show in the changelog :)

@mergify mergify bot added the pr-feature this PR introduces a new feature to the codebase label Dec 8, 2022
@b41sh
Copy link
Member Author

b41sh commented Dec 8, 2022

Since the pipelines are multi-threaded, adding multiple threads here may cause additional performance loss, so close this PR first.

@b41sh b41sh closed this Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sundy-li sundy-li Awaiting requested review from sundy-li

@dantengsky dantengsky Awaiting requested review from dantengsky

Assignees
No one assigned
Labels
pr-chore this PR only has small changes that no need to record, like coding styles. pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

performance: build BlockFilter and BlockStatistics in parallel
2 participants
@b41sh @BohuTANG