Experimental PQ reader utility to calculate total rows in input row groups #18716
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
mhaseeb123
changed the title
mhaseeb123
changed the title
mhaseeb123
added
feature request
mhaseeb123
commented
May 8, 2025
| .stats_level(cudf::io::statistics_freq::STATISTICS_COLUMN); | ||
|
|
||
| if constexpr (NumTableConcats > 1) { | ||
| out_opts.set_row_group_size_rows(20000); | ||
| out_opts.set_max_page_size_rows(5000); | ||
| out_opts.set_row_group_size_rows(num_ordered_rows); |
There was a problem hiding this comment.
Use constexpr variables from header instead of magic numbers to avoid problems false positive failures later on
There was a problem hiding this comment.
what problems do you expect later?
There was a problem hiding this comment.
Like if the value of num_ordered_rows changes in the header for some reason, which is used to generate columns, the test may fail due to magic numbers being used instead of the same constexprs
auto col0 = testdata::ascending<uint32_t>();
auto col1 = testdata::descending<int64_t>();
auto col2 = testdata::ascending<cudf::string_view>();| .row_group_size_rows(page_size_for_ordered_tests) | ||
| .max_page_size_rows(page_size_for_ordered_tests / 5) | ||
| .compression(compression) | ||
| .dictionary_policy(cudf::io::dictionary_policy::ALWAYS) |
There was a problem hiding this comment.
Always try to write dictionaries to test dictionary page pruning in future
vuule
reviewed
May 8, 2025
ttnghia
reviewed
May 8, 2025
3 tasks
vuule
approved these changes
May 9, 2025
davidwendt
reviewed
May 9, 2025
davidwendt
approved these changes
May 9, 2025
ttnghia
approved these changes
May 9, 2025
Co-authored-by: David Wendt <[email protected]>
mhaseeb123
added
5 - Ready to Merge
|
/merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a utility function to the experimental Parquet reader to calculate the total number of rows in the input list of parquet row groups.
Checklist