[BEAM-6394] Add support to write protobuf data using ProtoParquetReader #34063
Conversation
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
assign set of reviewers |
|
Assigning reviewers. If you would like to opt out of this review, comment R: @kennknowles for label java. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
@Abacn Could you please take a look? Thanks! |
| @@ -616,6 +621,9 @@ public abstract static class ReadFiles | |||
|
|
|||
| abstract Builder toBuilder(); | |||
|
|
|||
| // New: flag to indicate using ProtoParquetReader for protobuf data. | |||
| abstract boolean getUseProtoReader(); | |||
There was a problem hiding this comment.
consider using a enum type parameter (AVRO / PROTO) and defaults to AVRO? This allows flexibility for future additions.
|
|
||
| // Get the configuration and set the property using the literal. | ||
| Configuration conf2 = getConfWithModelClass(); | ||
| conf2.set("parquet.proto.ignore.unknown.fields", "TRUE"); |
There was a problem hiding this comment.
What is the purpose of "ignore.unknown.fields" here? In general we should not set unsafe configure for user if they are not aware of
| } | ||
| } | ||
|
|
||
| return; // exit after using the proto path |
There was a problem hiding this comment.
For readability, I would recommend structure the code, e.g.
getParquetFileReader() {
switch (FORMAT) {
case AVRO: return getParquetAvroReader()
case PROTO: return getParquetProtoReader()
}
}
instead of branching in place
|
Reminder, please take a look at this pr: @kennknowles @johnjcasey |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @robertwb for label java. Available commands:
|
|
waiting on author |
github-actions
bot
added
Next Action: Author
and removed
Next Action: Reviewers
slow-review
labels
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.