Dataflow Templates 2022-12-05-00_RC00

Release Week of 2022-12-05

Improvements

• [All templates] Introduce metadata annotations.
• [DataStreamToBigQuery] Expose mergeConcurrency option and re-throw error on merge statement fail.
• Trigger Java PR workflow when any XML is changed + files are deleted.
• [Classic templates] Support JSONB arrays.
• [PubSubCdcToBigQuery] Support maxStreamingBatchSize parameter
• [DatastreamToSpanner] Add changes for the new HarbourBridge session file with tableID and columnID support.
• [All templates] Upgrade Beam version to 2.43.
• [Classic templates] Prepare plugin infra to test classic templates + Create BulkCompressionIT
• [Integration Tests] Do not make artifactBucket mandatory (only if bucketName not provided for ITs)
• [DataStreamToSpanner] Change default values for dlqRetryMinutes and dlqMaxRetryCount params.
• [Integration Tests] Avoid Joiner conflict, and improve plugin staging speed
• [Flex templates] Plain text logging for Flex Templates unit tests
• [Integration Tests] Improve plugin bucket parameter requirements
• [SpannerChangeStreamsTemplates] Simplify the code of setting experiments for spanner change streams to BigQuery and spanner change streams to GCS templates.
• [Integration Tests] Create MongoDB Resource Manager
• [Integration Tests] Create MongoDBToBigQueryIntegrationTest
• [Integration Tests] Add TestContainers framework
• [Integration Tests] Create PubsubAvroToBigQueryIT + prepare profile to run integration tests together
• [MongoDBToBigQuery] Create udf for MongoDB BigQuery Templates
• [Security] Update hadoop version affected by CVE-2022-25168
• Improve Templates Plugin instructions
• [Syndeo templates] Separating JSON build

Bug Fixes

• [Classic templates] WindowedFilenamePolicy's dayPattern defaults to dd instead of DD
• [JDBC templates] Do not log unencrypted values/keys to the console
• [DataStream templates] Rethrow exception from ExtractGcsFile so that Dataflow will retry the pardo
• [Integration Tests] Fix integration tests parameters passing
• [Flex templates] Fix log dependencies (log4j initialization error)

Contributors

@bvolpato
@oleg-semenov
@pabloem
@Polber
@pranavbhandari24
@theshanbhag