Releases: GoogleCloudPlatform/DataflowTemplates
Dataflow Templates 2022-07-12-00_RC00
Release Week of 2022-07-11
Bug Fixes
- [Dataplex Templates] Bugfix: set partition location to the generated file's parent directory instead of the full file path when updating Dataplex metadata.
Other
- Added jacoco plugin to the project for unit test coverage tracking.
Contributors
Contributors
Assets 2
Dataflow Templates 2022-07-04-00_RC00
Release Week of July 4, 2022
New Templates
[Spanner to BigQuery] Initial template creation for Spanner -> BigQuery
Improvements
[Spanner change streams to BigQuery] Respect the primary key ordinal position when doing stale read for UPDATE mod
Contributors
Contributors
Assets 2
Dataflow Templates 2022-06-27-00_RC00
Release Week of 2022-06-27
Improvements
[PubsubToText] Add inputSubscription parameter
[DataplexBigQueryToGcs] Support updating metadata in Dataplex for newly generated data.
Contributors
@andreigurau
@oleg-semenov
@an2x
@oleg-semenov
@pranavbhandari24
Contributors
Assets 2
Dataflow Templates 2022-06-21-00_RC00
Release Week of 2022-06-20
Improvements
[File Format Conversion Template] New logDetailedCsvConversionErrors parameter.
[All JDBC templates] New extraFilesToStage parameter.
[All Datastream templates ] Use Datastream V1 client.
[All Flex Templates] New disabledAlgorithms and extraFilesToStage parameters.
[DatastreamToBigQuery] Merge sort key checks by all sort keys not only timestamp.
[All Spanner templates] Support change streams in RandomDdlGenerator used for import/export tests.
Contributors
Contributors
Assets 2
Dataflow Templates 2022-06-06-00_RC00
Release Week of 2022-06-06
Improvements
[DatastreamToBigQuery Template]: Added logic to clean Oracle rollbacks without tx data after ~24 hours.
[DatastreamToBigQuery Template]: Convert dot (invalid BigQuery character) in table name and schema name into a single underscore.
Bug Fixes
Minor refactoring and fixes in unit tests.
Contributors
Contributors
Assets 2
Dataflow Templates 2022-05-30-00_RC00
Contributors
Assets 2
Dataflow Templates 2022-05-23-00_RC01
Release Week of 2022-05-23
Improvements
[SpannerChangeStreamsToBigQuery] Upgrade google-cloud-bigquery from 1.128.0 to 2.9.0, and fix JSON support for Dataflow Spanner change stream to BigQuery template
Bug Fixes
[SpannerChangeStreamsToBigQuery] Fix naming of Spanner change stream to BigQuery Dataflow template
[SpannerChangeStreamsToBigQuery] Two fixes for Spanner change stream to BigQuery template.
Contributors
Contributors
Assets 2
Dataflow Templates 2022-05-16-00_RC00
Release Week of 2022-05-16
Improvements
[SpannerChangeStreamsToGcs] Update Spanner to GCS template to include Spanner metadata table parameter
[Spanner templates] Support for Automatic Dialect Detection
[Spanner templates] Support Spanner's PG interface
[Spanner templates] Support for Cloud Spanner Check Constraints, Views and Foreign Keys in PostgreSQL Interface
[Spanner templates] Support change streams in Cloud Spanner Import/Export pipelines
[All Elastic search templates] Add support for Elasticsearch 8
[Documentation] Add instructions for using Maven
Bug Fixes
[Datastream to SQL] clean empty or empty string numeric data to null for all numeric JDBC data types
[All BigQuery templates] JSON to table row exceptions handling fix
Contributors
Cherepushko
@dhercher
@pranavbhandari24
@zhoufek
Contributors
Assets 2
Dataflow Templates 2022-05-09-00_RC01
Release Week of 2022-05-09
Bug Fixes
[All Flex Templates] Resolve dependency conflicts.
[SpannerChangeStreams To BigQuery] Bug fixes.
Contributors
Contributors
Assets 2
Dataflow Templates 2022-05-04-00_RC00
Release Week of 2022-05-02
Improvements
[All Templates] Upgraded from Beam 2.37 to 2.38.
[Cloud Spanner Change Streams Template] Added ignoreFields param and more unit tests.
[JDBC to Pub/Sub Template] Javadoc updates.
Upgraded io.opencensus.opencensus-api to the latest version 0.31.0.
Bug Fixes
[All Spanner Templates] Minor bugfixes to support Beam 2.38.