Skip to content

e2e Spanner additional test scenarios #976

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

e2e Spanner additional test scenarios #976

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions src/e2e-test/features/spanner/sink/BigQueryToSpanner_WithMacro.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
@Spanner_Sink @SPANNER_TEST
Feature: Spanner Sink - Verification of BigQuery to Spanner successful data transfer with macro arguments

@BQ_SOURCE_TEST @SPANNER_SINK_TEST
Scenario:Validate successful records transfer from BigQuery to Spanner with macro arguments
Given Open Datafusion Project to configure pipeline
When Source is BigQuery
When Sink is Spanner
Then Open BigQuery source properties
Then Enter BigQuery property reference name
Then Enter BigQuery property "projectId" as macro argument "bqProjectId"
Then Enter BigQuery property "datasetProjectId" as macro argument "bqDatasetProjectId"
Then Enter BigQuery property "serviceAccountType" as macro argument "serviceAccountType"
Then Enter BigQuery property "serviceAccountFilePath" as macro argument "serviceAccount"
Then Enter BigQuery property "serviceAccountJSON" as macro argument "serviceAccount"
Then Enter BigQuery property "dataset" as macro argument "bqDataset"
Then Enter BigQuery property "table" as macro argument "bqSourceTable"
Then Validate "BigQuery" plugin properties
Then Close the BigQuery properties
Then Open Spanner sink properties
Then Enter Spanner property reference name
Then Enter Spanner property "projectId" as macro argument "spannerProjectId"
Then Enter Spanner property "serviceAccountType" as macro argument "serviceAccountType"
Then Enter Spanner property "serviceAccountFilePath" as macro argument "serviceAccount"
Then Enter Spanner property "serviceAccountJSON" as macro argument "serviceAccount"
Then Enter Spanner property "instanceId" as macro argument "spannerInstanceId"
Then Enter Spanner property "databaseName" as macro argument "spannerDatabaseName"
Then Enter Spanner property "table" as macro argument "spannerTablename"
Then Enter Spanner property "primaryKey" as macro argument "spannerSinkPrimaryKey"
Then Enter Spanner cmek property "encryptionKeyName" as macro argument "cmekSpanner" if cmek is enabled
Then Validate "Spanner" plugin properties
Then Close the Spanner properties
Then Connect source as "BigQuery" and sink as "Spanner" to establish connection
Then Save the pipeline
Then Preview and run the pipeline
Then Enter runtime argument value "projectId" for key "bqProjectId"
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
Then Enter runtime argument value "dataset" for key "bqDataset"
Then Enter runtime argument value for BigQuery source table name key "bqSourceTable"
Then Enter runtime argument value "projectId" for key "spannerProjectId"
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
Then Enter runtime argument value for Spanner Sink Table Name key "spannerTablename"
Then Enter runtime argument value "spannerSinkPrimaryKeyBQ" for key "spannerSinkPrimaryKey"
Then Enter runtime argument value "cmekSpanner" for Spanner cmek property key "cmekSpanner" if Spanner cmek is enabled
Then Run the preview of pipeline with runtime arguments
Then Verify the preview of pipeline is "success"
Then Close the preview
Then Deploy the pipeline
Then Run the Pipeline in Runtime
Then Enter runtime argument value "projectId" for key "bqProjectId"
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
Then Enter runtime argument value "dataset" for key "bqDataset"
Then Enter runtime argument value for BigQuery source table name key "bqSourceTable"
Then Enter runtime argument value "projectId" for key "spannerProjectId"
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
Then Enter runtime argument value for Spanner Sink Table Name key "spannerTablename"
Then Enter runtime argument value "spannerSinkPrimaryKeyBQ" for key "spannerSinkPrimaryKey"
Then Enter runtime argument value "cmekSpanner" for Spanner cmek property key "cmekSpanner" if Spanner cmek is enabled
Then Run the Pipeline in Runtime with runtime arguments
Then Wait till pipeline is in running state
Then Open and capture logs
Then Verify the pipeline status is "Succeeded"
Then Validate records transferred to target spanner table with record counts of BigQuery table
71 changes: 71 additions & 0 deletions src/e2e-test/features/spanner/sink/GCSToSpanner_WithMacro.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
@Spanner_Sink @SPANNER_TEST
Feature: Spanner Sink - Verification of GCS to Spanner successful data transfer with macro arguments

@GCS_CSV_TEST @SPANNER_SINK_TEST
Scenario:Validate successful records transfer from GCS to Spanner with macro arguments
Given Open Datafusion Project to configure pipeline
When Source is GCS
When Sink is Spanner
Then Open GCS source properties
Then Enter GCS property reference name
Then Enter GCS property "projectId" as macro argument "gcsProjectId"
Then Enter GCS property "serviceAccountType" as macro argument "serviceAccountType"
Then Enter GCS property "serviceAccountFilePath" as macro argument "serviceAccount"
Then Enter GCS property "serviceAccountJSON" as macro argument "serviceAccount"
Then Enter GCS property "path" as macro argument "gcsSourcePath"
Then Enter GCS property "format" as macro argument "gcsFormat"
Then Enter GCS source property "skipHeader" as macro argument "gcsSkipHeader"
Then Enter GCS source property output schema "outputSchema" as macro argument "gcsOutputSchema"
Then Validate "GCS" plugin properties
Then Close the GCS properties
Then Open Spanner sink properties
Then Enter Spanner property reference name
Then Enter Spanner property "projectId" as macro argument "spannerProjectId"
Then Enter Spanner property "serviceAccountType" as macro argument "serviceAccountType"
Then Enter Spanner property "serviceAccountFilePath" as macro argument "serviceAccount"
Then Enter Spanner property "serviceAccountJSON" as macro argument "serviceAccount"
Then Enter Spanner property "instanceId" as macro argument "spannerInstanceId"
Then Enter Spanner property "databaseName" as macro argument "spannerDatabaseName"
Then Enter Spanner property "table" as macro argument "spannerTablename"
Then Enter Spanner property "primaryKey" as macro argument "spannerSinkPrimaryKey"
Then Enter Spanner cmek property "encryptionKeyName" as macro argument "cmekSpanner" if cmek is enabled
Then Validate "Spanner" plugin properties
Then Close the Spanner properties
Then Connect source as "GCS" and sink as "Spanner" to establish connection
Then Save the pipeline
Then Preview and run the pipeline
Then Enter runtime argument value "projectId" for key "gcsProjectId"
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
Then Enter runtime argument value "gcsCsvFile" for GCS source property path key "gcsSourcePath"
Then Enter runtime argument value "gcsSkipHeaderTrue" for key "gcsSkipHeader"
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
Then Enter runtime argument value "gcsCSVFileOutputSchema" for key "gcsOutputSchema"
Then Enter runtime argument value "projectId" for key "spannerProjectId"
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
Then Enter runtime argument value for Spanner Sink Table Name key "spannerTablename"
Then Enter runtime argument value "spannerSinkPrimaryKeyGCS" for key "spannerSinkPrimaryKey"
Then Enter runtime argument value "cmekSpanner" for Spanner cmek property key "cmekSpanner" if Spanner cmek is enabled
Then Run the preview of pipeline with runtime arguments
Then Verify the preview of pipeline is "success"
Then Close the preview
Then Deploy the pipeline
Then Run the Pipeline in Runtime
Then Enter runtime argument value "projectId" for key "gcsProjectId"
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
Then Enter runtime argument value "gcsCsvFile" for GCS source property path key "gcsSourcePath"
Then Enter runtime argument value "gcsSkipHeaderTrue" for key "gcsSkipHeader"
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
Then Enter runtime argument value "gcsCSVFileOutputSchema" for key "gcsOutputSchema"
Then Enter runtime argument value "projectId" for key "spannerProjectId"
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
Then Enter runtime argument value for Spanner Sink Table Name key "spannerTablename"
Then Enter runtime argument value "spannerSinkPrimaryKeyGCS" for key "spannerSinkPrimaryKey"
Then Enter runtime argument value "cmekSpanner" for Spanner cmek property key "cmekSpanner" if Spanner cmek is enabled
Then Run the Pipeline in Runtime with runtime arguments
Then Wait till pipeline is in running state
Then Open and capture logs
Then Verify the pipeline status is "Succeeded"
39 changes: 39 additions & 0 deletions src/e2e-test/features/spanner/sink/SpannerToSpanner.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
@Spanner_Sink @SPANNER_TEST
Feature: Spanner Sink - Verification of Spanner to Spanner data transfer

@SPANNER_SINK_TEST
Scenario: To verify data is getting transferred successfully from Spanner to Spanner with all supported datatype
Given Open Datafusion Project to configure pipeline
When Source is Spanner
When Sink is Spanner
Then Connect source as "Spanner" and sink as "Spanner" to establish connection
Then Open Spanner source properties
Then Enter Spanner property reference name
Then Enter Spanner property projectId "projectId"
Then Override Service account details if set in environment variables
Then Enter Spanner property InstanceId
Then Enter Spanner source property DatabaseName
Then Enter Spanner source property TableName
Then Validate output schema with expectedSchema "spannerSourceSchema"
Then Validate "Spanner" plugin properties
Then Close the Spanner properties
Then Open Spanner sink properties
Then Enter Spanner property reference name
Then Enter Spanner property projectId "projectId"
Then Override Service account details if set in environment variables
Then Enter Spanner property InstanceId
Then Enter Spanner sink property DatabaseName
Then Enter Spanner sink property TableName
Then Enter Spanner sink property primary key "spannerSinkPrimaryKeySpanner"
Then Validate "Spanner" plugin properties
Then Close the Spanner properties
Then Save the pipeline
Then Preview and run the pipeline
Then Verify the preview of pipeline is "success"
Then Close the preview
Then Deploy the pipeline
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Then Open and capture logs
Then Verify the pipeline status is "Succeeded"
Then Validate records transferred to target spanner table with record counts of source spanner table
69 changes: 69 additions & 0 deletions src/e2e-test/features/spanner/source/SpannerToBigQuery_WithMacro.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
@Spanner_Source @SPANNER_TEST
Feature: Spanner Source - Verification of Spanner to BigQuery successful data transfer with macro arguments

@BQ_SINK_TEST
Scenario:Validate successful records transfer from Spanner to BigQuery with macro arguments
Given Open Datafusion Project to configure pipeline
When Source is Spanner
When Sink is BigQuery
Then Open Spanner source properties
Then Enter Spanner property reference name
Then Enter Spanner property "projectId" as macro argument "spannerProjectId"
Then Enter Spanner property "serviceAccountType" as macro argument "serviceAccountType"
Then Enter Spanner property "serviceAccountFilePath" as macro argument "serviceAccount"
Then Enter Spanner property "serviceAccountJSON" as macro argument "serviceAccount"
Then Enter Spanner property "instanceId" as macro argument "spannerInstanceId"
Then Enter Spanner property "databaseName" as macro argument "spannerDatabaseName"
Then Enter Spanner property "table" as macro argument "spannerTablename"
Then Validate "Spanner" plugin properties
Then Close the Spanner properties
Then Open BigQuery sink properties
Then Enter BigQuery property reference name
Then Enter BigQuery property "projectId" as macro argument "bqProjectId"
Then Enter BigQuery property "datasetProjectId" as macro argument "bqDatasetProjectId"
Then Enter BigQuery property "serviceAccountType" as macro argument "serviceAccountType"
Then Enter BigQuery property "serviceAccountFilePath" as macro argument "serviceAccount"
Then Enter BigQuery property "serviceAccountJSON" as macro argument "serviceAccount"
Then Enter BigQuery property "dataset" as macro argument "bqDataset"
Then Enter BigQuery property "table" as macro argument "bqTargetTable"
Then Enter BigQuery sink property "truncateTable" as macro argument "bqTruncateTable"
Then Enter BigQuery sink property "updateTableSchema" as macro argument "bqUpdateTableSchema"
Then Validate "BigQuery" plugin properties
Then Close the BigQuery properties
Then Connect source as "Spanner" and sink as "BigQuery" to establish connection
Then Save the pipeline
Then Preview and run the pipeline
Then Enter runtime argument value "projectId" for key "spannerProjectId"
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
Then Enter runtime argument value for Spanner Source Table Name key "spannerTablename"
Then Enter runtime argument value "projectId" for key "bqProjectId"
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
Then Enter runtime argument value "dataset" for key "bqDataset"
Then Enter runtime argument value for BigQuery sink table name key "bqTargetTable"
Then Enter runtime argument value "bqTruncateTableTrue" for key "bqTruncateTable"
Then Enter runtime argument value "bqUpdateTableSchemaTrue" for key "bqUpdateTableSchema"
Then Run the preview of pipeline with runtime arguments
Then Verify the preview of pipeline is "success"
Then Close the preview
Then Deploy the pipeline
Then Run the Pipeline in Runtime
Then Enter runtime argument value "projectId" for key "spannerProjectId"
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
Then Enter runtime argument value for Spanner Source Table Name key "spannerTablename"
Then Enter runtime argument value "projectId" for key "bqProjectId"
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
Then Enter runtime argument value "dataset" for key "bqDataset"
Then Enter runtime argument value for BigQuery sink table name key "bqTargetTable"
Then Enter runtime argument value "bqTruncateTableTrue" for key "bqTruncateTable"
Then Enter runtime argument value "bqUpdateTableSchemaTrue" for key "bqUpdateTableSchema"
Then Run the Pipeline in Runtime with runtime arguments
Then Wait till pipeline is in running state
Then Open and capture logs
Then Verify the pipeline status is "Succeeded"
Then Validate records transferred to target BigQuery table with record counts of spanner table
Loading