Skip to content

Commit 6c1f631

Browse files
sawantpritamsawantpritam
committed
e2e Spanner additional test scenarios
1 parent 9174bff commit 6c1f631

10 files changed

+374
-5
lines changed

‎src/e2e-test/features/spanner/sink/BigQueryToSpanner_WithMacro.feature

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
@Spanner_Sink @SPANNER_TEST
2+
Feature: Spanner Sink - Verification of BigQuery to Spanner successful data transfer with macro arguments
3+
4+
@BQ_SOURCE_TEST @SPANNER_SINK_TEST
5+
Scenario:Validate successful records transfer from BigQuery to Spanner with macro arguments
6+
Given Open Datafusion Project to configure pipeline
7+
When Source is BigQuery
8+
When Sink is Spanner
9+
Then Open BigQuery source properties
10+
Then Enter BigQuery property reference name
11+
Then Enter BigQuery property "projectId" as macro argument "bqProjectId"
12+
Then Enter BigQuery property "datasetProjectId" as macro argument "bqDatasetProjectId"
13+
Then Enter BigQuery property "serviceAccountFilePath" as macro argument "bqServiceAccount"
14+
Then Enter BigQuery property "dataset" as macro argument "bqDataset"
15+
Then Enter BigQuery property "table" as macro argument "bqSourceTable"
16+
Then Validate "BigQuery" plugin properties
17+
Then Close the BigQuery properties
18+
Then Open Spanner sink properties
19+
Then Enter Spanner property reference name
20+
Then Enter Spanner property "projectId" as macro argument "spannerProjectId"
21+
Then Enter Spanner property "serviceAccountFilePath" as macro argument "spannerServiceAccount"
22+
Then Enter Spanner property "instanceId" as macro argument "spannerInstanceId"
23+
Then Enter Spanner property "databaseName" as macro argument "spannerDatabaseName"
24+
Then Enter Spanner property "table" as macro argument "spannerTablename"
25+
Then Enter Spanner property "primaryKey" as macro argument "spannerSinkPrimaryKey"
26+
Then Enter Spanner cmek property "encryptionKeyName" as macro argument "cmekSpanner" if cmek is enabled
27+
Then Validate "Spanner" plugin properties
28+
Then Close the Spanner properties
29+
Then Connect source as "BigQuery" and sink as "Spanner" to establish connection
30+
Then Save the pipeline
31+
Then Preview and run the pipeline
32+
Then Enter runtime argument value "projectId" for key "bqProjectId"
33+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
34+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "bqServiceAccount"
35+
Then Enter runtime argument value "dataset" for key "bqDataset"
36+
Then Enter runtime argument value for BigQuery source table name key "bqSourceTable"
37+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
38+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "spannerServiceAccount"
39+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
40+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
41+
Then Enter runtime argument value for Spanner Sink Table Name key "spannerTablename"
42+
Then Enter runtime argument value "spannerSinkPrimaryKeyBQ" for key "spannerSinkPrimaryKey"
43+
Then Enter runtime argument value "cmekSpanner" for Spanner cmek property key "cmekSpanner" if Spanner cmek is enabled
44+
Then Run the preview of pipeline with runtime arguments
45+
Then Verify the preview of pipeline is "success"
46+
Then Close the preview
47+
Then Deploy the pipeline
48+
Then Run the Pipeline in Runtime
49+
Then Enter runtime argument value "projectId" for key "bqProjectId"
50+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
51+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "bqServiceAccount"
52+
Then Enter runtime argument value "dataset" for key "bqDataset"
53+
Then Enter runtime argument value for BigQuery source table name key "bqSourceTable"
54+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
55+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "spannerServiceAccount"
56+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
57+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
58+
Then Enter runtime argument value for Spanner Sink Table Name key "spannerTablename"
59+
Then Enter runtime argument value "spannerSinkPrimaryKeyBQ" for key "spannerSinkPrimaryKey"
60+
Then Enter runtime argument value "cmekSpanner" for Spanner cmek property key "cmekSpanner" if Spanner cmek is enabled
61+
Then Run the Pipeline in Runtime with runtime arguments
62+
Then Wait till pipeline is in running state
63+
Then Open and capture logs
64+
Then Verify the pipeline status is "Succeeded"
65+
Then Validate records transferred to target spanner table with record counts of BigQuery table

‎src/e2e-test/features/spanner/sink/GCSToSpanner_WithMacro.feature

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
@Spanner_Sink @SPANNER_TEST
2+
Feature: Spanner Sink - Verification of GCS to Spanner successful data transfer with macro arguments
3+
4+
@GCS_CSV_TEST @SPANNER_SINK_TEST
5+
Scenario:Validate successful records transfer from GCS to Spanner with macro arguments
6+
Given Open Datafusion Project to configure pipeline
7+
When Source is GCS
8+
When Sink is Spanner
9+
Then Open GCS source properties
10+
Then Enter GCS property reference name
11+
Then Enter GCS property "projectId" as macro argument "gcsProjectId"
12+
Then Enter GCS property "serviceAccountFilePath" as macro argument "gcsServiceAccount"
13+
Then Enter GCS property "path" as macro argument "gcsSourcePath"
14+
Then Enter GCS source property "skipHeader" as macro argument "gcsSkipHeader"
15+
Then Enter GCS property "format" as macro argument "gcsFormat"
16+
Then Enter GCS source property output schema "outputSchema" as macro argument "gcsOutputSchema"
17+
Then Validate "GCS" plugin properties
18+
Then Close the GCS properties
19+
Then Open Spanner sink properties
20+
Then Enter Spanner property reference name
21+
Then Enter Spanner property "projectId" as macro argument "spannerProjectId"
22+
Then Enter Spanner property "serviceAccountFilePath" as macro argument "spannerServiceAccount"
23+
Then Enter Spanner property "instanceId" as macro argument "spannerInstanceId"
24+
Then Enter Spanner property "databaseName" as macro argument "spannerDatabaseName"
25+
Then Enter Spanner property "table" as macro argument "spannerTablename"
26+
Then Enter Spanner property "primaryKey" as macro argument "spannerSinkPrimaryKey"
27+
Then Enter Spanner cmek property "encryptionKeyName" as macro argument "cmekSpanner" if cmek is enabled
28+
Then Validate "Spanner" plugin properties
29+
Then Close the Spanner properties
30+
Then Connect source as "GCS" and sink as "Spanner" to establish connection
31+
Then Save the pipeline
32+
Then Preview and run the pipeline
33+
Then Enter runtime argument value "projectId" for key "gcsProjectId"
34+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "gcsServiceAccount"
35+
Then Enter runtime argument value "gcsCsvFile" for GCS source property path key "gcsSourcePath"
36+
Then Enter runtime argument value "gcsSkipHeaderTrue" for key "gcsSkipHeader"
37+
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
38+
Then Enter runtime argument value "gcsCSVFileOutputSchema" for key "gcsOutputSchema"
39+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
40+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "spannerServiceAccount"
41+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
42+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
43+
Then Enter runtime argument value for Spanner Sink Table Name key "spannerTablename"
44+
Then Enter runtime argument value "spannerSinkPrimaryKeyGCS" for key "spannerSinkPrimaryKey"
45+
Then Enter runtime argument value "cmekSpanner" for Spanner cmek property key "cmekSpanner" if Spanner cmek is enabled
46+
Then Run the preview of pipeline with runtime arguments
47+
Then Verify the preview of pipeline is "success"
48+
Then Close the preview
49+
Then Deploy the pipeline
50+
Then Run the Pipeline in Runtime
51+
Then Enter runtime argument value "projectId" for key "gcsProjectId"
52+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "gcsServiceAccount"
53+
Then Enter runtime argument value "gcsCsvFile" for GCS source property path key "gcsSourcePath"
54+
Then Enter runtime argument value "gcsSkipHeaderTrue" for key "gcsSkipHeader"
55+
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
56+
Then Enter runtime argument value "gcsCSVFileOutputSchema" for key "gcsOutputSchema"
57+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
58+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "spannerServiceAccount"
59+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
60+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
61+
Then Enter runtime argument value for Spanner Sink Table Name key "spannerTablename"
62+
Then Enter runtime argument value "spannerSinkPrimaryKeyGCS" for key "spannerSinkPrimaryKey"
63+
Then Enter runtime argument value "cmekSpanner" for Spanner cmek property key "cmekSpanner" if Spanner cmek is enabled
64+
Then Run the Pipeline in Runtime with runtime arguments
65+
Then Wait till pipeline is in running state
66+
Then Open and capture logs
67+
Then Verify the pipeline status is "Succeeded"

‎src/e2e-test/features/spanner/sink/SpannerToSpanner.feature

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
@Spanner_Sink @SPANNER_TEST
2+
Feature: Spanner Sink - Verification of Spanner to Spanner data transfer
3+
4+
@SPANNER_SINK_TEST
5+
Scenario: To verify data is getting transferred successfully from Spanner to Spanner with all supported datatype
6+
Given Open Datafusion Project to configure pipeline
7+
When Source is Spanner
8+
When Sink is Spanner
9+
Then Connect source as "Spanner" and sink as "Spanner" to establish connection
10+
Then Open Spanner source properties
11+
Then Enter Spanner property reference name
12+
Then Enter Spanner property projectId "projectId"
13+
Then Enter Spanner property InstanceId
14+
Then Enter Spanner source property DatabaseName
15+
Then Enter Spanner source property TableName
16+
Then Validate output schema with expectedSchema "spannerSourceSchema"
17+
Then Validate "Spanner" plugin properties
18+
Then Close the Spanner properties
19+
Then Open Spanner sink properties
20+
Then Enter Spanner property reference name
21+
Then Enter Spanner property projectId "projectId"
22+
Then Enter Spanner property InstanceId
23+
Then Enter Spanner sink property DatabaseName
24+
Then Enter Spanner sink property TableName
25+
Then Enter Spanner sink property primary key "spannerSinkPrimaryKeySpanner"
26+
Then Validate "Spanner" plugin properties
27+
Then Close the Spanner properties
28+
Then Save the pipeline
29+
Then Preview and run the pipeline
30+
Then Verify the preview of pipeline is "success"
31+
Then Close the preview
32+
Then Deploy the pipeline
33+
Then Run the Pipeline in Runtime
34+
Then Wait till pipeline is in running state
35+
Then Open and capture logs
36+
Then Verify the pipeline status is "Succeeded"
37+
Then Validate records transferred to target spanner table with record counts of source spanner table

‎src/e2e-test/features/spanner/source/SpannerToBigQuery_WithMacro.feature

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
@Spanner_Source @SPANNER_TEST
2+
Feature: Spanner Source - Verification of Spanner to BigQuery successful data transfer with macro arguments
3+
4+
@BQ_SINK_TEST
5+
Scenario:Validate successful records transfer from Spanner to BigQuery with macro arguments
6+
Given Open Datafusion Project to configure pipeline
7+
When Source is Spanner
8+
When Sink is BigQuery
9+
Then Open Spanner source properties
10+
Then Enter Spanner property reference name
11+
Then Enter Spanner property "projectId" as macro argument "spannerProjectId"
12+
Then Enter Spanner property "serviceAccountFilePath" as macro argument "spannerServiceAccount"
13+
Then Enter Spanner property "instanceId" as macro argument "spannerInstanceId"
14+
Then Enter Spanner property "databaseName" as macro argument "spannerDatabaseName"
15+
Then Enter Spanner property "table" as macro argument "spannerTablename"
16+
Then Validate "Spanner" plugin properties
17+
Then Close the Spanner properties
18+
Then Open BigQuery sink properties
19+
Then Enter BigQuery property reference name
20+
Then Enter BigQuery property "projectId" as macro argument "bqProjectId"
21+
Then Enter BigQuery property "datasetProjectId" as macro argument "bqDatasetProjectId"
22+
Then Enter BigQuery property "serviceAccountFilePath" as macro argument "bqServiceAccount"
23+
Then Enter BigQuery property "dataset" as macro argument "bqDataset"
24+
Then Enter BigQuery property "table" as macro argument "bqTargetTable"
25+
Then Enter BigQuery sink property "truncateTable" as macro argument "bqTruncateTable"
26+
Then Enter BigQuery sink property "updateTableSchema" as macro argument "bqUpdateTableSchema"
27+
Then Validate "BigQuery" plugin properties
28+
Then Close the BigQuery properties
29+
Then Connect source as "Spanner" and sink as "BigQuery" to establish connection
30+
Then Save the pipeline
31+
Then Preview and run the pipeline
32+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
33+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "spannerServiceAccount"
34+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
35+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
36+
Then Enter runtime argument value for Spanner Source Table Name key "spannerTablename"
37+
Then Enter runtime argument value "projectId" for key "bqProjectId"
38+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
39+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "bqServiceAccount"
40+
Then Enter runtime argument value "dataset" for key "bqDataset"
41+
Then Enter runtime argument value for BigQuery sink table name key "bqTargetTable"
42+
Then Enter runtime argument value "bqTruncateTableTrue" for key "bqTruncateTable"
43+
Then Enter runtime argument value "bqUpdateTableSchemaTrue" for key "bqUpdateTableSchema"
44+
Then Run the preview of pipeline with runtime arguments
45+
Then Verify the preview of pipeline is "success"
46+
Then Close the preview
47+
Then Deploy the pipeline
48+
Then Run the Pipeline in Runtime
49+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
50+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "spannerServiceAccount"
51+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
52+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
53+
Then Enter runtime argument value for Spanner Source Table Name key "spannerTablename"
54+
Then Enter runtime argument value "projectId" for key "bqProjectId"
55+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
56+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "bqServiceAccount"
57+
Then Enter runtime argument value "dataset" for key "bqDataset"
58+
Then Enter runtime argument value for BigQuery sink table name key "bqTargetTable"
59+
Then Enter runtime argument value "bqTruncateTableTrue" for key "bqTruncateTable"
60+
Then Enter runtime argument value "bqUpdateTableSchemaTrue" for key "bqUpdateTableSchema"
61+
Then Run the Pipeline in Runtime with runtime arguments
62+
Then Wait till pipeline is in running state
63+
Then Open and capture logs
64+
Then Verify the pipeline status is "Succeeded"
65+
Then Validate records transferred to target BigQuery table with record counts of spanner table

‎src/e2e-test/features/spanner/source/SpannerToGCS_WithMacro.feature

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
@Spanner_Source @SPANNER_TEST
2+
Feature: Spanner Source - Verification of Spanner to GCS successful data transfer with macro arguments
3+
4+
@GCS_SINK_TEST
5+
Scenario:Validate successful records transfer from Spanner to GCS with macro arguments
6+
Given Open Datafusion Project to configure pipeline
7+
When Source is Spanner
8+
When Sink is GCS
9+
Then Open Spanner source properties
10+
Then Enter Spanner property reference name
11+
Then Enter Spanner property "projectId" as macro argument "spannerProjectId"
12+
Then Enter Spanner property "serviceAccountFilePath" as macro argument "spannerServiceAccount"
13+
Then Enter Spanner property "instanceId" as macro argument "spannerInstanceId"
14+
Then Enter Spanner property "databaseName" as macro argument "spannerDatabaseName"
15+
Then Enter BigQuery property "table" as macro argument "spannerTablename"
16+
Then Validate "Spanner" plugin properties
17+
Then Close the Spanner properties
18+
Then Open GCS sink properties
19+
Then Enter GCS property reference name
20+
Then Enter GCS property "projectId" as macro argument "gcsProjectId"
21+
Then Enter GCS property "serviceAccountFilePath" as macro argument "gcsServiceAccount"
22+
Then Enter GCS property "path" as macro argument "gcsSinkPath"
23+
Then Enter GCS sink property "pathSuffix" as macro argument "gcsPathSuffix"
24+
Then Enter GCS property "format" as macro argument "gcsFormat"
25+
Then Validate "GCS" plugin properties
26+
Then Close the GCS properties
27+
Then Connect source as "Spanner" and sink as "GCS" to establish connection
28+
Then Save the pipeline
29+
Then Preview and run the pipeline
30+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
31+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "spannerServiceAccount"
32+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
33+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
34+
Then Enter runtime argument value for Spanner Source Table Name key "spannerTablename"
35+
Then Enter runtime argument value "projectId" for key "gcsProjectId"
36+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "gcsServiceAccount"
37+
Then Enter runtime argument value for GCS sink property path key "gcsSinkPath"
38+
Then Enter runtime argument value "gcsPathDateSuffix" for key "gcsPathSuffix"
39+
Then Enter runtime argument value "avroFormat" for key "gcsFormat"
40+
Then Run the preview of pipeline with runtime arguments
41+
Then Verify the preview of pipeline is "success"
42+
Then Click on preview data for GCS sink
43+
Then Close the preview data
44+
Then Deploy the pipeline
45+
Then Run the Pipeline in Runtime
46+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
47+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "spannerServiceAccount"
48+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
49+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
50+
Then Enter runtime argument value for Spanner Source Table Name key "spannerTablename"
51+
Then Enter runtime argument value "projectId" for key "gcsProjectId"
52+
Then Enter runtime argument value "serviceAccountAutoDetect" for key "gcsServiceAccount"
53+
Then Enter runtime argument value for GCS sink property path key "gcsSinkPath"
54+
Then Enter runtime argument value "gcsPathDateSuffix" for key "gcsPathSuffix"
55+
Then Enter runtime argument value "avroFormat" for key "gcsFormat"
56+
Then Run the Pipeline in Runtime with runtime arguments
57+
Then Wait till pipeline is in running state
58+
Then Open and capture logs
59+
Then Verify the pipeline status is "Succeeded"
60+
Then Verify data is transferred to target GCS bucket

0 commit comments

Comments
 (0)