Skip to content

Commit d53a76f

Browse files
sawantpritamsawantpritam
committed
e2e Spanner additional test scenarios
1 parent 5ccf39d commit d53a76f

10 files changed

+392
-5
lines changed

‎src/e2e-test/features/spanner/sink/BigQueryToSpanner_WithMacro.feature

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
@Spanner_Sink @SPANNER_TEST
2+
Feature: Spanner Sink - Verification of BigQuery to Spanner successful data transfer with macro arguments
3+
4+
@BQ_SOURCE_TEST @SPANNER_SINK_TEST
5+
Scenario:Validate successful records transfer from BigQuery to Spanner with macro arguments
6+
Given Open Datafusion Project to configure pipeline
7+
When Source is BigQuery
8+
When Sink is Spanner
9+
Then Open BigQuery source properties
10+
Then Enter BigQuery property reference name
11+
Then Enter BigQuery property "projectId" as macro argument "bqProjectId"
12+
Then Enter BigQuery property "datasetProjectId" as macro argument "bqDatasetProjectId"
13+
Then Enter BigQuery property "serviceAccountType" as macro argument "serviceAccountType"
14+
Then Enter BigQuery property "serviceAccountFilePath" as macro argument "serviceAccount"
15+
Then Enter BigQuery property "serviceAccountJSON" as macro argument "serviceAccount"
16+
Then Enter BigQuery property "dataset" as macro argument "bqDataset"
17+
Then Enter BigQuery property "table" as macro argument "bqSourceTable"
18+
Then Validate "BigQuery" plugin properties
19+
Then Close the BigQuery properties
20+
Then Open Spanner sink properties
21+
Then Enter Spanner property reference name
22+
Then Enter Spanner property "projectId" as macro argument "spannerProjectId"
23+
Then Enter Spanner property "serviceAccountType" as macro argument "serviceAccountType"
24+
Then Enter Spanner property "serviceAccountFilePath" as macro argument "serviceAccount"
25+
Then Enter Spanner property "serviceAccountJSON" as macro argument "serviceAccount"
26+
Then Enter Spanner property "instanceId" as macro argument "spannerInstanceId"
27+
Then Enter Spanner property "databaseName" as macro argument "spannerDatabaseName"
28+
Then Enter Spanner property "table" as macro argument "spannerTablename"
29+
Then Enter Spanner property "primaryKey" as macro argument "spannerSinkPrimaryKey"
30+
Then Enter Spanner cmek property "encryptionKeyName" as macro argument "cmekSpanner" if cmek is enabled
31+
Then Validate "Spanner" plugin properties
32+
Then Close the Spanner properties
33+
Then Connect source as "BigQuery" and sink as "Spanner" to establish connection
34+
Then Save the pipeline
35+
Then Preview and run the pipeline
36+
Then Enter runtime argument value "projectId" for key "bqProjectId"
37+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
38+
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
39+
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
40+
Then Enter runtime argument value "dataset" for key "bqDataset"
41+
Then Enter runtime argument value for BigQuery source table name key "bqSourceTable"
42+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
43+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
44+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
45+
Then Enter runtime argument value for Spanner Sink Table Name key "spannerTablename"
46+
Then Enter runtime argument value "spannerSinkPrimaryKeyBQ" for key "spannerSinkPrimaryKey"
47+
Then Enter runtime argument value "cmekSpanner" for Spanner cmek property key "cmekSpanner" if Spanner cmek is enabled
48+
Then Run the preview of pipeline with runtime arguments
49+
Then Verify the preview of pipeline is "success"
50+
Then Close the preview
51+
Then Deploy the pipeline
52+
Then Run the Pipeline in Runtime
53+
Then Enter runtime argument value "projectId" for key "bqProjectId"
54+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
55+
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
56+
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
57+
Then Enter runtime argument value "dataset" for key "bqDataset"
58+
Then Enter runtime argument value for BigQuery source table name key "bqSourceTable"
59+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
60+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
61+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
62+
Then Enter runtime argument value for Spanner Sink Table Name key "spannerTablename"
63+
Then Enter runtime argument value "spannerSinkPrimaryKeyBQ" for key "spannerSinkPrimaryKey"
64+
Then Enter runtime argument value "cmekSpanner" for Spanner cmek property key "cmekSpanner" if Spanner cmek is enabled
65+
Then Run the Pipeline in Runtime with runtime arguments
66+
Then Wait till pipeline is in running state
67+
Then Open and capture logs
68+
Then Verify the pipeline status is "Succeeded"
69+
Then Validate records transferred to target spanner table with record counts of BigQuery table

‎src/e2e-test/features/spanner/sink/GCSToSpanner_WithMacro.feature

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
@Spanner_Sink @SPANNER_TEST
2+
Feature: Spanner Sink - Verification of GCS to Spanner successful data transfer with macro arguments
3+
4+
@GCS_CSV_TEST @SPANNER_SINK_TEST
5+
Scenario:Validate successful records transfer from GCS to Spanner with macro arguments
6+
Given Open Datafusion Project to configure pipeline
7+
When Source is GCS
8+
When Sink is Spanner
9+
Then Open GCS source properties
10+
Then Enter GCS property reference name
11+
Then Enter GCS property "projectId" as macro argument "gcsProjectId"
12+
Then Enter GCS property "serviceAccountType" as macro argument "serviceAccountType"
13+
Then Enter GCS property "serviceAccountFilePath" as macro argument "serviceAccount"
14+
Then Enter GCS property "serviceAccountJSON" as macro argument "serviceAccount"
15+
Then Enter GCS property "path" as macro argument "gcsSourcePath"
16+
Then Enter GCS property "format" as macro argument "gcsFormat"
17+
Then Enter GCS source property "skipHeader" as macro argument "gcsSkipHeader"
18+
Then Enter GCS source property output schema "outputSchema" as macro argument "gcsOutputSchema"
19+
Then Validate "GCS" plugin properties
20+
Then Close the GCS properties
21+
Then Open Spanner sink properties
22+
Then Enter Spanner property reference name
23+
Then Enter Spanner property "projectId" as macro argument "spannerProjectId"
24+
Then Enter Spanner property "serviceAccountType" as macro argument "serviceAccountType"
25+
Then Enter Spanner property "serviceAccountFilePath" as macro argument "serviceAccount"
26+
Then Enter Spanner property "serviceAccountJSON" as macro argument "serviceAccount"
27+
Then Enter Spanner property "instanceId" as macro argument "spannerInstanceId"
28+
Then Enter Spanner property "databaseName" as macro argument "spannerDatabaseName"
29+
Then Enter Spanner property "table" as macro argument "spannerTablename"
30+
Then Enter Spanner property "primaryKey" as macro argument "spannerSinkPrimaryKey"
31+
Then Enter Spanner cmek property "encryptionKeyName" as macro argument "cmekSpanner" if cmek is enabled
32+
Then Validate "Spanner" plugin properties
33+
Then Close the Spanner properties
34+
Then Connect source as "GCS" and sink as "Spanner" to establish connection
35+
Then Save the pipeline
36+
Then Preview and run the pipeline
37+
Then Enter runtime argument value "projectId" for key "gcsProjectId"
38+
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
39+
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
40+
Then Enter runtime argument value "gcsCsvFile" for GCS source property path key "gcsSourcePath"
41+
Then Enter runtime argument value "gcsSkipHeaderTrue" for key "gcsSkipHeader"
42+
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
43+
Then Enter runtime argument value "gcsCSVFileOutputSchema" for key "gcsOutputSchema"
44+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
45+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
46+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
47+
Then Enter runtime argument value for Spanner Sink Table Name key "spannerTablename"
48+
Then Enter runtime argument value "spannerSinkPrimaryKeyGCS" for key "spannerSinkPrimaryKey"
49+
Then Enter runtime argument value "cmekSpanner" for Spanner cmek property key "cmekSpanner" if Spanner cmek is enabled
50+
Then Run the preview of pipeline with runtime arguments
51+
Then Verify the preview of pipeline is "success"
52+
Then Close the preview
53+
Then Deploy the pipeline
54+
Then Run the Pipeline in Runtime
55+
Then Enter runtime argument value "projectId" for key "gcsProjectId"
56+
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
57+
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
58+
Then Enter runtime argument value "gcsCsvFile" for GCS source property path key "gcsSourcePath"
59+
Then Enter runtime argument value "gcsSkipHeaderTrue" for key "gcsSkipHeader"
60+
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
61+
Then Enter runtime argument value "gcsCSVFileOutputSchema" for key "gcsOutputSchema"
62+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
63+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
64+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
65+
Then Enter runtime argument value for Spanner Sink Table Name key "spannerTablename"
66+
Then Enter runtime argument value "spannerSinkPrimaryKeyGCS" for key "spannerSinkPrimaryKey"
67+
Then Enter runtime argument value "cmekSpanner" for Spanner cmek property key "cmekSpanner" if Spanner cmek is enabled
68+
Then Run the Pipeline in Runtime with runtime arguments
69+
Then Wait till pipeline is in running state
70+
Then Open and capture logs
71+
Then Verify the pipeline status is "Succeeded"

‎src/e2e-test/features/spanner/sink/SpannerToSpanner.feature

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
@Spanner_Sink @SPANNER_TEST
2+
Feature: Spanner Sink - Verification of Spanner to Spanner data transfer
3+
4+
@SPANNER_SINK_TEST
5+
Scenario: To verify data is getting transferred successfully from Spanner to Spanner with all supported datatype
6+
Given Open Datafusion Project to configure pipeline
7+
When Source is Spanner
8+
When Sink is Spanner
9+
Then Connect source as "Spanner" and sink as "Spanner" to establish connection
10+
Then Open Spanner source properties
11+
Then Enter Spanner property reference name
12+
Then Enter Spanner property projectId "projectId"
13+
Then Override Service account details if set in environment variables
14+
Then Enter Spanner property InstanceId
15+
Then Enter Spanner source property DatabaseName
16+
Then Enter Spanner source property TableName
17+
Then Validate output schema with expectedSchema "spannerSourceSchema"
18+
Then Validate "Spanner" plugin properties
19+
Then Close the Spanner properties
20+
Then Open Spanner sink properties
21+
Then Enter Spanner property reference name
22+
Then Enter Spanner property projectId "projectId"
23+
Then Override Service account details if set in environment variables
24+
Then Enter Spanner property InstanceId
25+
Then Enter Spanner sink property DatabaseName
26+
Then Enter Spanner sink property TableName
27+
Then Enter Spanner sink property primary key "spannerSinkPrimaryKeySpanner"
28+
Then Validate "Spanner" plugin properties
29+
Then Close the Spanner properties
30+
Then Save the pipeline
31+
Then Preview and run the pipeline
32+
Then Verify the preview of pipeline is "success"
33+
Then Close the preview
34+
Then Deploy the pipeline
35+
Then Run the Pipeline in Runtime
36+
Then Wait till pipeline is in running state
37+
Then Open and capture logs
38+
Then Verify the pipeline status is "Succeeded"
39+
Then Validate records transferred to target spanner table with record counts of source spanner table

‎src/e2e-test/features/spanner/source/SpannerToBigQuery_WithMacro.feature

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
@Spanner_Source @SPANNER_TEST
2+
Feature: Spanner Source - Verification of Spanner to BigQuery successful data transfer with macro arguments
3+
4+
@BQ_SINK_TEST
5+
Scenario:Validate successful records transfer from Spanner to BigQuery with macro arguments
6+
Given Open Datafusion Project to configure pipeline
7+
When Source is Spanner
8+
When Sink is BigQuery
9+
Then Open Spanner source properties
10+
Then Enter Spanner property reference name
11+
Then Enter Spanner property "projectId" as macro argument "spannerProjectId"
12+
Then Enter Spanner property "serviceAccountType" as macro argument "serviceAccountType"
13+
Then Enter Spanner property "serviceAccountFilePath" as macro argument "serviceAccount"
14+
Then Enter Spanner property "serviceAccountJSON" as macro argument "serviceAccount"
15+
Then Enter Spanner property "instanceId" as macro argument "spannerInstanceId"
16+
Then Enter Spanner property "databaseName" as macro argument "spannerDatabaseName"
17+
Then Enter Spanner property "table" as macro argument "spannerTablename"
18+
Then Validate "Spanner" plugin properties
19+
Then Close the Spanner properties
20+
Then Open BigQuery sink properties
21+
Then Enter BigQuery property reference name
22+
Then Enter BigQuery property "projectId" as macro argument "bqProjectId"
23+
Then Enter BigQuery property "datasetProjectId" as macro argument "bqDatasetProjectId"
24+
Then Enter BigQuery property "serviceAccountType" as macro argument "serviceAccountType"
25+
Then Enter BigQuery property "serviceAccountFilePath" as macro argument "serviceAccount"
26+
Then Enter BigQuery property "serviceAccountJSON" as macro argument "serviceAccount"
27+
Then Enter BigQuery property "dataset" as macro argument "bqDataset"
28+
Then Enter BigQuery property "table" as macro argument "bqTargetTable"
29+
Then Enter BigQuery sink property "truncateTable" as macro argument "bqTruncateTable"
30+
Then Enter BigQuery sink property "updateTableSchema" as macro argument "bqUpdateTableSchema"
31+
Then Validate "BigQuery" plugin properties
32+
Then Close the BigQuery properties
33+
Then Connect source as "Spanner" and sink as "BigQuery" to establish connection
34+
Then Save the pipeline
35+
Then Preview and run the pipeline
36+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
37+
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
38+
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
39+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
40+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
41+
Then Enter runtime argument value for Spanner Source Table Name key "spannerTablename"
42+
Then Enter runtime argument value "projectId" for key "bqProjectId"
43+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
44+
Then Enter runtime argument value "dataset" for key "bqDataset"
45+
Then Enter runtime argument value for BigQuery sink table name key "bqTargetTable"
46+
Then Enter runtime argument value "bqTruncateTableTrue" for key "bqTruncateTable"
47+
Then Enter runtime argument value "bqUpdateTableSchemaTrue" for key "bqUpdateTableSchema"
48+
Then Run the preview of pipeline with runtime arguments
49+
Then Verify the preview of pipeline is "success"
50+
Then Close the preview
51+
Then Deploy the pipeline
52+
Then Run the Pipeline in Runtime
53+
Then Enter runtime argument value "projectId" for key "spannerProjectId"
54+
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
55+
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
56+
Then Enter runtime argument value for Spanner Instance ID key "spannerInstanceId"
57+
Then Enter runtime argument value for Spanner Database Name key "spannerDatabaseName"
58+
Then Enter runtime argument value for Spanner Source Table Name key "spannerTablename"
59+
Then Enter runtime argument value "projectId" for key "bqProjectId"
60+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
61+
Then Enter runtime argument value "dataset" for key "bqDataset"
62+
Then Enter runtime argument value for BigQuery sink table name key "bqTargetTable"
63+
Then Enter runtime argument value "bqTruncateTableTrue" for key "bqTruncateTable"
64+
Then Enter runtime argument value "bqUpdateTableSchemaTrue" for key "bqUpdateTableSchema"
65+
Then Run the Pipeline in Runtime with runtime arguments
66+
Then Wait till pipeline is in running state
67+
Then Open and capture logs
68+
Then Verify the pipeline status is "Succeeded"
69+
Then Validate records transferred to target BigQuery table with record counts of spanner table

0 commit comments

Comments
 (0)