Skip to content

Commit b096591

Browse files
AnkitCLIAnkitCLI
committed
e2e tests PostgreSql Sink
1 parent 2f51d2d commit b096591

File tree

3 files changed

+204
-0
lines changed

3 files changed

+204
-0
lines changed

‎postgresql-plugin/src/e2e-test/features/postgresql/sink/PostgresqlRunTime.feature

+153
Original file line numberDiff line numberDiff line change
@@ -144,3 +144,156 @@ Feature: PostgreSQL - Verify data transfer from BigQuery source to PostgreSQL si
144144
Then Open and capture logs
145145
Then Verify the pipeline status is "Succeeded"
146146
Then Validate the values of records transferred to target PostgreSQL table is equal to the values from source BigQuery table
147+
148+
@POSTGRESQL_SOURCE_TEST @Postgresql_Required @POSTGRESQL_SINK_TEST
149+
Scenario Outline: To verify data is getting transferred from PostgreSQL source to PostgreSQL sink with Advanced operations Upsert
150+
Given Open Datafusion Project to configure pipeline
151+
Given Open Datafusion Project to configure pipeline
152+
When Expand Plugin group in the LHS plugins list: "Source"
153+
When Select plugin: "PostgreSQL" from the plugins list as: "Source"
154+
When Expand Plugin group in the LHS plugins list: "Sink"
155+
When Select plugin: "PostgreSQL" from the plugins list as: "Sink"
156+
Then Connect plugins: "PostgreSQL" and "PostgreSQL2" to establish connection
157+
Then Navigate to the properties page of plugin: "PostgreSQL"
158+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
159+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
160+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
161+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
162+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
163+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
164+
Then Replace input plugin property: "database" with value: "databaseName"
165+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
166+
Then Click on the Get Schema button
167+
Then Validate "PostgreSQL" plugin properties
168+
Then Close the Plugin Properties page
169+
Then Navigate to the properties page of plugin: "PostgreSQL2"
170+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
171+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
172+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
173+
Then Replace input plugin property: "database" with value: "databaseName"
174+
Then Replace input plugin property: "tableName" with value: "targetTable"
175+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
176+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
177+
Then Enter input plugin property: "referenceName" with value: "targetRef"
178+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
179+
Then Select radio button plugin property: "operationName" with value: "upsert"
180+
Then Click on the Add Button of the property: "relationTableKey" with value:
181+
| PostgreSQLTableKey |
182+
Then Validate "PostgreSQL" plugin properties
183+
Then Close the Plugin Properties page
184+
Then Save the pipeline
185+
Then Preview and run the pipeline
186+
Then Verify the preview of pipeline is "success"
187+
Then Click on preview data for PostgreSQL sink
188+
Then Close the preview data
189+
Then Deploy the pipeline
190+
Then Run the Pipeline in Runtime
191+
Then Wait till pipeline is in running state
192+
Then Open and capture logs
193+
Then Verify the pipeline status is "Succeeded"
194+
Then Validate the values of records transferred to target table is equal to the values from source table
195+
# Examples:
196+
# | options |
197+
# | upsert |
198+
# | update |
199+
200+
@POSTGRESQL_SOURCE_TEST @Postgresql_Required @POSTGRESQL_SINK_TEST
201+
Scenario Outline: To verify data is getting transferred from PostgreSQL source to PostgreSQL sink with Advanced operations Update for table key
202+
Given Open Datafusion Project to configure pipeline
203+
Given Open Datafusion Project to configure pipeline
204+
When Expand Plugin group in the LHS plugins list: "Source"
205+
When Select plugin: "PostgreSQL" from the plugins list as: "Source"
206+
When Expand Plugin group in the LHS plugins list: "Sink"
207+
When Select plugin: "PostgreSQL" from the plugins list as: "Sink"
208+
Then Connect plugins: "PostgreSQL" and "PostgreSQL2" to establish connection
209+
Then Navigate to the properties page of plugin: "PostgreSQL"
210+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
211+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
212+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
213+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
214+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
215+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
216+
Then Replace input plugin property: "database" with value: "databaseName"
217+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
218+
Then Click on the Get Schema button
219+
Then Validate "PostgreSQL" plugin properties
220+
Then Close the Plugin Properties page
221+
Then Navigate to the properties page of plugin: "PostgreSQL2"
222+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
223+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
224+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
225+
Then Replace input plugin property: "database" with value: "databaseName"
226+
Then Replace input plugin property: "tableName" with value: "targetTable"
227+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
228+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
229+
Then Enter input plugin property: "referenceName" with value: "targetRef"
230+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
231+
Then Select radio button plugin property: "operationName" with value: "update"
232+
Then Click on the Add Button of the property: "relationTableKey" with value:
233+
| PostgreSQLTableKey |
234+
Then Validate "PostgreSQL" plugin properties
235+
Then Close the Plugin Properties page
236+
Then Save the pipeline
237+
Then Preview and run the pipeline
238+
Then Verify the preview of pipeline is "success"
239+
Then Click on preview data for PostgreSQL sink
240+
Then Close the preview data
241+
Then Deploy the pipeline
242+
Then Run the Pipeline in Runtime
243+
Then Wait till pipeline is in running state
244+
Then Open and capture logs
245+
Then Verify the pipeline status is "Succeeded"
246+
Then Validate the values of records transferred to target table is equal to the values from source table
247+
248+
@POSTGRESQL_SOURCE_TEST @POSTGRESQL_SINK_TEST @Postgresql_Required @CONNECTION
249+
Scenario: To verify data is getting transferred from PostgreSQL source to PostgreSQL sink successfully using Connection
250+
Given Open Datafusion Project to configure pipeline
251+
When Expand Plugin group in the LHS plugins list: "Source"
252+
When Select plugin: "PostgreSQL" from the plugins list as: "Source"
253+
When Expand Plugin group in the LHS plugins list: "Sink"
254+
When Select plugin: "PostgreSQL" from the plugins list as: "Sink"
255+
Then Connect plugins: "PostgreSQL" and "PostgreSQL2" to establish connection
256+
Then Navigate to the properties page of plugin: "PostgreSQL"
257+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
258+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
259+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
260+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
261+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
262+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
263+
Then Replace input plugin property: "database" with value: "databaseName"
264+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
265+
Then Click on the Get Schema button
266+
Then Validate "PostgreSQL" plugin properties
267+
Then Close the Plugin Properties page
268+
Then Navigate to the properties page of plugin: "PostgreSQL2"
269+
And Click plugin property: "switch-useConnection"
270+
And Click on the Browse Connections button
271+
And Click on the Add Connection button
272+
Then Click plugin property: "connector-PostgreSQL"
273+
And Enter input plugin property: "name" with value: "connection.name"
274+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
275+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
276+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
277+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
278+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
279+
Then Replace input plugin property: "database" with value: "databaseName"
280+
Then Click on the Test Connection button
281+
And Verify the test connection is successful
282+
Then Click on the Create button
283+
Then Select connection: "connection.name"
284+
Then Enter input plugin property: "referenceName" with value: "targetRef"
285+
Then Replace input plugin property: "tableName" with value: "targetTable"
286+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
287+
Then Validate "PostgreSQL" plugin properties
288+
Then Close the Plugin Properties page
289+
Then Save the pipeline
290+
Then Preview and run the pipeline
291+
Then Verify the preview of pipeline is "success"
292+
Then Click on preview data for PostgreSQL sink
293+
Then Close the preview data
294+
Then Deploy the pipeline
295+
Then Run the Pipeline in Runtime
296+
Then Wait till pipeline is in running state
297+
Then Open and capture logs
298+
Then Verify the pipeline status is "Succeeded"
299+
Then Validate the values of records transferred to target table is equal to the values from source table

‎postgresql-plugin/src/e2e-test/features/postgresql/sink/PostgresqlRunTimeMacro.feature

+48
Original file line numberDiff line numberDiff line change
@@ -136,3 +136,51 @@ Feature: PostgreSQL - Verify data transfer to PostgreSQL sink with macro argumen
136136
Then Verify the pipeline status is "Succeeded"
137137
Then Close the pipeline logs
138138
Then Validate the values of records transferred to target PostgreSQL table is equal to the values from source BigQuery table
139+
140+
@BQ_SOURCE_TEST @Postgresql_Required @POSTGRESQL_TEST_TABLE
141+
Scenario: To verify data is getting transferred from BigQuery source to PostgreSQL sink using connection arguments and operations as macro
142+
Given Open Datafusion Project to configure pipeline
143+
When Expand Plugin group in the LHS plugins list: "Source"
144+
When Select plugin: "BigQuery" from the plugins list as: "Source"
145+
When Expand Plugin group in the LHS plugins list: "Sink"
146+
When Select plugin: "PostgreSQL" from the plugins list as: "Sink"
147+
Then Connect plugins: "BigQuery" and "PostgreSQL" to establish connection
148+
Then Navigate to the properties page of plugin: "BigQuery"
149+
Then Enter input plugin property: "referenceName" with value: "BQReferenceName"
150+
Then Click on the Macro button of Property: "projectId" and set the value to: "bqProjectId"
151+
Then Click on the Macro button of Property: "datasetProjectId" and set the value to: "bqDatasetProjectId"
152+
Then Click on the Macro button of Property: "dataset" and set the value to: "bqDataset"
153+
Then Click on the Macro button of Property: "table" and set the value to: "bqTable"
154+
Then Validate "BigQuery" plugin properties
155+
Then Close the Plugin Properties page
156+
Then Navigate to the properties page of plugin: "PostgreSQL"
157+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
158+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
159+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
160+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
161+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
162+
Then Enter input plugin property: "referenceName" with value: "targetRef"
163+
Then Replace input plugin property: "database" with value: "databaseName"
164+
Then Click on the Macro button of Property: "connectionArguments" and set the value to: "PostgreSQLConnectionArguments"
165+
Then Click on the Macro button of Property: "operationName" and set the value to: "PostgreSQLOperationName"
166+
Then Click on the Macro button of Property: "tableName" and set the value to: "PostgreSQLTableName"
167+
Then Click on the Macro button of Property: "dbSchemaName" and set the value to: "PostgreSQLSchemaName"
168+
Then Validate "PostgreSQL" plugin properties
169+
Then Close the Plugin Properties page
170+
Then Save the pipeline
171+
Then Deploy the pipeline
172+
Then Run the Pipeline in Runtime
173+
Then Enter runtime argument value "projectId" for key "bqProjectId"
174+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
175+
Then Enter runtime argument value "dataset" for key "bqDataset"
176+
Then Enter runtime argument value "bqSourceTable" for key "bqTable"
177+
Then Enter runtime argument value "PostgreSQLConnectionArgumentsList" for key "PostgreSQLConnectionArguments"
178+
Then Enter runtime argument value "PostgreSQLOperationName" for key "PostgreSQLOperationName"
179+
Then Enter runtime argument value "targetTable" for key "PostgreSQLTableName"
180+
Then Enter runtime argument value "schema" for key "PostgreSQLSchemaName"
181+
Then Run the Pipeline in Runtime with runtime arguments
182+
Then Wait till pipeline is in running state
183+
Then Open and capture logs
184+
Then Verify the pipeline status is "Succeeded"
185+
Then Close the pipeline logs
186+
Then Validate the values of records transferred to target PostgreSQL table is equal to the values from source BigQuery table

‎postgresql-plugin/src/e2e-test/resources/pluginParameters.properties

+3
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,9 @@ invalidBoundingQuery=SELECT MIN(id),MAX(id) FROM table
7474
invalidBoundingQueryValue=select;
7575
invalidTable=table
7676
#POSTGRESQL Valid Properties
77+
PostgreSQLConnectionArgumentsList=fetchsize=1000
78+
PostgreSQLOperationName=insert
79+
PostgreSQLTableKey=col2
7780
connectionArgumentsList=[{"key":"queryTimeout","value":"-1"}]
7881
connectionTimeout=150
7982
numberOfSplits=2

0 commit comments

Comments
 (0)