Add Integration Tests for E2E Multi-type transformations #1354

lewijacn · 2025-03-13T23:18:00Z

Description

This change accomplishes a few high level tasks:

Introduces new Integration Test case model that K8s will utilize while preserving the current integration tests model for our existing setup

This new pattern allows us to clearly group the operations and checks a particular test case will perform, while also restricting particular test scenarios to specific ES or OS version combinations, as well as load specific libraries based on the source or target cluster version used. This also gives test scenarios much more flexibility to control the actual migration events, such as providing a particular transformation to a metadata migration

Adds two new E2E test cases for Union Multi-type Transformation and Split Multi-type Transformation
Adds shared logs volume and mounts to relevant pods for K8s

Issues Resolved

https://opensearch.atlassian.net/browse/MIGRATIONS-2340

Testing

Local K8s testing for new model
Cloud testing existing tests

Check List

New functionality includes testing
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Tanner Lewis <[email protected]>

codecov · 2025-03-14T15:49:04Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.12%. Comparing base (a5d393f) to head (8727423).
Report is 18 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1354      +/-   ##
============================================
+ Coverage     79.91%   80.12%   +0.20%     
  Complexity     2977     2977              
============================================
  Files           443      440       -3     
  Lines         16217    16188      -29     
  Branches       1087     1078       -9     
============================================
+ Hits          12960    12970      +10     
+ Misses         2617     2583      -34     
+ Partials        640      635       -5

Flag	Coverage Δ
unittests	`80.12% <ø> (+0.20%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Tanner Lewis <[email protected]>

peternied

Took a brief look, added some high level comments.

...apture/dockerSolution/src/main/docker/migrationConsole/lib/integ_test/integ_test/conftest.py

...ution/src/main/docker/migrationConsole/lib/integ_test/integ_test/elasticsearch_operations.py

Signed-off-by: Tanner Lewis <[email protected]>

...on/src/main/docker/migrationConsole/lib/integ_test/integ_test/test_cases/multi_type_tests.py

deployment/k8s/charts/aggregates/migrationAssistant/values.yaml

deployment/k8s/charts/components/migrationConsole/values.yaml

gregschohn · 2025-03-17T18:32:26Z

...kerSolution/src/main/docker/migrationConsole/lib/integ_test/integ_test/default_operations.py

+                index_dict[index_name]['index'] = index_name
+        return index_dict
+
+    def check_doc_counts_match(self, cluster: Cluster,


nit: This is a little bit weird that you have a verify function above that throws and a check function here that fails the test case.

Yes agreed, this was copied from the existing integ test common_operations.py file that got removed. Once we remove the existing integration tests I'd like to remove passing the test case to any of these functions and allow them to just throw an error if a check fails.

gregschohn · 2025-03-17T18:35:08Z

...ockerSolution/src/main/docker/migrationConsole/lib/integ_test/integ_test/ma_workflow_test.py

+
+    # Cleanup generated transformation files
+    try:
+        shutil.rmtree("/shared-logs-output/test-transformations")


how much effort would it be to delete these when we're done and only upon success? Adding and remembering some uniqifier (like timestamp) could be helpful too.

It's probably a small effort, especially if the teardown function has some way of determining success or failure. It seems like we really want a lot of things to get generated somewhere persistent, such as service logs, cluster logs, test_output, etc. Currently items aren't cleaned up until another test case is executed.

gregschohn · 2025-03-17T18:36:22Z

...ockerSolution/src/main/docker/migrationConsole/lib/integ_test/integ_test/ma_workflow_test.py

+    run_on_all_cases(test_cases=test_cases, operation=lambda case: case.perform_initial_operations())
+    control_test_case.perform_snapshot_create()


ni: consistent indenting for all of the following lines would make it easier to quickly understand the whole block

Agreed will update

gregschohn · 2025-03-17T18:40:54Z

...ion/src/main/docker/migrationConsole/lib/integ_test/integ_test/operations_library_factory.py

+
+
+def get_operations_library_by_version(version: ClusterVersion):
+    if is_incoming_version_supported(limiting_version=ElasticsearchV5_X, incoming_version=version):


I wish that we could setup numeric values for versions, but because of the reset and fork, it's a lot harder to manage (and would have some exceptional cases).

Yes, it seems we have ended up on this ES|OS_x.y pattern

gregschohn · 2025-03-17T18:46:55Z

...olution/src/main/docker/migrationConsole/lib/integ_test/integ_test/test_cases/basic_tests.py

+
+class Test0002IndexWithNoDocumentsMetadataMigration(MATestBase):
+    def __init__(self, console_config_path: str, console_link_env: Environment, unique_id: str):
+        allow_combinations = [


nit/future: it might be better to model these as a function/lambda. These seems like they could start to get pretty when big fully materialized.

My next iteration was going to be to model this as (source_version, list(target_version)), but curious what you had in mind for a lambda around this?

gregschohn · 2025-03-17T19:19:32Z

deployment/k8s/charts/aggregates/migrationAssistant/values.yaml

+        endpoint: "http://ma-capture-proxy:9200"
+        directEndpoint: "http://elasticsearch-master:9200"


why do we have this split now?

The capture proxy will need both of these values. In the future I also think the Migration Console will want the ability to hit the source directly or hit the capture proxy, we don't currently have support for this yet though

gregschohn · 2025-03-17T19:23:37Z

deployment/k8s/charts/aggregates/migrationAssistant/values.yaml

+      allowRuntimeOverride: true
+    docTransformerConfigFile:
+      source: parameterConfig
+      value: "/shared-logs-output/test-transformations/transformation.json"


See https://opensearch.atlassian.net/browse/MIGRATIONS-2401 for a better way to send configs/arguments into processes.

Yes I expect this pattern to change when we have support for modifying configmaps as part of a workflow

gregschohn · 2025-03-17T19:24:37Z

deployment/k8s/charts/components/bulkLoad/values.yaml

@@ -1,3 +1,5 @@
+sharedLogsVolumeEnabled: false
+sharedLogsPvc: ""


hmmm - this isn't very obvious (to clear the pvc) - what happens if this isn't done?

Not sure what you mean. As a default we will not mount any shared logs pvc, but our MA chart will fill this out to mount the shared logs pvc

deployment/k8s/charts/components/migrationConsole/values.yaml

gregschohn · 2025-03-17T19:30:01Z

deployment/k8s/minikubeLocal.sh

 }

 pause() {
+  kill_minikube_processes


When do you typically do this in your workflows? I usually leave mine running and don't think much about it.

Only when I need to restart normally. Really this is just a means for cleaning up if a user wants to.

Signed-off-by: Tanner Lewis <[email protected]>

… tests Signed-off-by: Tanner Lewis <[email protected]>

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn added 7 commits March 7, 2025 18:57

Baseline a working test case division

80d80b2

Signed-off-by: Tanner Lewis <[email protected]>

Updates for passing test id filtering

0b38819

Signed-off-by: Tanner Lewis <[email protected]>

Add shared logs volume support and union multi type test support

e810a88

Signed-off-by: Tanner Lewis <[email protected]>

Merge remote-tracking branch 'origin/main' into refactor-integ-tests

47c768b

Add recent changes to tests

68faae0

Signed-off-by: Tanner Lewis <[email protected]>

Remove version params for integ test library and realign old tests

8803ac3

Signed-off-by: Tanner Lewis <[email protected]>

Flake8 fixes

bf7201b

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn had a problem deploying to migrations-cicd March 13, 2025 23:18 — with GitHub Actions Failure

Minor updates for test cases

a221946

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn had a problem deploying to migrations-cicd March 14, 2025 15:18 — with GitHub Actions Failure

lewijacn marked this pull request as ready for review March 14, 2025 15:30

lewijacn requested review from AndreKurait, chelma, gregschohn, jugal-chauhan, mikaylathompson, peternied and sumobrian as code owners March 14, 2025 15:30

Fix imported function

d72ae87

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn temporarily deployed to migrations-cicd March 14, 2025 16:16 — with GitHub Actions Inactive

peternied reviewed Mar 14, 2025

View reviewed changes

...apture/dockerSolution/src/main/docker/migrationConsole/lib/integ_test/integ_test/conftest.py Show resolved Hide resolved

...ution/src/main/docker/migrationConsole/lib/integ_test/integ_test/elasticsearch_operations.py Show resolved Hide resolved

lewijacn added 2 commits March 14, 2025 13:45

Merge remote-tracking branch 'origin/main' into refactor-integ-tests

9cc9e44

Signed-off-by: Tanner Lewis <[email protected]>

Minor enhancements to allow developer testing and align values

7c24683

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn temporarily deployed to migrations-cicd March 17, 2025 15:20 — with GitHub Actions Inactive

peternied reviewed Mar 17, 2025

View reviewed changes

gregschohn reviewed Mar 17, 2025

View reviewed changes

Partial PR fixes, more to come

ef49eb2

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn temporarily deployed to migrations-cicd March 18, 2025 17:20 — with GitHub Actions Inactive

Rename workflow functions for integ tests and minor edits

f2afc51

Signed-off-by: Tanner Lewis <[email protected]>

Flake8 fix

391a760

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn temporarily deployed to migrations-cicd March 19, 2025 20:57 — with GitHub Actions Inactive

Merge remote-tracking branch 'origin/main' into refactor-integ-tests

0e22f2b

lewijacn temporarily deployed to migrations-cicd March 19, 2025 21:01 — with GitHub Actions Inactive

peternied approved these changes Mar 19, 2025

View reviewed changes

Minor fixes for proper test functions and clearing index templates in…

35f4659

… tests Signed-off-by: Tanner Lewis <[email protected]>

lewijacn temporarily deployed to migrations-cicd March 20, 2025 03:13 — with GitHub Actions Inactive

Fix function name

8727423

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn temporarily deployed to migrations-cicd March 20, 2025 15:11 — with GitHub Actions Inactive

lewijacn merged commit 9dd0390 into opensearch-project:main Mar 20, 2025
60 checks passed

lewijacn deleted the refactor-integ-tests branch August 4, 2025 15:47

		run_on_all_cases(test_cases=test_cases, operation=lambda case: case.perform_initial_operations())
		control_test_case.perform_snapshot_create()



		def get_operations_library_by_version(version: ClusterVersion):
		if is_incoming_version_supported(limiting_version=ElasticsearchV5_X, incoming_version=version):

		endpoint: "http://ma-capture-proxy:9200"
		directEndpoint: "http://elasticsearch-master:9200"

		@@ -1,3 +1,5 @@
		sharedLogsVolumeEnabled: false
		sharedLogsPvc: ""

Add Integration Tests for E2E Multi-type transformations #1354

Add Integration Tests for E2E Multi-type transformations #1354

Uh oh!

Conversation

lewijacn commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Testing

Check List

Uh oh!

codecov bot commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

peternied left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lewijacn commented Mar 13, 2025 •

edited

Loading

codecov bot commented Mar 14, 2025 •

edited

Loading

peternied left a comment •

edited

Loading