Cleanup RFS source and target versions, add support for RFS 5.x, 7.x target with same-version migrations #1373

AndreKurait · 2025-03-20T22:31:19Z

Description

Add rfs-only for 5x->5x, 7.10->7.10 migrations with tests.
Cleanup how work coordination is done, use refresh=wait_for over individual _refresh calls

Issues Resolved

Testing

Unit tests

Check List

New functionality includes testing
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ations Signed-off-by: Andre Kurait <[email protected]>

Signed-off-by: Andre Kurait <[email protected]>

…y across versions Signed-off-by: Andre Kurait <[email protected]>

codecov · 2025-03-21T20:57:44Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (1ea4f15) to head (823efa2).
Report is 20 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #1373       +/-   ##
============================================
- Coverage     80.12%       0   -80.13%     
============================================
  Files           440       0      -440     
  Lines         16184       0    -16184     
  Branches       1078       0     -1078     
============================================
- Hits          12968       0    -12968     
+ Misses         2582       0     -2582     
+ Partials        634       0      -634

Flag	Coverage Δ
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

AndreKurait · 2025-03-21T21:16:46Z

...c/testFixtures/java/org/opensearch/migrations/bulkload/framework/SearchClusterContainer.java

        BASE(Map.of("discovery.type", "single-node",
            "path.repo", CLUSTER_SNAPSHOT_DIR,
-            "ES_JAVA_OPTS", "-Xms512m -Xmx512m")),
+            "ES_JAVA_OPTS", "-Xms2g -Xmx2g",


Had more consistent ES performance with this larger memory and settings below. We should be running all containers in isolatedTests so less concern of contention with parallelization

mikaylathompson · 2025-03-21T21:55:28Z

Wait, do we support/want to support 5.x as a target? Why?

DataGenerator/src/test/java/org/opensearch/migrations/DataGeneratorEndToEndTest.java

...mSnapshotMigration/src/test/java/org/opensearch/migrations/bulkload/LeaseExpirationTest.java

...tsFromSnapshotMigration/src/test/java/org/opensearch/migrations/bulkload/SourceTestBase.java

MetadataMigration/src/test/java/org/opensearch/migrations/CustomTransformationTest.java

...java/org/opensearch/migrations/bulkload/version_es_6_8/OpenSearchWorkCoordinator_ES_6_8.java

...main/java/org/opensearch/migrations/bulkload/workcoordination/OpenSearchWorkCoordinator.java

...t/java/org/opensearch/migrations/bulkload/workcoordination/OpenSearchWorkCoodinatorTest.java

...ensearch/migrations/bulkload/workcoordination/WorkCoordinatorErrantAcquisitonsRetryTest.java

AndreKurait · 2025-03-21T22:28:42Z

Wait, do we support/want to support 5.x as a target? Why?

This is useful to us in an rfs-only support which will allow us to use RFS to generate large snapshots (e.g. 10 TB)

peternied

With this change the runtime goes up to 57m - that seems like a huge jump can we see about reducing that?

gradle-tests (:DocumentsFromSnapshotMigration:jacocoTestReport)
succeeded 3 days ago in 57m 15s

...rc/main/java/org/opensearch/migrations/bulkload/workcoordination/WorkCoordinatorFactory.java

RFS/src/test/java/org/opensearch/migrations/bulkload/workcoordination/WorkCoordinatorTest.java

peternied · 2025-03-24T15:40:28Z

Thanks for creating that draft PR [1] for improving the test stripping - that would reduce my concerns about the test runtime

[1] Implement striping across all tests for GHA #1376

Signed-off-by: Andre Kurait <[email protected]>

peternied · 2025-03-24T16:16:09Z

Thanks for creating that draft PR [1] for improving the test stripping - that would reduce my concerns about the test runtime

[1] Implement striping across all tests for GHA #1376

@AndreKurait Would you mind holding off on merging this change until we've [1] has been merged?

Addressed feedback with #1376 merged into main

sumobrian · 2025-03-27T16:19:24Z

Related fix for supporting ES 5 as a target: https://opensearch.atlassian.net/browse/MIGRATIONS-2464.

AndreKurait had a problem deploying to migrations-cicd March 20, 2025 22:31 — with GitHub Actions Failure

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from 02e04dc to 0fa5a7a Compare March 21, 2025 01:00

AndreKurait temporarily deployed to migrations-cicd March 21, 2025 01:00 — with GitHub Actions Inactive

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from 0fa5a7a to 1d2377d Compare March 21, 2025 14:25

AndreKurait had a problem deploying to migrations-cicd March 21, 2025 14:25 — with GitHub Actions Failure

Add experamental support in RFS for same version to same version migr…

e50862e

…ations Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from 1d2377d to e50862e Compare March 21, 2025 14:26

AndreKurait had a problem deploying to migrations-cicd March 21, 2025 14:27 — with GitHub Actions Failure

Disable WorkCoordinatorTest for ES 5 and 6

ff63a4c

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait had a problem deploying to migrations-cicd March 21, 2025 14:51 — with GitHub Actions Failure

SpotlessFix

5a3438e

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait had a problem deploying to migrations-cicd March 21, 2025 15:31 — with GitHub Actions Failure

AndreKurait temporarily deployed to migrations-cicd March 21, 2025 15:33 — with GitHub Actions Inactive

AndreKurait had a problem deploying to migrations-cicd March 21, 2025 20:16 — with GitHub Actions Failure

Update how refresh is handled in the work coordinators for reliabilit…

085ef1e

…y across versions Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from 64cd4b1 to 085ef1e Compare March 21, 2025 20:21

AndreKurait temporarily deployed to migrations-cicd March 21, 2025 20:21 — with GitHub Actions Inactive

AndreKurait temporarily deployed to migrations-cicd March 21, 2025 21:15 — with GitHub Actions Inactive

AndreKurait changed the title ~~Add experimental support in RFS for same version to same version migrations~~ Add experimental support in RFS for same version to same version migrations Mar 21, 2025

AndreKurait commented Mar 21, 2025

View reviewed changes

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from 114adf0 to adae298 Compare March 21, 2025 21:31

AndreKurait temporarily deployed to migrations-cicd March 21, 2025 21:31 — with GitHub Actions Inactive

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from adae298 to 9adf692 Compare March 21, 2025 21:35

AndreKurait temporarily deployed to migrations-cicd March 21, 2025 21:35 — with GitHub Actions Inactive

AndreKurait marked this pull request as ready for review March 21, 2025 21:35

AndreKurait requested review from chelma, gregschohn, lewijacn and mikaylathompson as code owners March 21, 2025 21:35

AndreKurait changed the title ~~Add experimental support in RFS for same version to same version migrations~~ Cleanup RFS source and target versions Mar 21, 2025

mikaylathompson reviewed Mar 21, 2025

View reviewed changes

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from 9adf692 to 61e693f Compare March 21, 2025 22:27

AndreKurait temporarily deployed to migrations-cicd March 21, 2025 22:28 — with GitHub Actions Inactive

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from 61e693f to a7f3a41 Compare March 21, 2025 22:34

AndreKurait temporarily deployed to migrations-cicd March 21, 2025 22:34 — with GitHub Actions Inactive

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from a7f3a41 to 4d87ee5 Compare March 21, 2025 22:34

AndreKurait temporarily deployed to migrations-cicd March 21, 2025 22:35 — with GitHub Actions Inactive

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from 4d87ee5 to 0a342b4 Compare March 21, 2025 22:35

AndreKurait temporarily deployed to migrations-cicd March 21, 2025 22:35 — with GitHub Actions Inactive

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from 0a342b4 to 9af9f68 Compare March 21, 2025 22:36

AndreKurait temporarily deployed to migrations-cicd March 21, 2025 22:36 — with GitHub Actions Inactive

AndreKurait changed the title ~~Cleanup RFS source and target versions~~ Cleanup RFS source and target versions, add support for RFS 5.x, 7.x target with same-version migrations Mar 24, 2025

mikaylathompson approved these changes Mar 24, 2025

View reviewed changes

peternied previously requested changes Mar 24, 2025

View reviewed changes

...rc/main/java/org/opensearch/migrations/bulkload/workcoordination/WorkCoordinatorFactory.java Show resolved Hide resolved

RFS/src/test/java/org/opensearch/migrations/bulkload/workcoordination/WorkCoordinatorTest.java Outdated Show resolved Hide resolved

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from 9af9f68 to cde7841 Compare March 24, 2025 15:49

AndreKurait temporarily deployed to migrations-cicd March 24, 2025 15:49 — with GitHub Actions Inactive

Address PR Comments

0be43e3

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the AddES5and7TargetExpirimentalRFSOnlySupport branch from cde7841 to 0be43e3 Compare March 24, 2025 15:56

AndreKurait temporarily deployed to migrations-cicd March 24, 2025 15:56 — with GitHub Actions Inactive

Merge branch 'main' into AddES5and7TargetExpirimentalRFSOnlySupport

823efa2

AndreKurait temporarily deployed to migrations-cicd March 24, 2025 21:56 — with GitHub Actions Inactive

AndreKurait merged commit 3def26a into opensearch-project:main Mar 25, 2025
54 checks passed

AndreKurait deleted the AddES5and7TargetExpirimentalRFSOnlySupport branch March 25, 2025 00:31

Cleanup RFS source and target versions, add support for RFS 5.x, 7.x target with same-version migrations #1373

Cleanup RFS source and target versions, add support for RFS 5.x, 7.x target with same-version migrations #1373

Uh oh!

Conversation

AndreKurait commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Testing

Check List

Uh oh!

codecov bot commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

AndreKurait Mar 21, 2025

Choose a reason for hiding this comment

Uh oh!

mikaylathompson commented Mar 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AndreKurait commented Mar 21, 2025

Uh oh!

peternied left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

peternied commented Mar 24, 2025

Uh oh!

peternied commented Mar 24, 2025

Uh oh!

Uh oh!

sumobrian commented Mar 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AndreKurait commented Mar 20, 2025 •

edited

Loading

codecov bot commented Mar 21, 2025 •

edited

Loading