Skip to content

Comments

Fix flaky rollup test by stopping jobs before index cleanup#1530

Open
bowenlan-amzn wants to merge 3 commits intoopensearch-project:mainfrom
bowenlan-amzn:fix-rollup-flaky-2
Open

Fix flaky rollup test by stopping jobs before index cleanup#1530
bowenlan-amzn wants to merge 3 commits intoopensearch-project:mainfrom
bowenlan-amzn:fix-rollup-flaky-2

Conversation

@bowenlan-amzn
Copy link
Member

@bowenlan-amzn bowenlan-amzn commented Nov 17, 2025

Example failure: https://github.com/opensearch-project/index-management/actions/runs/19418611760/job/55551674679?pr=1455

The test was flaky because rollup job coroutines continued running after test cleanup, causing a race condition where the config index was recreated with incorrect dynamic mappings.

Root cause:

  • Tests create enabled rollups -> JobScheduler schedules them
  • RollupRunner.runJob() launches coroutines via launch {}
  • Test @After wipes indices
  • BUT coroutines are still running in background
  • Coroutines write metadata AFTER indices wiped
  • Index auto-creates with wrong dynamic mappings (long vs date)
  • Next test fails with mapping conflict on rollup_metadata.continuous.next_window_end_time

Related Issues

Resolves #90

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@codecov
Copy link

codecov bot commented Nov 17, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.43%. Comparing base (ffe568f) to head (d383ebc).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1530      +/-   ##
==========================================
+ Coverage   76.39%   76.43%   +0.03%     
==========================================
  Files         381      381              
  Lines       18058    18058              
  Branches     2501     2501              
==========================================
+ Hits        13796    13803       +7     
+ Misses       2991     2987       -4     
+ Partials     1271     1268       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bowenlan-amzn bowenlan-amzn marked this pull request as ready for review November 17, 2025 17:22
Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>
@tandonks
Copy link
Collaborator

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky tests

2 participants