Skip to content

Conversation

bowenlan-amzn
Copy link
Member

@bowenlan-amzn bowenlan-amzn commented Oct 3, 2025

Description

Fixes a flaky test in QueryPhaseTests.testMaxScoreWithSortOnScoreAndCollapsingResults that fails intermittently with certain random seeds, like A86CC83E4E8823F2.

./gradlew ':server:test' --tests 'org.opensearch.search.query.QueryPhaseTests' \
    -Dtests.method='testMaxScoreWithSortOnScoreAndCollapsingResults' \
    -Dtests.seed=A86CC83E4E8823F2

Example failure: https://build.ci.opensearch.org/job/gradle-check/64947/testReport/junit/org.opensearch.search.query/QueryPhaseTests/testMaxScoreWithSortOnScoreAndCollapsingResults__p0_5_p1_org_opensearch_search_query_ConcurrentQueryPhaseSearcher_1f6ba47b_/

Test came from #19181

The test was making order-dependent assertions on collapsed search results:

assertThat(topDocs.collapseValues[0], equalTo(0L)); // user == 0
assertThat(topDocs.collapseValues[1], equalTo(1L)); // user == 1

However, the order of collapsed results is non-deterministic when documents have equal scores because:

  1. All test documents match the query foo:bar with identical scores
  2. Documents are indexed with alternating user values (0, 1, 0, 1, ...)
  3. RandomIndexWriter may reorder/merge documents during indexing based on the random seed
  4. With certain seeds (e.g., A86CC83E4E8823F2), documents with user=1 end up at lower doc IDs than user=0
  5. When collapsing on the user field with equal scores, the document encountered first (lower doc ID) is returned
  6. This causes the collapse values to be [1, 0] instead of the expected [0, 1]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Contributor

github-actions bot commented Oct 3, 2025

❕ Gradle check result for 2d9b1d3: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Copy link

codecov bot commented Oct 3, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.01%. Comparing base (ac6dfa1) to head (2d9b1d3).

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #19513      +/-   ##
============================================
- Coverage     73.09%   73.01%   -0.08%     
+ Complexity    70553    70484      -69     
============================================
  Files          5716     5716              
  Lines        322926   322926              
  Branches      46770    46770              
============================================
- Hits         236032   235786     -246     
- Misses        67882    68102     +220     
- Partials      19012    19038      +26     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bowenlan-amzn
Copy link
Member Author

@gaobinlong please take a look when you got time

@gaobinlong
Copy link
Contributor

Thanks @bowenlan-amzn, by comparing with testCollapseQuerySearchResults(), I found that index sorting is missing in this method, so add an index sorting configuration to the IndexWriter is a better solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants