Skip to content

[codex] Fix flaky CACM reproduction test assertion#3324

Open
lintool wants to merge 1 commit into
masterfrom
codex/fix-cacm-reproduction-flaky-test
Open

[codex] Fix flaky CACM reproduction test assertion#3324
lintool wants to merge 1 commit into
masterfrom
codex/fix-cacm-reproduction-flaky-test

Conversation

@lintool

@lintool lintool commented Jun 27, 2026

Copy link
Copy Markdown
Member

Summary

  • Fix a flaky assertion in ReproduceFromPrebuiltIndexesTest.testCacmEndToEnd.
  • Allow the pre-run index-size summary to report either 0 of 1 or 1 of 1 CACM indexes.
  • Assert that the generated CACM run file exists and is non-empty.

Root Cause

ReproduceFromPrebuiltIndexes prints the index-size table before retrieval runs. If CACM is already cached, the table says Total size across 1 of 1 indexes; if the cache is cold, it says 0 of 1 indexes.

That cache state is not the actual end-to-end success condition. The child SearchCollection command can lazily fetch/open the CACM prebuilt index after the summary is printed, then produce the expected run and metrics. The old assertion therefore depended on whether the local cache had been warmed by a prior run.

Validation

  • mvn -Dtest=ReproduceFromPrebuiltIndexesTest#testCacmEndToEnd -Dpyserini.cache=tmp/pyserini-cache-flake-check test
  • mvn -Dtest=ReproduceFromPrebuiltIndexesTest test

Both passed.

@codecov

codecov Bot commented Jun 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.40%. Comparing base (78e7e16) to head (5c33c2b).

Additional details and impacted files
@@            Coverage Diff            @@
##             master    #3324   +/-   ##
=========================================
  Coverage     73.39%   73.40%           
- Complexity     1912     1914    +2     
=========================================
  Files           206      206           
  Lines         13493    13493           
  Branches       1867     1867           
=========================================
+ Hits           9903     9904    +1     
  Misses         2843     2843           
+ Partials        747      746    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@lintool lintool marked this pull request as ready for review June 28, 2026 11:15
@lintool lintool requested review from lilyjge and sahel-sh June 28, 2026 11:15
@lilyjge

lilyjge commented Jun 28, 2026

Copy link
Copy Markdown
Member

hm, seems like this is a misleading bug we should fix in ReproduceFromPrebuiltIndexes instead of patching the test?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants