[test optimization] Prevent ATF tests from being incorrectly tagged as new in cucumber#7961
Conversation
…n cucumber Same bug as jest: the shared `numRetriesByPickleId` map was used to determine if a test is new, but ATF tests are also added to this map. Introduces a dedicated `newTestPickleIds` Set for genuine new-test tracking. Also adds regression integration tests for all frameworks (cucumber, mocha, playwright, vitest) that assert ATF tests present in the known tests list are never tagged with `test.is_new`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Overall package sizeSelf size: 5.48 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.0.1 | 82.56 kB | 817.39 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
|
✅ Tests 🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: ee30798 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback! |
BenchmarksBenchmark execution time: 2026-04-10 08:52:39 Comparing candidate commit ee30798 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 228 metrics, 32 unstable metrics. |
juan-fernandez
changed the title
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #7961 +/- ##
=======================================
Coverage 73.82% 73.82%
=======================================
Files 773 773
Lines 35972 35974 +2
=======================================
+ Hits 26556 26559 +3
+ Misses 9416 9415 -1 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
juan-fernandez
deleted the
juan-fernandez/fix-atf-is-new-other-frameworks
branch
…s new in cucumber (#7961)
2 participants
What does this PR do?
Fixes the same bug as #7960 but in the Cucumber instrumentation: attempt-to-fix (ATF) tests were incorrectly tagged as
test.is_new: truewhen the known tests feature was enabled. The sharednumRetriesByPickleIdmap was used to determine if a test was new, but ATF tests are also added to this map.Introduces a dedicated
newTestPickleIdsSet populated only when a test is genuinely absent from the known tests list.Also adds regression integration tests across all test frameworks (Cucumber, Mocha, Playwright, Vitest) that assert ATF tests present in the known tests list are never tagged with
test.is_new.Motivation
Same root cause as #7960 (Jest fix). The Cucumber instrumentation had the identical pattern of using a shared retry-tracking map (
numRetriesByPickleId) to determine if a test is new, which incorrectly detected ATF tests as new. This caused EFD's lenient pass criteria to override ATF's strict criteria.Mocha, Playwright, and Vitest are not affected (they use dedicated data structures for new-test tracking), but regression tests are added for all frameworks to prevent future regressions.
Additional Notes