Skip to content

Improve cypress integration test performance and fix flaky tests#3350

Open
lresende wants to merge 16 commits intoelyra-ai:mainfrom
lresende:fix-integration-test-timeout
Open

Improve cypress integration test performance and fix flaky tests#3350
lresende wants to merge 16 commits intoelyra-ai:mainfrom
lresende:fix-integration-test-timeout

Conversation

@lresende
Copy link
Copy Markdown
Member

@lresende lresende commented Mar 16, 2026

What changes were proposed in this pull request?

Improve cypress integration test performance and fix flaky tests

Fix flaky integration tests by replacing ~30 cy.wait() calls across all test files with proper Cypress retry-able assertions. Key changes:

  • Fix openFileAndCheckContent race condition using .should('contain.text')
  • Add editor/launcher visibility waits in openHelloWorld and resetJupyterLab
  • Replace savePipeline wait with dirty-state check
  • Scope uncaught exception handler to known benign errors
  • Increase defaultCommandTimeout from 8s to 10s

Fixes #3301

How was this pull request tested?

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 16, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@737bc78). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3350   +/-   ##
=======================================
  Coverage        ?   84.16%           
=======================================
  Files           ?      157           
  Lines           ?    19067           
  Branches        ?      504           
=======================================
  Hits            ?    16048           
  Misses          ?     2827           
  Partials        ?      192           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lresende lresende force-pushed the fix-integration-test-timeout branch 2 times, most recently from eaf7544 to 0db71ba Compare March 16, 2026 15:56
@lresende lresende changed the title Fix flaky cypress integration tests by removing hard-coded cy.wait() Improve cypress integration test performance and fix flaky tests Mar 16, 2026
@lresende lresende force-pushed the fix-integration-test-timeout branch from 0db71ba to 8fbd1e0 Compare March 20, 2026 05:05
@lresende lresende force-pushed the fix-integration-test-timeout branch 2 times, most recently from 61f12da to afbc593 Compare April 3, 2026 05:57
@lresende lresende force-pushed the fix-integration-test-timeout branch 4 times, most recently from d2d3ad7 to 6da2022 Compare April 12, 2026 23:10
lresende added 11 commits April 12, 2026 16:11
Replace the blanket return false that suppressed all uncaught
exceptions with an allowlist of known benign JupyterLab errors.
Unknown errors now propagate and fail the test, surfacing real
bugs that were previously hidden.

Signed-off-by: Luciano Resende <lresende@apple.com>
Replace cy.wait(1000) with a PUT request intercept guarded by
a dirty-state check. When the document is dirty, wait for the
server write to complete; when clean, skip the wait to avoid
hanging. Confirm the document is no longer dirty afterward.

Signed-off-by: Luciano Resende <lresende@apple.com>
Callers already close the tab after calling this command. The
internal closeTab caused "element not found" failures when the
tab was already gone. Also replaced synchronous content extraction
with a retryable should('contain.text') assertion.

Signed-off-by: Luciano Resende <lresende@apple.com>
Batch multiple file deletion patterns into a single find command
instead of spawning N individual cy.exec() calls. This reduces
setup/teardown overhead in tests that clean up many file types.

Signed-off-by: Luciano Resende <lresende@apple.com>
- createPipeline: replace cy.wait(300) with
  should('be.visible') on canvas drop div
- resetJupyterLab: add jp-Launcher visibility check so
  callers don't need their own cy.wait after reset

Signed-off-by: Luciano Resende <lresende@apple.com>
- Replace cy.wait(300) with should('be.visible') on canvas
  drop div for consistent pipeline editor readiness
- Guard runtime image select() calls with option existence
  checks to prevent failures when SWR fetch hasn't resolved
- Extract selectRuntimeImage helper to reduce duplication

Signed-off-by: Luciano Resende <lresende@apple.com>
- Remove ~20 cy.wait() calls replaced with proper Cypress
  retryable assertions and DOM readiness checks
- Sidebar switches now verified via .lm-mod-current selector
- createValidCodeSnippet waits for data-item-id to appear
- Editor readiness uses .elyra-ScriptEditor should('be.visible')
- saveAndCloseMetadataEditor no longer has a trailing wait

Signed-off-by: Luciano Resende <lresende@apple.com>
- Replace cy.wait(2000) with kernel idle status check
  (data-status="idle") before interacting with cells
- Fix populateCells() stale DOM: re-query cells by index
  instead of using .each() with stale wrappers
- Add openCellContextMenuWithSnippetItem() retry helper
  that dismisses and re-opens the menu up to 5 times
  until the extension command is registered
- Use body click to dismiss menus instead of Esc key

Signed-off-by: Luciano Resende <lresende@apple.com>
- pythoneditor: replace cy.wait(1000) for editor load with
  cm-content visibility check with timeout
- pythoneditor: replace cy.wait(2000) for output with 15s
  timeout on output area assertion
- pythoneditor: keep cy.wait(500) for LSP debounce with comment
- submitnotebook: remove redundant cy.wait(2000) after reset
  since resetJupyterLab now waits for Launcher
- submitnotebook: replace cy.wait(500) with notebook existence
  check after opening new notebook

Signed-off-by: Luciano Resende <lresende@apple.com>
- Use deleteFiles batch command to reduce 12 individual
  cy.exec() calls to 2 batched find commands
- Combine 3 elyra-metadata remove calls into a single
  shell command to reduce process spawn overhead

Signed-off-by: Luciano Resende <lresende@apple.com>
- Bump defaultCommandTimeout from 8s to 10s to reduce
  spurious timeouts on slow CI runners
- Lower numTestsKeptInMemory from 10 to 5 to reduce
  memory pressure during long test runs
- Enable experimentalMemoryManagement for GC optimization
- Switch cy:run to headless mode (~20% faster, no X11)
- Add cy:debug script for local headed debugging

Signed-off-by: Luciano Resende <lresende@apple.com>
@lresende lresende force-pushed the fix-integration-test-timeout branch from 6da2022 to b8a9913 Compare April 12, 2026 23:11
Signed-off-by: Luciano Resende <lresende@apple.com>
Signed-off-by: Luciano Resende <lresende@apple.com>
The openCellContextMenuWithSnippetItem helper asserted the
context menu was 'visible', but in headless Electron the menu
can render with 0x0 dimensions initially. The empty cell test
does not need the retry helper since the snippet command is
already registered (just disabled). Use a direct rightclick
with a timeout on the specific menu item instead. Also change
the retry helper's menu assertion from 'be.visible' to 'exist'
for headless compatibility.

Signed-off-by: Luciano Resende <lresende@apple.com>
In headless mode, the launcher takes longer to appear after
closing the last tab. Add a launcher visibility check before
clicking the launcher card to prevent cascading failures when
the launcher hasn't rendered yet between tests.

Signed-off-by: Luciano Resende <lresende@apple.com>
The extension command registers asynchronously even for empty
cells, so the retry helper (dismiss and re-open menu) is needed
here too. The previous fix incorrectly assumed the command was
already registered for empty cells.

Signed-off-by: Luciano Resende <lresende@apple.com>
@lresende lresende force-pushed the fix-integration-test-timeout branch 2 times, most recently from 7ba05c1 to 6cb8072 Compare April 13, 2026 05:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Review and fix broken and flaky integration tests

1 participant