-
Notifications
You must be signed in to change notification settings - Fork 128
Open
Labels
Description
Due to CI pipeline being seriously affected by a number of LS specific tests that seem to randomly fail, we were forced to disabled them until they are reworked to be reliable.
- lightspeedUiTestPlaybookExpTest
- "Playbook explanation webview with a playbook with no tasks"
- "Playbook explanation thumbs up/down button disabled after thumbs up"
- "Playbook explanation thumbs up/down button disabled after thumbs down"
- "Ansible Lightspeed status bar item absent when settings not enabled"
- "Ensures we can go from DevTools to Playbook generation"
- presence of lightspeed element in the status bar and the explorer view (all iterations)
- Verify playbook explanation features when no explanation is returned
- Verify playbook generation features work as expected
Full list of flaky tests listed here on Codecov Junit reports: https://app.codecov.io/github/ansible/vscode-ansible/tests?term=Lightspeed
Enhancements to assist/augment this:
The current way we call UI tests (test-launcher.sh) is complex and devtools is working on remediation (some work is already in progress).
- ensure that extest/mocha itself would start LS mock server instead of using the bash script
- get rid of the need to copy/reset settings.json before each test
- eliminate (most) of the UI tests that change settings using UI and instead have already configured settings.json for each test. This should improve the test speed considerably. The only exception would be if we have a case where changing a setting would not get applied right away but I am not sure if we need this particular case. The reality is that vscode knows to reload settings.json automatically even if is modified by another process, so we could replace the UI settings modification with a simple settings.json swap inside the same test, making the execution much faster.
- LS code is using the unmaintained webview-ui-toolkit (1.5y)
- This prevents or makes it harder each update of the ui testing framework (extest) because these UI test would fail and require manual updates which are not straight forward.
- security issues, webview-ui-toolkit uses some already compromised dependencies.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
No status