Skip to content

Stabilizing UI tests after migration to Cody agent v1.66 and v1.114 - #278

Merged
PiotrKarczmarz merged 30 commits into
mainfrom
piotr/updating-ui-tests-to-match-agent
Aug 18, 2025
Merged

Stabilizing UI tests after migration to Cody agent v1.66 and v1.114#278
PiotrKarczmarz merged 30 commits into
mainfrom
piotr/updating-ui-tests-to-match-agent

Conversation

@PiotrKarczmarz

@PiotrKarczmarz PiotrKarczmarz commented May 21, 2025

Copy link
Copy Markdown
Contributor

CLOSE https://linear.app/sourcegraph/issue/CODY-5899/stabilizing-ui-tests-after-migration-to-cody-agent-v166

1. UI Test Improvements

  • Updated Playwright test selectors to use data-testid attributes instead of roles/text for more reliable selection
  • Added new test for verifying line selection changes in chat input - Active_File_Name_And_Line_Selection_Is_Changing_In_Chat_Input
  • Fixed and enabled previously skipped tests
  • Improved screenshot capture for better debugging of test failures
  • Added more detailed test output with additional logging

2. Solution Loading and Document Handling

  • Enhanced solution loading process with better verification that solution is fully loaded
  • Added CloseAllDocuments() to ensure clean state between tests

3. Authentication and Token Handling

  • Added AccessTokenRefreshed event to ISecretStorageService
  • Implemented proper waiting for login and logout states

4. General Stability Improvements

  • Better handling of chat loading state with explicit checks for loading indicators
  • More robust waiting for various UI states

Test plan

  1. Green UI Tests in CI

@github-actions

github-actions Bot commented May 21, 2025 •
edited
Loading

Copy link
Copy Markdown

Test Results

24 tests  +24   24 ✅ +24   2m 55s ⏱️ + 2m 55s
 1 suites + 1    0 💤 ± 0 
 1 files   + 1    0 ❌ ± 0 

Results for commit f615742. ± Comparison against base commit 5eb005f.

♻️ This comment has been updated with latest results.

@PiotrKarczmarz
PiotrKarczmarz force-pushed the piotr/updating-ui-tests-to-match-agent branch from 73fc3a1 to 05fb47f Compare August 14, 2025 10:51
@PiotrKarczmarz
Update UI test to work with agent v1.88.0:
902c554 
  - added Active_File_Name_And_Line_Selection_Is_Changing_In_Chat_Input() test
@PiotrKarczmarz
Forcing agent configuration change after access token refresh.
0da4bb7
@PiotrKarczmarz
Turned off tests from ChatLoggedBasicTests
ce7e7ba
@PiotrKarczmarz
Turned on Solution_Name_Is_Added_To_Chat_Input()
f33f051
@PiotrKarczmarz
Additional layer check to wait for the solution to fully load.
195d4e1
@PiotrKarczmarz
Waiting for Cody chat to be fully loaded and initialized.
a7dec4b
@PiotrKarczmarz
Added additional step when taking a screenshot.
47fab3f
@PiotrKarczmarz
Fixed Solution_Name_Is_Added_To_Chat_Input() to search only first tag…
22e92ae 
… (no files open).
@PiotrKarczmarz
Closing all opened documents after opening a solution to have a clean…
c0701eb 
… state (they are remembered between open solution "sessions").
@PiotrKarczmarz
Add call to CloseAllDocuments()
fb5e7cf
@PiotrKarczmarz
Logging tags found in the chat.
5d431f0
@PiotrKarczmarz
Support custom ToString() for ContextTag
f88f20a
@PiotrKarczmarz
Additional logging.
d50f5e1
@PiotrKarczmarz
Fixed waiting for log in/log out states.
ab29fba
@PiotrKarczmarz
Fixed WaitForLogInState().
072a310
@PiotrKarczmarz
Fixed build.
ea3a829
@PiotrKarczmarz
Turn on Cody_Enterprise_Section_Is_Present()
880db42
@PiotrKarczmarz
Turned on Logins_With_GitLab_Google_Are_Present()
000be69
@PiotrKarczmarz
Turned on Can_Chat_Tool_Window_Be_Closed_And_Opened_Again()
920d98d
@PiotrKarczmarz
Removed PLG related tests (Free/Pro accounts)
a4d7d31 
Updated agent to vscode-v1.114.0
DefaultServerEndpoint is now pointing to sourcegraph.sourcegraph.com
@PiotrKarczmarz
Using access token for S2.
bd5e805
@PiotrKarczmarz
Fixed issue with tags no cleared after closing all files at IDE level.
72ae454
@PiotrKarczmarz
Turned on Active_File_Name_And_Line_Selection_Is_Showing_In_Chat_Input()
d7aab84
@PiotrKarczmarz
Turned on other tests.
e592237
@PiotrKarczmarz
Fixes.
c3c3846
@PiotrKarczmarz
Turned off Entered_Prompt_Show_Up_In_Today_History()
5b9b1d2
@PiotrKarczmarz
PiotrKarczmarz force-pushed the piotr/updating-ui-tests-to-match-agent branch from d3f2a2a to 5b9b1d2 Compare August 15, 2025 11:23
@PiotrKarczmarz
PiotrKarczmarz force-pushed the piotr/updating-ui-tests-to-match-agent branch from 269c2d2 to 5b9b1d2 Compare August 15, 2025 18:15
@PiotrKarczmarz PiotrKarczmarz changed the title Stabilizing UI tests after migration to Cody agent v1.66 Stabilizing UI tests after migration to Cody agent v1.66 an v1.114 Aug 18, 2025
@PiotrKarczmarz PiotrKarczmarz changed the title Stabilizing UI tests after migration to Cody agent v1.66 an v1.114 Stabilizing UI tests after migration to Cody agent v1.66 and v1.114 Aug 18, 2025
@PiotrKarczmarz
PiotrKarczmarz merged commit 3d21dbe into main Aug 18, 2025
13 checks passed
@PiotrKarczmarz
PiotrKarczmarz deleted the piotr/updating-ui-tests-to-match-agent branch August 18, 2025 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers
@tomaszgolebiowski tomaszgolebiowski tomaszgolebiowski approved these changes
@abeatrix abeatrix abeatrix approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@PiotrKarczmarz @tomaszgolebiowski @abeatrix