Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1303 +/- ##
==========================================
+ Coverage 80.08% 86.44% +6.36%
==========================================
Files 20 99 +79
Lines 984 6685 +5701
Branches 0 111 +111
==========================================
+ Hits 788 5779 +4991
- Misses 196 850 +654
- Partials 0 56 +56 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
adombeck
force-pushed
the
fix-e2e-tests
branch
4 times, most recently
from
9163911 to
7d85d7d
Compare
adombeck
changed the title
adombeck
force-pushed
the
fix-e2e-tests
branch
3 times, most recently
from
15ca5f3 to
6c29807
Compare
YARF combines the screenshots it takes for image matching into a video. That misses everything that happens in between those screenshots. Let's record an actual video of the VNC output instead.
By default, VNC viewers can ask the VNC server for an exclusive connection. That would break YARF and the recording of the VNC output. Setting the share policy of the VNC server to "force-shared" disallows exclusive connections.
When using logger.error, the videos of all failing test cases are printed in the separate "Execution Errors" section of the HTML log file, without any context of which test case the video is from and the relevant error messages which caused the test case to fail. By using set_test_message, the video is printed as part of the error message of the test case.
We use the filename as the titles of the videos. They look better in uppercase.
Now that we use set_test_message instead of logger.error to log the video, we can't use the 'console=False' argument anymore which we added to Robot Framework. Work around that by filtering out the video from the output of run-tests.sh.
We log a video ourselves in a better way.
Sometimes the commit status is not created. Let's see if retrying helps.
The name was a bit confusing because steps starting with "Post" are usually from GitHub Actions using the 'runs.post' keyword to execute steps at the end of the job.
Should help us debug the cases where the commit status is not created.
Avoid creating an error annotation: Process completed with exit code <exit code> and instead create an error annotation with a link to the HTML logs.
Use a name that's consistent with the URL we use to publish the HTML log.
Seems like Microsoft changed the URL they return in the verification_uri field of the Device Authorization Response.
The OCR didn't recognize the string "Google" on the GDM login screen. Let's improve the OCR results by increasing the text size (same as in the user session).
Redacting the secrets by post-processing the HTML log files didn't work correctly, because Robot Framework stores most of the file content in as gzipped compressed JavaScript. Let's avoid logging the secrets in the first place. To achieve that, we pass them via environment variables instead of the command-line, so that they are not included in the log message which prints the failed command.
As explained in the previous commit, the redaction didn't work correctly. Let's avoid giving a false sense of security and remove the incomplete attempt to redact the secrets.
The regex allowed backticks,literal question marks and numbers. The Google user code can only contain uppercase letters and hyphens. We also allow lowercase letters because the OCR sometimes reads letters as lowercase (as explained in the code comment).
I've seen
subprocess.TimeoutExpired: Command '[...]' timed out after 30 seconds
without the command's stderr, which makes it hard to debug. Let's print
also print the command's stderr in that case.
I've seen the command time out after 30 seconds.
adombeck
force-pushed
the
fix-e2e-tests
branch
4 times, most recently
from
7592f43 to
5deff65
Compare
Cloudflare R2 has several benefits over GitHub Pages: * Allows access control * Logs don't have to be committed to the git repo (the repo size increased significantly, even if we delete the logs from closed PRs). * Shorter deployment delay. With GitHub it took several minutes before the logs could be accessed after deployment. Cloudflare Pages would also have been an option, but it doesn't allow directory listing for easy navigation of the uploaded logs.
Allows browsing the directories containing the log files.
Makes it easier to navigate to the logs of other CI jobs
I regularly see the setup-oras action fail. Let's see if installing it via Go is more robust.
Installing via Go also didn't work well, the most efficient way is to install it directly from the GitHub releases using the GITHUB_TOKEN to avoid rate limiting.
I've seen it time out.
Contributor
Author
|
A few notes on the failing CI jobs:
|
adombeck
commented
Mar 11, 2026
denisonbarbosa
approved these changes
Mar 11, 2026
denisonbarbosa
approved these changes
Mar 11, 2026
Member
denisonbarbosa
left a comment
denisonbarbosa
left a comment
There was a problem hiding this comment.
Wow, this was a big one. A lot of great changes and improvements!
Less code duplication
Avoid errors during browser_login.py ending with: During handling of the above exception, another exception occurred: [...] TimeoutError: Timed out waiting for text: "Unacceptable TLS certificate"
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix many issues causing e2e-tests to fail. See commit messages for details.
UDENG-9381
UDENG-9382