-
Notifications
You must be signed in to change notification settings - Fork 574
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Try to stabilize tests in CI #1128
Conversation
|
c4b5980
to
42c419e
Compare
src/services/checkpoints/__tests__/ShadowCheckpointService.test.ts
Outdated
Show resolved
Hide resolved
working-directory: webview-ui | ||
run: npx jest | ||
|
||
unit-test: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have a branch rule dictating that "unit-test" must pass in order for the PR to be merged, hence this composite job.
02c54e4
to
bbd8a96
Compare
"jest": "^27.5.1", | ||
"jest-environment-jsdom": "^27.5.1", | ||
"jest-simple-dot-reporter": "^1.0.5", | ||
"jest": "^29.7.0", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might as well match the jest and ts-jest versions that we use to test the extension.
18cdf84
to
9b734c5
Compare
@@ -1,3 +1,5 @@ | |||
/// <reference types="jest" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since @types/mocha
and @types/jest
both define expect
, beforeEach
, afterEach
, etc we need to indicate which version of these global functions are being used. I'm not sure what the "correct" fix for this is, but it's unfortunate that we have this conflict.
Mocha is only needed for integration tests, and it seems like that could live in a separate package (kind of like webview-ui
).
2e6bda9
to
f11c0af
Compare
@@ -1,3 +1,4 @@ | |||
{ | |||
"extends": "react-app" | |||
"extends": "react-app", | |||
"ignorePatterns": ["!.storybook"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@@ -275,7 +277,7 @@ describe("applyGitFallback", () => { | |||
expect(result.result.join("\n")).toEqual("line1\nnew line2\nline3") | |||
expect(result.confidence).toBe(1) | |||
expect(result.strategy).toBe("git-fallback") | |||
}) | |||
}, 10_000) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So this implies that whatever we’re doing with git in the experimental diff algo is taking between 5 and 10 seconds?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I encountered this timeout while debugging, and increasing it seemed to suddenly make the problems go away. Undoing this change seemed to make the problem reliably come back. One theory I had was that it impacted the order or sequencing of the tests such that the failure stopped happening. I added the --randomize flag to jest to test that theory but we get a bunch of other failures since some of our tests do seem to rely on being run in a particular order (which we should fix).
Description
applyGitFallback
testsjest
andts-jest
inwebview-ui
I used this to debug the flakey test on the CI machine, but it seemed to run fine in isolation.
I first tried increasing the V8 heap size, but that just allowed the tests to run long enough to hit some jest timeouts.
Next I tried logging the performance of the shadow checkpointer tests, and I found that simply disabling the dot-reporter and enabling console.log calls seemed to fix things. That also turned out to be a red herring.
Digging a little bit deeper, it seems like there's an interaction between the shadow checkpointer and the
applyGitFallback
function, which isn't surprising because they both create git repositories, and one might end up creating a nested git repo which could cause problems. Increasing the timeout on theapplyGitFallback
tests seem to mitigate the issue, but we'll see if this continues to be flakey with future CI runs. As a follow-up, I will write additional tests to validate that the shadow checkpoint correctly handles nested git repos. It's possible that the tests environment is problematic b/c we are mockingglobby
.Type of change
How Has This Been Tested?
Checklist:
Additional context
Related Issues
Reviewers
Important
Stabilize CI tests by improving isolation, increasing timeouts, separating jobs, and cleaning logs.
test-extension
andtest-webview
jobs in.github/workflows/code-qa.yml
.npx jest --silent
instead ofjest-simple-dot-reporter
for cleaner logs.applyGitFallback
tests inedit-strategies.test.ts
.LocalCheckpointService.test.ts
andShadowCheckpointService.test.ts
.jest-simple-dot-reporter
frompackage.json
andwebview-ui/package.json
.This description was created by
for 2e6bda9. It will automatically update as commits are pushed.