This repository was archived by the owner on Aug 1, 2025. It is now read-only.
e2e-flakiness-detector #502
Annotations
13 errors, 2 warnings, and 2 notices
|
command-edit.test.ts:33:3 › edit (fixup) task:
vscode\test\e2e\command-edit.test.ts#L104
6) command-edit.test.ts:33:3 › edit (fixup) task ─────────────────────────────────────────────────
Error: expect(locator).not.toBeVisible()
Locator: getByText('appleName')
Expected: not visible
Received: visible
Call log:
- expect.not.toBeVisible with timeout 5000ms
- waiting for getByText('appleName')
- locator resolved to <span class="monaco-highlighted-label">appleName</span>
- unexpected value "visible"
- locator resolved to <span class="monaco-highlighted-label">appleName</span>
- unexpected value "visible"
- locator resolved to <span class="monaco-highlighted-label">appleName</span>
- unexpected value "visible"
102 | await page.keyboard.press('Enter')
103 | await nap()
> 104 | await expect(page.getByText('appleName')).not.toBeVisible()
| ^
105 | await expect(page.getByText('bananaName')).toBeVisible()
106 |
107 | // Accept: remove all the changes made by edit
at D:\a\cody\cody\vscode\test\e2e\command-edit.test.ts:104:51
|
|
command-edit.test.ts:33:3 › edit (fixup) task:
vscode\test\e2e\command-edit.test.ts#L0
6) command-edit.test.ts:33:3 › edit (fixup) task ─────────────────────────────────────────────────
Error: EBUSY: resource busy or locked, rename 'D:\a\cody\cody\playwright\edit__fixup__task\temp-videos\ddf0470b37d5cf446d54a1383768fb6c.webm' -> 'D:\a\cody\cody\playwright\edit__fixup__task\videos\run_0_retry_0_failure.webm'
|
|
command-edit.test.ts:33:3 › edit (fixup) task:
vscode\test\e2e\command-edit.test.ts#L53
6) command-edit.test.ts:33:3 › edit (fixup) task ─────────────────────────────────────────────────
Error: Timed out 5000ms waiting for expect(locator).toBeVisible()
Locator: getByLabel('$(anthropic-logo) Claude 3.5 Sonnet').locator('a')
Expected: visible
Received: hidden
Call log:
- expect.toBeVisible with timeout 5000ms
- waiting for getByLabel('$(anthropic-logo) Claude 3.5 Sonnet').locator('a')
- locator resolved to <a class="label-name">…</a>
- unexpected value "hidden"
- locator resolved to <a class="label-name">…</a>
- unexpected value "hidden"
- locator resolved to <a class="label-name">…</a>
- unexpected value "hidden"
- locator resolved to <a class="label-name">…</a>
- unexpected value "hidden"
51 | const modelItem = page.getByLabel('$(anthropic-logo) Claude 3.5 Sonnet').locator('a')
52 | await nap()
> 53 | expect(modelItem).toBeVisible()
| ^
54 |
55 | // Open the model input and check it has the correct item selected
56 | await modelItem.click()
at D:\a\cody\cody\vscode\test\e2e\command-edit.test.ts:53:23
|
|
command-commit.test.ts:23:3 › use terminal output as context:
vscode\test\e2e\helpers.ts#L407
5) command-commit.test.ts:23:3 › use terminal output as context ──────────────────────────────────
Error: Failed to rmSync C:\Users\RUNNER~1\AppData\Local\Temp\cody-vsceSPmlZ1 after 5 attempts: Error: EBUSY: resource busy or locked, unlink '\\?\C:\Users\RUNNER~1\AppData\Local\Temp\cody-vsceSPmlZ1\logs\20250724T141537\window1\exthost\exthost.log'
at helpers.ts:407
405 | } catch (error) {
406 | if (attempts === 1) {
> 407 | throw new Error(`Failed to rmSync ${path} after ${maxAttempts} attempts: ${error}`)
| ^
408 | }
409 |
410 | await new Promise(resolve => setTimeout(resolve, 100))
at rmSyncWithRetries (D:\a\cody\cody\vscode\test\e2e\helpers.ts:407:23)
at Object.app (D:\a\cody\cody\vscode\test\e2e\helpers.ts:288:13)
|
|
chat-input.test.ts:198:3 › chat model selector:
vscode\test\e2e\chat-input.test.ts#L218
4) chat-input.test.ts:198:3 › chat model selector ────────────────────────────────────────────────
Error: locator.click: Target page, context or browser has been closed
Call log:
- waiting for frameLocator('iframe.webview:first-child:last-child').frameLocator('iframe').getByRole('listbox', { name: 'Suggestions' }).getByRole('option', { name: 'Claude 3 Haiku' })
- locator resolved to <div id=":r13:" cmdk-item="" role="option" aria-disabled="false" aria-selected="false" data-disabled="false" data-selected="false" title="Claude 3 Haiku by Anthropic" data-testid="chat-model-popover-option" data-value="anthropic::2023-06-01::claude-3-haiku-20240307" class="tw-relative tw-flex tw-cursor-pointer tw-select-none tw-items-center tw-py-3 tw-px-3 tw-text-md tw-outline-none aria-selected:tw-bg-accent aria-selected:tw-text-accent-foreground hover:tw-bg-accent hover:tw-text-accent-foreground…>…</div>
- attempting click action
- waiting for element to be visible, enabled and stable
- element is not stable
- retrying click action, attempt #1
- waiting for element to be visible, enabled and stable
- element was detached from the DOM, retrying
216 | await modelSelect.click()
217 | const modelChoices = chatFrame.getByRole('listbox', { name: 'Suggestions' })
> 218 | await modelChoices.getByRole('option', { name: 'Claude 3 Haiku' }).click()
| ^
219 | const lastChatInput = getChatInputs(chatFrame).last()
220 | await expect(lastChatInput).toBeFocused()
221 | await expect(modelSelect).toHaveText(/^Claude 3 Haiku/)
at D:\a\cody\cody\vscode\test\e2e\chat-input.test.ts:218:72
|
|
chat-input.test.ts:198:3 › chat model selector:
vscode\test\e2e\chat-input.test.ts#L0
4) chat-input.test.ts:198:3 › chat model selector ────────────────────────────────────────────────
Test timeout of 30000ms exceeded.
|
|
chat-input.test.ts:198:3 › chat model selector:
vscode\test\e2e\chat-input.test.ts#L218
3) chat-input.test.ts:198:3 › chat model selector ────────────────────────────────────────────────
Error: locator.click: Target page, context or browser has been closed
Call log:
- waiting for frameLocator('iframe.webview:first-child:last-child').frameLocator('iframe').getByRole('listbox', { name: 'Suggestions' }).getByRole('option', { name: 'Claude 3 Haiku' })
- locator resolved to <div id=":r13:" cmdk-item="" role="option" aria-disabled="false" aria-selected="false" data-disabled="false" data-selected="false" title="Claude 3 Haiku by Anthropic" data-testid="chat-model-popover-option" data-value="anthropic::2023-06-01::claude-3-haiku-20240307" class="tw-relative tw-flex tw-cursor-pointer tw-select-none tw-items-center tw-py-3 tw-px-3 tw-text-md tw-outline-none aria-selected:tw-bg-accent aria-selected:tw-text-accent-foreground hover:tw-bg-accent hover:tw-text-accent-foreground…>…</div>
- attempting click action
- waiting for element to be visible, enabled and stable
- element is not stable
- retrying click action, attempt #1
- waiting for element to be visible, enabled and stable
- element was detached from the DOM, retrying
216 | await modelSelect.click()
217 | const modelChoices = chatFrame.getByRole('listbox', { name: 'Suggestions' })
> 218 | await modelChoices.getByRole('option', { name: 'Claude 3 Haiku' }).click()
| ^
219 | const lastChatInput = getChatInputs(chatFrame).last()
220 | await expect(lastChatInput).toBeFocused()
221 | await expect(modelSelect).toHaveText(/^Claude 3 Haiku/)
at D:\a\cody\cody\vscode\test\e2e\chat-input.test.ts:218:72
|
|
chat-input.test.ts:198:3 › chat model selector:
vscode\test\e2e\chat-input.test.ts#L0
3) chat-input.test.ts:198:3 › chat model selector ────────────────────────────────────────────────
Test timeout of 30000ms exceeded.
|
|
chat-context.test.ts:11:5 › chat followup context:
vscode\test\e2e\chat-context.test.ts#L36
2) chat-context.test.ts:11:5 › chat followup context ─────────────────────────────────────────────
Error: expect(locator).toHaveCount(expected)
Locator: frameLocator('.simple-find-part-wrapper + iframe.webview').last().frameLocator('iframe').locator('[data-testid="context"]').filter({ hasText: 'Context' })
Expected: 3
Received: 2
Call log:
- expect.toHaveCount with timeout 5000ms
- waiting for frameLocator('.simple-find-part-wrapper + iframe.webview').last().frameLocator('iframe').locator('[data-testid="context"]').filter({ hasText: 'Context' })
- locator resolved to 2 elements
- unexpected value "2"
- locator resolved to 2 elements
- unexpected value "2"
- locator resolved to 2 elements
- unexpected value "2"
- locator resolved to 2 elements
- unexpected value "2"
- locator resolved to 2 elements
- unexpected value "2"
- locator resolved to 2 elements
- unexpected value "2"
- locator resolved to 2 elements
- unexpected value "2"
34 | await selectMentionMenuItem(chatFrame, 'var.go')
35 | await chatInput.press('Enter')
> 36 | expect(contextCells).toHaveCount(3)
| ^
37 | const lastContextCell = contextCells.last()
38 | expect(lastContextCell).toHaveText(/1 new item/)
39 | await openContextCell(lastContextCell)
at D:\a\cody\cody\vscode\test\e2e\chat-context.test.ts:36:26
|
|
chat-atFile.test.ts:49:7 › @-mention file in chat:
vscode\test\e2e\chat-atFile.test.ts#L167
1) chat-atFile.test.ts:49:7 › @-mention file in chat ─────────────────────────────────────────────
Error: Timed out 5000ms waiting for expect(locator).toHaveText(expected)
Locator: frameLocator('.simple-find-part-wrapper + iframe.webview').last().frameLocator('iframe').getByRole('textbox', { name: 'Chat message' }).last()
Expected string: "Main.java and Main.java "
Received string: "Main.java and @Main.ja"
Call log:
- expect.toHaveText with timeout 5000ms
- waiting for frameLocator('.simple-find-part-wrapper + iframe.webview').last().frameLocator('iframe').getByRole('textbox', { name: 'Chat message' }).last()
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
165 | await chatInput.pressSequentially('and @Main.ja', { delay: 10 })
166 | await chatInput.press('Tab')
> 167 | await expect(chatInput).toHaveText('Main.java and Main.java ')
| ^
168 | await expect(chatInputMentions(chatInput)).toHaveText(['Main.java', 'Main.java'])
|
|
test-e2e (ubuntu)
Process completed with exit code 1.
|
|
chat-input.test.ts:198:3 › chat model selector:
vscode/test/e2e/chat-input.test.ts#L221
2) chat-input.test.ts:198:3 › chat model selector ────────────────────────────────────────────────
Error: Timed out 5000ms waiting for expect(locator).toHaveText(expected)
Locator: frameLocator('iframe.webview:first-child:last-child').frameLocator('iframe').getByRole('combobox', { name: 'Select a model' }).last()
Expected pattern: /^Claude 3 Haiku/
Received string: "Claude 3.5 Sonnet"
Call log:
- expect.toHaveText with timeout 5000ms
- waiting for frameLocator('iframe.webview:first-child:last-child').frameLocator('iframe').getByRole('combobox', { name: 'Select a model' }).last()
- locator resolved to <button type="button" role="combobox" data-state="closed" aria-expanded="false" aria-haspopup="dialog" aria-controls="radix-:rp:" data-testid="chat-model-selector" aria-label="Select a model or an agent" class="tw-border-none tw-flex tw-items-center focus-visible:tw-outline-none tw-justify-between tw-mr-1 _button_1wufw_5 _button--secondary_1wufw_60 _button--no-icon-start_1wufw_51">…</button>
- unexpected value "Claude 3.5 Sonnet"
- locator resolved to <button type="button" role="combobox" data-state="closed" aria-expanded="false" aria-haspopup="dialog" aria-controls="radix-:rp:" data-testid="chat-model-selector" aria-label="Select a model or an agent" class="tw-border-none tw-flex tw-items-center focus-visible:tw-outline-none tw-justify-between tw-mr-1 _button_1wufw_5 _button--secondary_1wufw_60 _button--no-icon-start_1wufw_51">…</button>
- unexpected value "Claude 3.5 Sonnet"
- locator resolved to <button type="button" role="combobox" data-state="closed" aria-expanded="false" aria-haspopup="dialog" aria-controls="radix-:rp:" data-testid="chat-model-selector" aria-label="Select a model or an agent" class="tw-border-none tw-flex tw-items-center focus-visible:tw-outline-none tw-justify-between tw-mr-1 _button_1wufw_5 _button--secondary_1wufw_60 _button--no-icon-start_1wufw_51">…</button>
- unexpected value "Claude 3.5 Sonnet"
- locator resolved to <button type="button" role="combobox" data-state="closed" aria-expanded="false" aria-haspopup="dialog" aria-controls="radix-:rp:" data-testid="chat-model-selector" aria-label="Select a model or an agent" class="tw-border-none tw-flex tw-items-center focus-visible:tw-outline-none tw-justify-between tw-mr-1 _button_1wufw_5 _button--secondary_1wufw_60 _button--no-icon-start_1wufw_51">…</button>
- unexpected value "Claude 3.5 Sonnet"
- locator resolved to <button type="button" role="combobox" data-state="closed" aria-expanded="false" aria-haspopup="dialog" aria-controls="radix-:rp:" data-testid="chat-model-selector" aria-label="Select a model or an agent" class="tw-border-none tw-flex tw-items-center focus-visible:tw-outline-none tw-justify-between tw-mr-1 _button_1wufw_5 _button--secondary_1wufw_60 _button--no-icon-start_1wufw_51">…</button>
- unexpected value "Claude 3.5 Sonnet"
- locator resolved to <button type="button" role="combobox" data-state="closed" aria-expanded="false" aria-haspopup="dialog" aria-controls="radix-:rp:" data-testid="chat-model-selector" aria-label="Select a model or an agent" class="tw-border-none tw-flex tw-items-center focus-visible:tw-outline-none tw-justify-between tw-mr-1 _button_1wufw_5 _button--secondary_1wufw_60 _button--no-icon-start_1wufw_51">…</button>
- unexpected value "Claude 3.5 Sonnet"
- locator resolved to <button type="button" role="combobox" data-state="closed" aria-expanded="false" aria-haspopup="dialog" aria-controls="radix-:rp:" data-testid="chat-model-selector" aria-label="Select a model or an agent" class="tw-border-none tw-flex tw-items-center focus-visible:tw-outline-none tw-justify-between tw-mr-1 _button_1wufw_5 _button--secondary_1wufw_60 _button--no-icon-start_1wufw_51">…</button>
- unexpected value "Claude 3.5 Sonnet"
- locator resolved to <button type="button" role="combobox" data-state="closed" aria-expanded="false" aria-haspopup="dialog" aria-controls="radix-:rp:" data-testi
|
|
chat-atFile.test.ts:49:7 › @-mention file in chat:
vscode/test/e2e/chat-atFile.test.ts#L167
1) chat-atFile.test.ts:49:7 › @-mention file in chat ─────────────────────────────────────────────
Error: Timed out 5000ms waiting for expect(locator).toHaveText(expected)
Locator: frameLocator('.simple-find-part-wrapper + iframe.webview').last().frameLocator('iframe').getByRole('textbox', { name: 'Chat message' }).last()
Expected string: "Main.java and Main.java "
Received string: "Main.java and @Main.ja"
Call log:
- expect.toHaveText with timeout 5000ms
- waiting for frameLocator('.simple-find-part-wrapper + iframe.webview').last().frameLocator('iframe').getByRole('textbox', { name: 'Chat message' }).last()
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
- locator resolved to <div role="textbox" spellcheck="true" contenteditable="true" aria-label="Chat message" data-lexical-editor="true" aria-controls="typeahead-menu" aria-activedescendant="typeahead-item-0" class="_content-editable_s4gjm_6 _editor-content-editable_1eiqs_30">…</div>
- unexpected value "Main.java and @Main.ja"
165 | await chatInput.pressSequentially('and @Main.ja', { delay: 10 })
166 | await chatInput.press('Tab')
> 167 | await expect(chatInput).toHaveText('Main.java and Main.java ')
| ^
168 | await expect(chatInputMentions(chatInput)).toHaveText(['Main.java', 'Main.java'])
|
|
test-e2e (windows)
The process 'C:\Program Files\Git\bin\git.exe' failed with exit code 128
|
|
test-e2e (ubuntu)
The process '/usr/bin/git' failed with exit code 128
|
|
🎭 Playwright Run Summary
13 failed
chat-atFile.test.ts:49:7 › @-mention file in chat ──────────────────────────────────────────────
chat-context.test.ts:11:5 › chat followup context ──────────────────────────────────────────────
chat-input.test.ts:198:3 › chat model selector ─────────────────────────────────────────────────
chat-input.test.ts:198:3 › chat model selector ─────────────────────────────────────────────────
command-commit.test.ts:23:3 › use terminal output as context ───────────────────────────────────
command-edit.test.ts:33:3 › edit (fixup) task ──────────────────────────────────────────────────
command-edit.test.ts:33:3 › edit (fixup) task ──────────────────────────────────────────────────
command-edit.test.ts:33:3 › edit (fixup) task ──────────────────────────────────────────────────
command-edit.test.ts:33:3 › edit (fixup) task ──────────────────────────────────────────────────
command-edit.test.ts:121:1 › edit (fixup) input - range selection ──────────────────────────────
enterprise-server-sent-models.test.ts:6:5 › allows multiple enterprise models when server-sent models is enabled
initial-context.test.ts:46:18 › initial context - file ─────────────────────────────────────────
initial-context.test.ts:46:18 › initial context - file ─────────────────────────────────────────
60 skipped
151 passed (53.6m)
|
|
🎭 Playwright Run Summary
2 failed
chat-atFile.test.ts:49:7 › @-mention file in chat ──────────────────────────────────────────────
chat-input.test.ts:198:3 › chat model selector ─────────────────────────────────────────────────
150 skipped
408 passed (1.8h)
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
playwright-recordings ubuntu
Expired
|
1.05 MB |
sha256:2f760748b3bcc5e401789085c5cf387ea9241db70372b2c4c7bec3da4166e479
|
|
|
playwright-recordings windows
Expired
|
3.07 MB |
sha256:8015c5fe28ba6a6e3459f4d2a107c6fffb2800e33cb57aa633e172a41e2f5b12
|
|