-
### Repository permissions
@@ -35,10 +35,6 @@ Create a fine-grained PAT in GitHub settings and grant the permissions below.
- Pull requests: Read and write
- Metadata: Read-only (required)
-### Account permissions
-
-- Models: Read-only
-
### Repository access scope
Use either of these scopes depending on your needs:
@@ -55,7 +51,7 @@ Use either of these scopes depending on your needs:
3. Paste token into the BYOT input and click add.
4. Verify repository list loads.
5. Select your target repository.
-6. Use AI chat as needed after connecting your token.
+6. Use PR and repository workflows after connecting your token.
## Screenshots
diff --git a/docs/localstorage-state.md b/docs/localstorage-state.md
index ae273a2..7e7485b 100644
--- a/docs/localstorage-state.md
+++ b/docs/localstorage-state.md
@@ -8,9 +8,11 @@ This document is the source of truth for what `@knighted/develop` stores in `loc
1. `knighted:develop:github-pat`
- GitHub personal access token used for API calls.
-2. `knighted-develop:render-mode`
+2. `knighted:develop:openrouter-key`
+ - OpenRouter API key used by AI chat requests.
+3. `knighted-develop:render-mode`
- Last selected render mode (`dom` or `react`).
-3. Theme/UI preference keys managed by layout theme modules.
+4. Theme/UI preference keys managed by layout theme modules.
## Not Allowed In localStorage
diff --git a/docs/openrouter-byok.md b/docs/openrouter-byok.md
new file mode 100644
index 0000000..6d1a417
--- /dev/null
+++ b/docs/openrouter-byok.md
@@ -0,0 +1,40 @@
+# OpenRouter BYOK Setup for AI Chat in @knighted/develop
+
+This guide explains how to create and use an OpenRouter API key for AI chat in `@knighted/develop`.
+
+## What this key does
+
+The OpenRouter key is used only for AI chat requests and model catalog requests in the chat drawer.
+
+- It enables chat completions against `https://openrouter.ai/api/v1/chat/completions`.
+- It enables loading model options from `https://openrouter.ai/api/v1/models`.
+
+The key is independent from the GitHub PAT used by PR and repository workflows.
+
+## Free model limits
+
+OpenRouter free models still require an API key.
+
+- Free models have no per-token charge.
+- Accounts without purchased credits are currently limited to 50 requests per day.
+- Accounts that have purchased at least $10 in credits are currently limited to 1000 requests per day.
+
+## Privacy and storage behavior
+
+- Your OpenRouter key is stored only in your browser `localStorage`.
+- The key is sent only to OpenRouter endpoints used by chat.
+- The key is never sent to GitHub endpoints.
+- You can remove it any time from the chat drawer key controls.
+
+## Create and connect an OpenRouter key
+
+1. Open https://openrouter.ai/keys and create an API key.
+2. Open the Chat drawer in `@knighted/develop`.
+3. Paste the key into the `OpenRouter API key` input.
+4. Click `Save OpenRouter API key`.
+5. Send a test prompt and confirm the assistant response appears.
+
+## Related docs
+
+- GitHub PAT setup for PR/repository workflows: [byot.md](byot.md)
+- Local storage keys: [localstorage-state.md](localstorage-state.md)
diff --git a/docs/openrouter-migration-plan.md b/docs/openrouter-migration-plan.md
index 06b0ab6..e606a8a 100644
--- a/docs/openrouter-migration-plan.md
+++ b/docs/openrouter-migration-plan.md
@@ -43,7 +43,7 @@ Chat no longer depends on a selected repository. Local-mode users can chat to up
editor tab with no GitHub connection at all. A selected repository remains useful context
when one is connected, but it is never a precondition.
-## Implementation status (updated 2026-09-06)
+## Implementation status (updated 2026-09-07)
### Done
@@ -67,20 +67,21 @@ when one is connected, but it is never a precondition.
- Tests and checks completed for the implemented behaviors:
- Focused Playwright coverage added for intent gating, tab-context sending, and apply behavior.
- Lint checks are passing.
-
-### Remaining
-
-- Phase 4 model catalog work is not yet implemented in runtime code:
- - No live `/api/v1/models` fetch integration yet.
- - Free vs paid grouping in the model picker is still pending.
- - Tool-support filtering from live model metadata is still pending.
-- Phase 5 remains partial:
- - Chat tests still live inside `playwright/github-byot-ai.spec.ts` rather than a split chat spec path.
- - Dedicated OpenRouter usage docs listed below are not fully completed.
-- Live production verification still pending for exhaustion states:
- - 402 out-of-credits behavior.
- - 429 rate-limit behavior.
-- Optional one-time migration notice behavior is still pending.
+ - Chat test coverage is split into `playwright/chat/ai-chat.spec.ts`, with PR/BYOT
+ coverage retained in `playwright/github-byot-ai.spec.ts`.
+ - OpenRouter migration docs are in place (`docs/openrouter-byok.md`) and cross-linked
+ from README/BYOT docs.
+- Phase 4 completed in runtime code:
+ - Live `/api/v1/models` catalog fetch is wired in `src/modules/chat/api/models.js`.
+ - Model picker groups models into Free and Paid sections.
+ - Model catalog entries are filtered to tool-capable models.
+
+### Remaining (non-blocking)
+
+- Optional follow-up coverage:
+ - Explicit targeted specs for 402/429 status messaging and catalog-fetch degradation.
+- Optional UX follow-up:
+ - One-time migration notice on first load after upgrade.
### Correction to a common assumption
diff --git a/playwright/chat/ai-chat.spec.ts b/playwright/chat/ai-chat.spec.ts
new file mode 100644
index 0000000..6106ab0
--- /dev/null
+++ b/playwright/chat/ai-chat.spec.ts
@@ -0,0 +1,1089 @@
+import { expect, test } from '@playwright/test'
+import { defaultChatModel } from '../../src/modules/chat/api/completions.js'
+import type { ChatRequestBody, ChatRequestMessage } from '../helpers/app-test-helpers.js'
+import {
+ appEntryPath,
+ connectByotWithSingleRepo,
+ connectOpenRouterKey,
+ ensureWorkspacesDrawerClosed,
+ openRouterTestKey,
+ openWorkspaceTab,
+ setComponentEditorSource,
+ setStylesEditorSource,
+ waitForAppReady,
+} from '../helpers/app-test-helpers.js'
+import {
+ openStoredWorkspaceContextById,
+ seedLocalWorkspaceContexts,
+} from '../github-pr-drawer/github-pr-drawer.helpers.js'
+
+test('chat drawer prompts for an OpenRouter key and gates the composer', async ({
+ page,
+}) => {
+ await waitForAppReady(page)
+
+ await page.getByRole('button', { name: 'Chat', exact: true }).click()
+ await expect(page.getByRole('complementary', { name: 'AI Chat' })).toBeVisible()
+
+ const keyInput = page.getByLabel('OpenRouter API key', { exact: true })
+ await expect(keyInput).toBeVisible()
+ await expect(
+ page.getByRole('button', { name: 'Save OpenRouter API key' }),
+ ).toBeVisible()
+ await expect(
+ page.getByRole('button', { name: 'Remove OpenRouter API key' }),
+ ).toBeHidden()
+
+ await expect(page.getByLabel('Ask AI assistant')).toBeDisabled()
+ await expect(page.getByRole('button', { name: 'Send' })).toBeDisabled()
+ await expect(page.getByLabel('Chat model')).toBeDisabled()
+
+ await connectOpenRouterKey(page)
+
+ await expect(page.getByLabel('Ask AI assistant')).toBeEnabled()
+ await expect(page.getByRole('button', { name: 'Send' })).toBeEnabled()
+ await expect(page.getByLabel('Chat model')).toBeEnabled()
+})
+
+test('GitHub token is never sent to OpenRouter and the chat key is never sent to GitHub', async ({
+ page,
+}) => {
+ const openRouterAuthHeaders: string[] = []
+ const githubAuthHeaders: string[] = []
+
+ page.on('request', request => {
+ const auth = request.headers().authorization ?? ''
+ if (!auth) {
+ return
+ }
+
+ if (request.url().includes('openrouter.ai')) {
+ openRouterAuthHeaders.push(auth)
+ }
+
+ if (request.url().includes('api.github.com')) {
+ githubAuthHeaders.push(auth)
+ }
+ })
+
+ await page.route('https://openrouter.ai/api/v1/chat/completions', async route => {
+ await route.fulfill({
+ status: 200,
+ contentType: 'application/json',
+ body: JSON.stringify({
+ choices: [{ message: { role: 'assistant', content: 'ok' } }],
+ }),
+ })
+ })
+
+ await waitForAppReady(page)
+ await connectByotWithSingleRepo(page)
+ await connectOpenRouterKey(page)
+
+ await page.getByLabel('Ask AI assistant').fill('hello')
+ await page.getByRole('button', { name: 'Send' }).click()
+ await expect(page.getByText('ok', { exact: true })).toBeVisible()
+
+ expect(openRouterAuthHeaders.length).toBeGreaterThan(0)
+ expect(githubAuthHeaders.length).toBeGreaterThan(0)
+ expect(openRouterAuthHeaders.every(header => header.includes(openRouterTestKey))).toBe(
+ true,
+ )
+ expect(openRouterAuthHeaders.some(header => header.includes('github_pat'))).toBe(false)
+ expect(githubAuthHeaders.some(header => header.includes(openRouterTestKey))).toBe(false)
+})
+
+test('chat stays usable after opening a Local workspace with PAT connected', async ({
+ page,
+}) => {
+ const localWorkspaceId = 'local_chat_issue_128'
+ let streamRequestBody: ChatRequestBody | undefined
+
+ await page.route('https://openrouter.ai/api/v1/chat/completions', async route => {
+ streamRequestBody = route.request().postDataJSON() as ChatRequestBody
+
+ await route.fulfill({
+ status: 200,
+ contentType: 'text/event-stream',
+ body: [
+ 'data: {"choices":[{"delta":{"content":"Local workspace chat works"}}]}',
+ '',
+ 'data: [DONE]',
+ '',
+ ].join('\n'),
+ })
+ })
+
+ await waitForAppReady(page)
+
+ await seedLocalWorkspaceContexts(page, [
+ {
+ id: localWorkspaceId,
+ repo: '',
+ workspaceScope: 'local',
+ head: 'feat/local-chat-issue-128',
+ prTitle: 'Issue 128 local workspace',
+ prContextState: 'inactive',
+ tabs: [
+ {
+ id: 'component',
+ path: 'src/component.tsx',
+ language: 'tsx',
+ role: 'component',
+ content: 'export const App = () => Visible
\n```', + }, + }, + ], + }), + }) + }) + + await waitForAppReady(page, `${appEntryPath}`) + await connectByotWithSingleRepo(page) + await setComponentEditorSource(page, 'const App = () =>Before
') + await openWorkspaceTab(page, 'App.tsx') + await connectOpenRouterKey(page) + + await page.getByLabel('Ask AI assistant').fill('Can you see my editor content?') + await page.getByRole('button', { name: 'Send' }).click() + + await expect(page.getByText('Fallback response loaded.', { exact: true })).toHaveText( + 'Fallback response loaded.', + ) + await expect(page.locator('button[data-action="request-apply"]')).toHaveCount(0) + expect(streamRequestBody?.tool_choice).toBeUndefined() + expect(streamRequestBody?.tools).toBeUndefined() +}) + +test('AI chat can disable editor context payload via checkbox', async ({ page }) => { + let streamRequestBody: ChatRequestBody | undefined + + await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { + streamRequestBody = route.request().postDataJSON() as ChatRequestBody + + await route.fulfill({ + status: 200, + contentType: 'text/event-stream', + body: [ + 'data: {"choices":[{"delta":{"content":"ok"}}]}', + '', + 'data: [DONE]', + '', + ].join('\n'), + }) + }) + + await waitForAppReady(page, `${appEntryPath}`) + await connectByotWithSingleRepo(page) + await connectOpenRouterKey(page) + + const includeEditorsToggle = page.getByLabel('Send tab content') + await expect(includeEditorsToggle).toBeChecked() + await includeEditorsToggle.uncheck() + + await page.getByLabel('Ask AI assistant').fill('No editor source this time.') + await page.getByRole('button', { name: 'Send' }).click() + await expect(page.getByText('Response streamed.', { exact: true })).toHaveText( + 'Response streamed.', + ) + + expect(streamRequestBody?.metadata).toBeUndefined() + expect(streamRequestBody?.tool_choice).toBeUndefined() + expect(streamRequestBody?.tools).toBeUndefined() + const systemMessages = streamRequestBody?.messages?.filter( + (message: ChatRequestMessage) => message.role === 'system', + ) + expect( + systemMessages?.some((message: ChatRequestMessage) => + message.content?.includes('Selected repository context'), + ), + ).toBe(true) + expect( + systemMessages?.some((message: ChatRequestMessage) => + message.content?.includes( + 'Repository URL: https://github.com/knightedcodemonkey/develop', + ), + ), + ).toBe(true) + expect( + systemMessages?.some((message: ChatRequestMessage) => + message.content?.includes('Editor context:'), + ), + ).toBe(false) +}) + +test('AI chat proposals can be confirmed, applied, and undone per active tab', async ({ + page, +}) => { + await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { + const body = route.request().postDataJSON() as ChatRequestBody | null + + if (body?.stream) { + await route.fulfill({ + status: 502, + contentType: 'application/json', + body: JSON.stringify({ message: 'stream intentionally disabled in this test' }), + }) + return + } + + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ + choices: [ + { + message: { + role: 'assistant', + content: 'Prepared updates for both editors.', + tool_calls: [ + { + id: 'call_component', + type: 'function', + function: { + name: 'propose_editor_update', + arguments: JSON.stringify({ + target: 'src/components/App.tsx', + content: 'const App = () => ', + rationale: 'Use explicit App component output.', + }), + }, + }, + { + id: 'call_styles', + type: 'function', + function: { + name: 'propose_editor_update', + arguments: JSON.stringify({ + target: 'src/styles/app.css', + content: '.button { color: rgb(10 20 30); }', + rationale: 'Provide deterministic button styling.', + }), + }, + }, + ], + }, + }, + ], + }), + }) + }) + + await waitForAppReady(page, `${appEntryPath}`) + await connectByotWithSingleRepo(page) + await setComponentEditorSource(page, 'const App = () => ') + await setStylesEditorSource(page, '.button { color: red; }') + await openWorkspaceTab(page, 'App.tsx') + await connectOpenRouterKey(page) + + await page.getByLabel('Ask AI assistant').fill('Suggest updates for both editors.') + await page.getByRole('button', { name: 'Send' }).click() + + await expect( + page.getByText('Prepared updates for both editors.', { exact: true }), + ).toBeVisible() + + await expect( + page.getByRole('button', { name: 'Apply update to App.tsx' }), + ).toBeVisible() + await expect( + page.getByRole('button', { name: 'Apply update to app.css' }), + ).toBeVisible() + + await page.getByRole('button', { name: 'Apply update to App.tsx' }).click() + + await expect(page.getByRole('button', { name: 'Apply update to App.tsx' })).toBeHidden() + await expect( + page.getByRole('button', { name: 'Undo last apply for App.tsx' }), + ).toBeVisible() + await expect( + page.getByRole('button', { name: 'Undo last apply for app.css' }), + ).toBeHidden() + await expect( + page.locator('.editor-panel[data-editor-kind="component"] .cm-content').first(), + ).toContainText('Updated') + + await openWorkspaceTab(page, 'app.css') + await expect( + page.getByRole('button', { name: 'Undo last apply for App.tsx' }), + ).toBeHidden() + await expect( + page.getByRole('button', { name: 'Apply update to app.css' }), + ).toBeVisible() + await page.getByRole('button', { name: 'Apply update to app.css' }).click() + + await expect( + page.locator('.editor-panel[data-editor-kind="styles"] .cm-content').first(), + ).toContainText('rgb(10 20 30)') + await expect( + page.getByRole('button', { name: 'Undo last apply for app.css' }), + ).toBeVisible() + await expect( + page.getByRole('button', { name: 'Undo last apply for App.tsx' }), + ).toBeHidden() + + await page.getByRole('button', { name: 'Undo last apply for app.css' }).click() + await expect( + page.locator('.editor-panel[data-editor-kind="styles"] .cm-content').first(), + ).toContainText('red') + + await openWorkspaceTab(page, 'App.tsx') + await expect( + page.getByRole('button', { name: 'Undo last apply for App.tsx' }), + ).toBeVisible() + await expect( + page.getByRole('button', { name: 'Undo last apply for app.css' }), + ).toBeHidden() + + await page.getByRole('button', { name: 'Undo last apply for App.tsx' }).click() + await expect( + page.locator('.editor-panel[data-editor-kind="component"] .cm-content').first(), + ).toContainText('Before') +}) + +test('AI chat apply actions resolve dynamic tab targets', async ({ page }) => { + await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { + const body = route.request().postDataJSON() as ChatRequestBody | null + + if (body?.stream) { + await route.fulfill({ + status: 502, + contentType: 'application/json', + body: JSON.stringify({ message: 'stream intentionally disabled in this test' }), + }) + return + } + + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ + choices: [ + { + message: { + role: 'assistant', + content: 'Prepared updates for both editors.', + tool_calls: [ + { + id: 'call_component', + type: 'function', + function: { + name: 'propose_editor_update', + arguments: JSON.stringify({ + target: 'src/components/App.tsx', + content: 'const App = () => ', + }), + }, + }, + { + id: 'call_styles', + type: 'function', + function: { + name: 'propose_editor_update', + arguments: JSON.stringify({ + target: 'src/styles/app.css', + content: '.button { color: rgb(10 20 30); }', + }), + }, + }, + ], + }, + }, + ], + }), + }) + }) + + await waitForAppReady(page, `${appEntryPath}`) + await connectByotWithSingleRepo(page) + await setComponentEditorSource(page, 'const App = () => ') + await setStylesEditorSource(page, '.button { color: red; }') + await openWorkspaceTab(page, 'App.tsx') + await connectOpenRouterKey(page) + + await page.getByLabel('Ask AI assistant').fill('Suggest updates for both editors.') + await page.getByRole('button', { name: 'Send' }).click() + + await expect( + page.getByText('Prepared updates for both editors.', { exact: true }), + ).toBeVisible() + + await expect( + page.getByRole('button', { name: 'Apply update to App.tsx' }), + ).toBeVisible() + await expect( + page.getByRole('button', { name: 'Apply update to app.css' }), + ).toBeVisible() + + await openWorkspaceTab(page, 'app.css') + + await expect( + page.getByRole('button', { name: 'Apply update to App.tsx' }), + ).toBeVisible() + await expect( + page.getByRole('button', { name: 'Apply update to app.css' }), + ).toBeVisible() +}) + +test('AI chat applies the correct proposal when unresolved targets are filtered out', async ({ + page, +}) => { + await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { + const body = route.request().postDataJSON() as ChatRequestBody | null + + if (body?.stream) { + await route.fulfill({ + status: 502, + contentType: 'application/json', + body: JSON.stringify({ message: 'stream intentionally disabled in this test' }), + }) + return + } + + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ + choices: [ + { + message: { + role: 'assistant', + content: 'Prepared updates for App tab.', + tool_calls: [ + { + id: 'call_unresolved', + type: 'function', + function: { + name: 'propose_editor_update', + arguments: JSON.stringify({ + target: 'src/components/missing.tsx', + content: 'const Missing = () => null', + }), + }, + }, + { + id: 'call_component', + type: 'function', + function: { + name: 'propose_editor_update', + arguments: JSON.stringify({ + target: 'src/components/App.tsx', + content: 'const App = () =>Resolved update
', + }), + }, + }, + ], + }, + }, + ], + }), + }) + }) + + await waitForAppReady(page, `${appEntryPath}`) + await connectByotWithSingleRepo(page) + await setComponentEditorSource(page, 'const App = () =>Before
') + await openWorkspaceTab(page, 'App.tsx') + await connectOpenRouterKey(page) + + await page.getByLabel('Ask AI assistant').fill('Update App tab only.') + await page.getByRole('button', { name: 'Send' }).click() + + await expect( + page.getByRole('button', { name: 'Apply update to App.tsx' }), + ).toBeVisible() + await page.getByRole('button', { name: 'Apply update to App.tsx' }).click() + + await expect( + page.locator('.editor-panel[data-editor-kind="component"] .cm-content').first(), + ).toContainText('Resolved update') +}) + +test('AI chat renders a single apply action for multiple targets resolving to the same tab', async ({ + page, +}) => { + await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { + const body = route.request().postDataJSON() as ChatRequestBody | null + + if (body?.stream) { + await route.fulfill({ + status: 502, + contentType: 'application/json', + body: JSON.stringify({ message: 'stream intentionally disabled in this test' }), + }) + return + } + + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ + choices: [ + { + message: { + role: 'assistant', + content: 'Prepared updates for App tab.', + tool_calls: [ + { + id: 'call_component_id', + type: 'function', + function: { + name: 'propose_editor_update', + arguments: JSON.stringify({ + target: 'component', + content: 'const App = () =>By id
', + }), + }, + }, + { + id: 'call_component_path', + type: 'function', + function: { + name: 'propose_editor_update', + arguments: JSON.stringify({ + target: 'src/components/App.tsx', + content: 'const App = () =>By path
', + }), + }, + }, + ], + }, + }, + ], + }), + }) + }) + + await waitForAppReady(page, `${appEntryPath}`) + await connectByotWithSingleRepo(page) + await setComponentEditorSource(page, 'const App = () =>Before
') + await openWorkspaceTab(page, 'App.tsx') + await connectOpenRouterKey(page) + + await page.getByLabel('Ask AI assistant').fill('Update App tab once.') + await page.getByRole('button', { name: 'Send' }).click() + + await expect(page.getByRole('button', { name: 'Apply update to App.tsx' })).toHaveCount( + 1, + ) +}) + +test('AI chat shows guidance when an editor update target cannot be matched', async ({ + page, +}) => { + await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { + const body = route.request().postDataJSON() as ChatRequestBody | null + + if (body?.stream) { + await route.fulfill({ + status: 502, + contentType: 'application/json', + body: JSON.stringify({ message: 'stream intentionally disabled in this test' }), + }) + return + } + + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ + choices: [ + { + message: { + role: 'assistant', + content: '', + tool_calls: [ + { + id: 'call_unknown_target', + type: 'function', + function: { + name: 'propose_editor_update', + arguments: JSON.stringify({ + target: 'src/does-not-exist.ts', + content: 'export const value = 1', + }), + }, + }, + ], + }, + }, + ], + }), + }) + }) + + await waitForAppReady(page, `${appEntryPath}`) + await connectByotWithSingleRepo(page) + await setComponentEditorSource(page, 'const App = () =>Before
') + await openWorkspaceTab(page, 'App.tsx') + await connectOpenRouterKey(page) + + await page.getByLabel('Ask AI assistant').fill('Can you still see my tab content?') + await page.getByRole('button', { name: 'Send' }).click() + + await expect( + page.getByText( + 'Proposed editor update is ready, but I could not match its target to an open tab. Ask me to target the active tab or one of the listed tab ids or paths.', + ), + ).toHaveCount(1) + await expect(page.locator('button[data-action="request-apply"]')).toHaveCount(0) +}) + +test('AI chat sends the currently active tab when context is enabled', async ({ + page, +}) => { + let streamRequestBody: ChatRequestBody | undefined + + await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { + streamRequestBody = route.request().postDataJSON() as ChatRequestBody + + await route.fulfill({ + status: 200, + contentType: 'text/event-stream', + body: [ + 'data: {"choices":[{"delta":{"content":"ok"}}]}', + '', + 'data: [DONE]', + '', + ].join('\n'), + }) + }) + + await waitForAppReady(page, `${appEntryPath}`) + await connectByotWithSingleRepo(page) + await setStylesEditorSource(page, '.button { color: red; }') + await connectOpenRouterKey(page) + + await page.getByLabel('Ask AI assistant').fill('Use active tab context only.') + await page.getByRole('button', { name: 'Send' }).click() + await expect(page.getByText('Response streamed.', { exact: true })).toHaveText( + 'Response streamed.', + ) + + const systemMessages = streamRequestBody?.messages?.filter( + (message: ChatRequestMessage) => message.role === 'system', + ) + expect( + systemMessages?.some( + (message: ChatRequestMessage) => + message.content?.includes('- Active tab:') && + message.content?.includes('app.css'), + ), + ).toBe(true) + expect( + systemMessages?.some((message: ChatRequestMessage) => + message.content?.includes('Active tab source:'), + ), + ).toBe(true) + expect( + systemMessages?.some((message: ChatRequestMessage) => + message.content?.includes('Available tab targets (id and path):'), + ), + ).toBe(true) +}) + +test('AI chat streaming text still updates while latest undo actions are visible', async ({ + page, +}) => { + let requestCount = 0 + + await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { + requestCount += 1 + const body = route.request().postDataJSON() as ChatRequestBody | null + + if (requestCount <= 2) { + if (body?.stream) { + await route.fulfill({ + status: 502, + contentType: 'application/json', + body: JSON.stringify({ message: 'force fallback for proposal setup' }), + }) + return + } + + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ + choices: [ + { + message: { + role: 'assistant', + content: 'Prepared updates for styles editor.', + tool_calls: [ + { + id: 'call_styles', + type: 'function', + function: { + name: 'propose_editor_update', + arguments: JSON.stringify({ + target: 'src/styles/app.css', + content: '.button { color: rgb(10 20 30); }', + }), + }, + }, + ], + }, + }, + ], + }), + }) + return + } + + if (body?.stream) { + await route.fulfill({ + status: 200, + contentType: 'text/event-stream', + body: [ + 'data: {"choices":[{"delta":{"content":"Streaming "}}]}', + '', + 'data: {"choices":[{"delta":{"content":"works with undo visible."}}]}', + '', + 'data: [DONE]', + '', + ].join('\n'), + }) + return + } + + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ + choices: [{ message: { role: 'assistant', content: 'fallback text' } }], + }), + }) + }) + + await waitForAppReady(page, `${appEntryPath}`) + await connectByotWithSingleRepo(page) + await setStylesEditorSource(page, '.button { color: red; }') + await connectOpenRouterKey(page) + + await page.getByLabel('Ask AI assistant').fill('Suggest a styles update.') + await page.getByRole('button', { name: 'Send' }).click() + + await expect( + page.getByText('Prepared updates for styles editor.', { exact: true }), + ).toBeVisible() + await page.getByRole('button', { name: 'Apply update to app.css' }).click() + await expect( + page.getByRole('button', { name: 'Undo last apply for app.css' }), + ).toBeVisible() + + await page + .getByLabel('Ask AI assistant') + .fill('Are you still working on that last request?') + await page.getByRole('button', { name: 'Send' }).click() + + await expect(page.getByText('Streaming works with undo visible.')).toBeVisible() +}) + +test('AI chat falls back to non-streaming response when streaming fails', async ({ + page, +}) => { + let streamAttemptCount = 0 + let fallbackAttemptCount = 0 + const attemptedModels: string[] = [] + + await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { + const body = route.request().postDataJSON() as ChatRequestBody | null + if (typeof body?.model === 'string') { + attemptedModels.push(body.model) + } + + if (body?.stream) { + streamAttemptCount += 1 + await route.fulfill({ + status: 502, + contentType: 'application/json', + body: JSON.stringify({ message: 'stream failed' }), + }) + return + } + + fallbackAttemptCount += 1 + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ + rate_limit: { + remaining: 17, + reset: 1704067200, + }, + choices: [ + { + message: { + role: 'assistant', + content: 'Fallback response from JSON path.', + }, + }, + ], + }), + }) + }) + + await waitForAppReady(page, `${appEntryPath}`) + await connectByotWithSingleRepo(page) + await connectOpenRouterKey(page) + + const selectedModel = 'openai/gpt-6-astra' + await page.getByLabel('Chat model').selectOption(selectedModel) + await expect(page.getByLabel('Chat model')).toHaveValue(selectedModel) + + await page.getByLabel('Ask AI assistant').fill('Use fallback path.') + await page.getByRole('button', { name: 'Send' }).click() + + await expect(page.getByText('Fallback response loaded.', { exact: true })).toHaveText( + 'Fallback response loaded.', + ) + await expect(page.getByText('Fallback response from JSON path.')).toBeVisible() + expect(streamAttemptCount).toBeGreaterThan(0) + expect(fallbackAttemptCount).toBeGreaterThan(0) + expect(attemptedModels.length).toBeGreaterThan(0) + expect(attemptedModels.every(model => model === selectedModel)).toBe(true) +}) + +test('clearing chat removes previous conversation context from new request', async ({ + page, +}) => { + const streamBodies: ChatRequestBody[] = [] + + await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { + const body = route.request().postDataJSON() as ChatRequestBody + if (body?.stream) { + streamBodies.push(body) + await route.fulfill({ + status: 200, + contentType: 'text/event-stream', + body: [ + 'data: {"choices":[{"delta":{"content":"ok"}}]}', + '', + 'data: [DONE]', + '', + ].join('\n'), + }) + return + } + + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ + choices: [{ message: { role: 'assistant', content: 'ok' } }], + }), + }) + }) + + await waitForAppReady(page, `${appEntryPath}`) + await connectByotWithSingleRepo(page) + await connectOpenRouterKey(page) + + await page.getByLabel('Ask AI assistant').fill('First conversation prompt') + await page.getByRole('button', { name: 'Send' }).click() + await expect(page.getByText('Response streamed.', { exact: true })).toBeVisible() + + await page.getByRole('button', { name: 'Clear', exact: true }).click() + await expect(page.getByText('Chat cleared.', { exact: true })).toBeVisible() + + await page.getByLabel('Ask AI assistant').fill('Second conversation prompt') + await page.getByRole('button', { name: 'Send' }).click() + await expect(page.getByText('Response streamed.', { exact: true })).toBeVisible() + + expect(streamBodies.length).toBeGreaterThanOrEqual(2) + const latestMessages = streamBodies[streamBodies.length - 1]?.messages ?? [] + const allLatestContent = latestMessages.map(message => message.content ?? '').join('\n') + + expect(allLatestContent).toContain('Second conversation prompt') + expect(allLatestContent).not.toContain('First conversation prompt') +}) diff --git a/playwright/github-byot-ai.spec.ts b/playwright/github-byot-ai.spec.ts index b491630..9c8e18d 100644 --- a/playwright/github-byot-ai.spec.ts +++ b/playwright/github-byot-ai.spec.ts @@ -1,22 +1,15 @@ import { expect, test } from '@playwright/test' -import { defaultChatModel } from '../src/modules/chat/api/completions.js' -import type { ChatRequestBody, ChatRequestMessage } from './helpers/app-test-helpers.js' import { appEntryPath, connectByotWithSingleRepo, ensureWorkspacesDrawerClosed, - connectOpenRouterKey, - openRouterTestKey, ensureOpenPrDrawerOpen, mockRepositoryBranches, - openWorkspaceTab, setComponentEditorSource, - setStylesEditorSource, waitForAppReady, } from './helpers/app-test-helpers.js' import { getAllWorkspaceRecords, - openStoredWorkspaceContextById, seedLocalWorkspaceContexts, } from './github-pr-drawer/github-pr-drawer.helpers.js' import { selectWorkspacesRepositoryFilter } from './github-pr-drawer/github-pr-drawer.helpers.js' @@ -48,34 +41,6 @@ test('PR/BYOT controls are visible and chat is available without a GitHub token' await expect(workspacesToggle).toBeVisible() }) -test('chat drawer prompts for an OpenRouter key and gates the composer', async ({ - page, -}) => { - await waitForAppReady(page) - - await page.getByRole('button', { name: 'Chat', exact: true }).click() - await expect(page.getByRole('complementary', { name: 'AI Chat' })).toBeVisible() - - const keyInput = page.getByLabel('OpenRouter API key', { exact: true }) - await expect(keyInput).toBeVisible() - await expect( - page.getByRole('button', { name: 'Save OpenRouter API key' }), - ).toBeVisible() - await expect( - page.getByRole('button', { name: 'Remove OpenRouter API key' }), - ).toBeHidden() - - await expect(page.getByLabel('Ask AI assistant')).toBeDisabled() - await expect(page.getByRole('button', { name: 'Send' })).toBeDisabled() - await expect(page.getByLabel('Chat model')).toBeDisabled() - - await connectOpenRouterKey(page) - - await expect(page.getByLabel('Ask AI assistant')).toBeEnabled() - await expect(page.getByRole('button', { name: 'Send' })).toBeEnabled() - await expect(page.getByLabel('Chat model')).toBeEnabled() -}) - test('Workspaces repository filter is local-only and read-only without PAT', async ({ page, }) => { @@ -325,54 +290,6 @@ test('PAT connect after Local-only session preserves Local records and enables r ).toBe('local') }) -test('GitHub token is never sent to OpenRouter and the chat key is never sent to GitHub', async ({ - page, -}) => { - const openRouterAuthHeaders: string[] = [] - const githubAuthHeaders: string[] = [] - - page.on('request', request => { - const auth = request.headers().authorization ?? '' - if (!auth) { - return - } - - if (request.url().includes('openrouter.ai')) { - openRouterAuthHeaders.push(auth) - } - - if (request.url().includes('api.github.com')) { - githubAuthHeaders.push(auth) - } - }) - - await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { - await route.fulfill({ - status: 200, - contentType: 'application/json', - body: JSON.stringify({ - choices: [{ message: { role: 'assistant', content: 'ok' } }], - }), - }) - }) - - await waitForAppReady(page) - await connectByotWithSingleRepo(page) - await connectOpenRouterKey(page) - - await page.getByLabel('Ask AI assistant').fill('hello') - await page.getByRole('button', { name: 'Send' }).click() - await expect(page.getByText('ok', { exact: true })).toBeVisible() - - expect(openRouterAuthHeaders.length).toBeGreaterThan(0) - expect(githubAuthHeaders.length).toBeGreaterThan(0) - expect(openRouterAuthHeaders.every(header => header.includes(openRouterTestKey))).toBe( - true, - ) - expect(openRouterAuthHeaders.some(header => header.includes('github_pat'))).toBe(false) - expect(githubAuthHeaders.some(header => header.includes(openRouterTestKey))).toBe(false) -}) - test('workspace context status stays visible without PAT and after PAT connect', async ({ page, }) => { @@ -672,79 +589,6 @@ test('Repository-scoped workspace cannot be renamed from Workspaces drawer', asy await expect(renameButton).toBeHidden() }) -test('chat stays usable after opening a Local workspace with PAT connected', async ({ - page, -}) => { - const localWorkspaceId = 'local_chat_issue_128' - let streamRequestBody: ChatRequestBody | undefined - - await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { - streamRequestBody = route.request().postDataJSON() as ChatRequestBody - - await route.fulfill({ - status: 200, - contentType: 'text/event-stream', - body: [ - 'data: {"choices":[{"delta":{"content":"Local workspace chat works"}}]}', - '', - 'data: [DONE]', - '', - ].join('\n'), - }) - }) - - await waitForAppReady(page) - - await seedLocalWorkspaceContexts(page, [ - { - id: localWorkspaceId, - repo: '', - workspaceScope: 'local', - head: 'feat/local-chat-issue-128', - prTitle: 'Issue 128 local workspace', - prContextState: 'inactive', - tabs: [ - { - id: 'component', - path: 'src/component.tsx', - language: 'tsx', - role: 'component', - content: 'export const App = () =>Visible
\n```', - }, - }, - ], - }), - }) - }) - - await waitForAppReady(page, `${appEntryPath}`) - await connectByotWithSingleRepo(page) - await setComponentEditorSource(page, 'const App = () =>Before
') - await openWorkspaceTab(page, 'App.tsx') - await connectOpenRouterKey(page) - - await page.getByLabel('Ask AI assistant').fill('Can you see my editor content?') - await page.getByRole('button', { name: 'Send' }).click() - - await expect(page.getByText('Fallback response loaded.', { exact: true })).toHaveText( - 'Fallback response loaded.', - ) - await expect(page.locator('button[data-action="request-apply"]')).toHaveCount(0) - expect(streamRequestBody?.tool_choice).toBeUndefined() - expect(streamRequestBody?.tools).toBeUndefined() -}) - -test('AI chat can disable editor context payload via checkbox', async ({ page }) => { - let streamRequestBody: ChatRequestBody | undefined - - await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { - streamRequestBody = route.request().postDataJSON() as ChatRequestBody - - await route.fulfill({ - status: 200, - contentType: 'text/event-stream', - body: [ - 'data: {"choices":[{"delta":{"content":"ok"}}]}', - '', - 'data: [DONE]', - '', - ].join('\n'), - }) - }) - - await waitForAppReady(page, `${appEntryPath}`) - await connectByotWithSingleRepo(page) - await connectOpenRouterKey(page) - - const includeEditorsToggle = page.getByLabel('Send tab content') - await expect(includeEditorsToggle).toBeChecked() - await includeEditorsToggle.uncheck() - - await page.getByLabel('Ask AI assistant').fill('No editor source this time.') - await page.getByRole('button', { name: 'Send' }).click() - await expect(page.getByText('Response streamed.', { exact: true })).toHaveText( - 'Response streamed.', - ) - - expect(streamRequestBody?.metadata).toBeUndefined() - expect(streamRequestBody?.tool_choice).toBeUndefined() - expect(streamRequestBody?.tools).toBeUndefined() - const systemMessages = streamRequestBody?.messages?.filter( - (message: ChatRequestMessage) => message.role === 'system', - ) - expect( - systemMessages?.some((message: ChatRequestMessage) => - message.content?.includes('Selected repository context'), - ), - ).toBe(true) - expect( - systemMessages?.some((message: ChatRequestMessage) => - message.content?.includes( - 'Repository URL: https://github.com/knightedcodemonkey/develop', - ), - ), - ).toBe(true) - expect( - systemMessages?.some((message: ChatRequestMessage) => - message.content?.includes('Editor context:'), - ), - ).toBe(false) -}) - -test('AI chat proposals can be confirmed, applied, and undone per active tab', async ({ - page, -}) => { - await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { - const body = route.request().postDataJSON() as ChatRequestBody | null - - if (body?.stream) { - await route.fulfill({ - status: 502, - contentType: 'application/json', - body: JSON.stringify({ message: 'stream intentionally disabled in this test' }), - }) - return - } - - await route.fulfill({ - status: 200, - contentType: 'application/json', - body: JSON.stringify({ - choices: [ - { - message: { - role: 'assistant', - content: 'Prepared updates for both editors.', - tool_calls: [ - { - id: 'call_component', - type: 'function', - function: { - name: 'propose_editor_update', - arguments: JSON.stringify({ - target: 'src/components/App.tsx', - content: 'const App = () => ', - rationale: 'Use explicit App component output.', - }), - }, - }, - { - id: 'call_styles', - type: 'function', - function: { - name: 'propose_editor_update', - arguments: JSON.stringify({ - target: 'src/styles/app.css', - content: '.button { color: rgb(10 20 30); }', - rationale: 'Provide deterministic button styling.', - }), - }, - }, - ], - }, - }, - ], - }), - }) - }) - - await waitForAppReady(page, `${appEntryPath}`) - await connectByotWithSingleRepo(page) - await setComponentEditorSource(page, 'const App = () => ') - await setStylesEditorSource(page, '.button { color: red; }') - await openWorkspaceTab(page, 'App.tsx') - await connectOpenRouterKey(page) - - await page.getByLabel('Ask AI assistant').fill('Suggest updates for both editors.') - await page.getByRole('button', { name: 'Send' }).click() - - await expect( - page.getByText('Prepared updates for both editors.', { exact: true }), - ).toBeVisible() - - await expect( - page.getByRole('button', { name: 'Apply update to App.tsx' }), - ).toBeVisible() - await expect( - page.getByRole('button', { name: 'Apply update to app.css' }), - ).toBeVisible() - - await page.getByRole('button', { name: 'Apply update to App.tsx' }).click() - - await expect(page.getByRole('button', { name: 'Apply update to App.tsx' })).toBeHidden() - await expect( - page.getByRole('button', { name: 'Undo last apply for App.tsx' }), - ).toBeVisible() - await expect( - page.getByRole('button', { name: 'Undo last apply for app.css' }), - ).toBeHidden() - await expect( - page.locator('.editor-panel[data-editor-kind="component"] .cm-content').first(), - ).toContainText('Updated') - - await openWorkspaceTab(page, 'app.css') - await expect( - page.getByRole('button', { name: 'Undo last apply for App.tsx' }), - ).toBeHidden() - await expect( - page.getByRole('button', { name: 'Apply update to app.css' }), - ).toBeVisible() - await page.getByRole('button', { name: 'Apply update to app.css' }).click() - - await expect( - page.locator('.editor-panel[data-editor-kind="styles"] .cm-content').first(), - ).toContainText('rgb(10 20 30)') - await expect( - page.getByRole('button', { name: 'Undo last apply for app.css' }), - ).toBeVisible() - await expect( - page.getByRole('button', { name: 'Undo last apply for App.tsx' }), - ).toBeHidden() - - await page.getByRole('button', { name: 'Undo last apply for app.css' }).click() - await expect( - page.locator('.editor-panel[data-editor-kind="styles"] .cm-content').first(), - ).toContainText('red') - - await openWorkspaceTab(page, 'App.tsx') - await expect( - page.getByRole('button', { name: 'Undo last apply for App.tsx' }), - ).toBeVisible() - await expect( - page.getByRole('button', { name: 'Undo last apply for app.css' }), - ).toBeHidden() - - await page.getByRole('button', { name: 'Undo last apply for App.tsx' }).click() - await expect( - page.locator('.editor-panel[data-editor-kind="component"] .cm-content').first(), - ).toContainText('Before') -}) - -test('AI chat apply actions resolve dynamic tab targets', async ({ page }) => { - await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { - const body = route.request().postDataJSON() as ChatRequestBody | null - - if (body?.stream) { - await route.fulfill({ - status: 502, - contentType: 'application/json', - body: JSON.stringify({ message: 'stream intentionally disabled in this test' }), - }) - return - } - - await route.fulfill({ - status: 200, - contentType: 'application/json', - body: JSON.stringify({ - choices: [ - { - message: { - role: 'assistant', - content: 'Prepared updates for both editors.', - tool_calls: [ - { - id: 'call_component', - type: 'function', - function: { - name: 'propose_editor_update', - arguments: JSON.stringify({ - target: 'src/components/App.tsx', - content: 'const App = () => ', - }), - }, - }, - { - id: 'call_styles', - type: 'function', - function: { - name: 'propose_editor_update', - arguments: JSON.stringify({ - target: 'src/styles/app.css', - content: '.button { color: rgb(10 20 30); }', - }), - }, - }, - ], - }, - }, - ], - }), - }) - }) - - await waitForAppReady(page, `${appEntryPath}`) - await connectByotWithSingleRepo(page) - await setComponentEditorSource(page, 'const App = () => ') - await setStylesEditorSource(page, '.button { color: red; }') - await openWorkspaceTab(page, 'App.tsx') - await connectOpenRouterKey(page) - - await page.getByLabel('Ask AI assistant').fill('Suggest updates for both editors.') - await page.getByRole('button', { name: 'Send' }).click() - - await expect( - page.getByText('Prepared updates for both editors.', { exact: true }), - ).toBeVisible() - - await expect( - page.getByRole('button', { name: 'Apply update to App.tsx' }), - ).toBeVisible() - await expect( - page.getByRole('button', { name: 'Apply update to app.css' }), - ).toBeVisible() - - await openWorkspaceTab(page, 'app.css') - - await expect( - page.getByRole('button', { name: 'Apply update to App.tsx' }), - ).toBeVisible() - await expect( - page.getByRole('button', { name: 'Apply update to app.css' }), - ).toBeVisible() -}) - -test('AI chat applies the correct proposal when unresolved targets are filtered out', async ({ - page, -}) => { - await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { - const body = route.request().postDataJSON() as ChatRequestBody | null - - if (body?.stream) { - await route.fulfill({ - status: 502, - contentType: 'application/json', - body: JSON.stringify({ message: 'stream intentionally disabled in this test' }), - }) - return - } - - await route.fulfill({ - status: 200, - contentType: 'application/json', - body: JSON.stringify({ - choices: [ - { - message: { - role: 'assistant', - content: 'Prepared updates for App tab.', - tool_calls: [ - { - id: 'call_unresolved', - type: 'function', - function: { - name: 'propose_editor_update', - arguments: JSON.stringify({ - target: 'src/components/missing.tsx', - content: 'const Missing = () => null', - }), - }, - }, - { - id: 'call_component', - type: 'function', - function: { - name: 'propose_editor_update', - arguments: JSON.stringify({ - target: 'src/components/App.tsx', - content: 'const App = () =>Resolved update
', - }), - }, - }, - ], - }, - }, - ], - }), - }) - }) - - await waitForAppReady(page, `${appEntryPath}`) - await connectByotWithSingleRepo(page) - await setComponentEditorSource(page, 'const App = () =>Before
') - await openWorkspaceTab(page, 'App.tsx') - await connectOpenRouterKey(page) - - await page.getByLabel('Ask AI assistant').fill('Update App tab only.') - await page.getByRole('button', { name: 'Send' }).click() - - await expect( - page.getByRole('button', { name: 'Apply update to App.tsx' }), - ).toBeVisible() - await page.getByRole('button', { name: 'Apply update to App.tsx' }).click() - - await expect( - page.locator('.editor-panel[data-editor-kind="component"] .cm-content').first(), - ).toContainText('Resolved update') -}) - -test('AI chat renders a single apply action for multiple targets resolving to the same tab', async ({ - page, -}) => { - await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { - const body = route.request().postDataJSON() as ChatRequestBody | null - - if (body?.stream) { - await route.fulfill({ - status: 502, - contentType: 'application/json', - body: JSON.stringify({ message: 'stream intentionally disabled in this test' }), - }) - return - } - - await route.fulfill({ - status: 200, - contentType: 'application/json', - body: JSON.stringify({ - choices: [ - { - message: { - role: 'assistant', - content: 'Prepared updates for App tab.', - tool_calls: [ - { - id: 'call_component_id', - type: 'function', - function: { - name: 'propose_editor_update', - arguments: JSON.stringify({ - target: 'component', - content: 'const App = () =>By id
', - }), - }, - }, - { - id: 'call_component_path', - type: 'function', - function: { - name: 'propose_editor_update', - arguments: JSON.stringify({ - target: 'src/components/App.tsx', - content: 'const App = () =>By path
', - }), - }, - }, - ], - }, - }, - ], - }), - }) - }) - - await waitForAppReady(page, `${appEntryPath}`) - await connectByotWithSingleRepo(page) - await setComponentEditorSource(page, 'const App = () =>Before
') - await openWorkspaceTab(page, 'App.tsx') - await connectOpenRouterKey(page) - - await page.getByLabel('Ask AI assistant').fill('Update App tab once.') - await page.getByRole('button', { name: 'Send' }).click() - - await expect(page.getByRole('button', { name: 'Apply update to App.tsx' })).toHaveCount( - 1, - ) -}) - -test('AI chat shows guidance when an editor update target cannot be matched', async ({ - page, -}) => { - await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { - const body = route.request().postDataJSON() as ChatRequestBody | null - - if (body?.stream) { - await route.fulfill({ - status: 502, - contentType: 'application/json', - body: JSON.stringify({ message: 'stream intentionally disabled in this test' }), - }) - return - } - - await route.fulfill({ - status: 200, - contentType: 'application/json', - body: JSON.stringify({ - choices: [ - { - message: { - role: 'assistant', - content: '', - tool_calls: [ - { - id: 'call_unknown_target', - type: 'function', - function: { - name: 'propose_editor_update', - arguments: JSON.stringify({ - target: 'src/does-not-exist.ts', - content: 'export const value = 1', - }), - }, - }, - ], - }, - }, - ], - }), - }) - }) - - await waitForAppReady(page, `${appEntryPath}`) - await connectByotWithSingleRepo(page) - await setComponentEditorSource(page, 'const App = () =>Before
') - await openWorkspaceTab(page, 'App.tsx') - await connectOpenRouterKey(page) - - await page.getByLabel('Ask AI assistant').fill('Can you still see my tab content?') - await page.getByRole('button', { name: 'Send' }).click() - - await expect( - page.getByText( - 'Proposed editor update is ready, but I could not match its target to an open tab. Ask me to target the active tab or one of the listed tab ids or paths.', - ), - ).toHaveCount(1) - await expect(page.locator('button[data-action="request-apply"]')).toHaveCount(0) -}) - -test('AI chat sends the currently active tab when context is enabled', async ({ - page, -}) => { - let streamRequestBody: ChatRequestBody | undefined - - await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { - streamRequestBody = route.request().postDataJSON() as ChatRequestBody - - await route.fulfill({ - status: 200, - contentType: 'text/event-stream', - body: [ - 'data: {"choices":[{"delta":{"content":"ok"}}]}', - '', - 'data: [DONE]', - '', - ].join('\n'), - }) - }) - - await waitForAppReady(page, `${appEntryPath}`) - await connectByotWithSingleRepo(page) - await setStylesEditorSource(page, '.button { color: red; }') - await connectOpenRouterKey(page) - - await page.getByLabel('Ask AI assistant').fill('Use active tab context only.') - await page.getByRole('button', { name: 'Send' }).click() - await expect(page.getByText('Response streamed.', { exact: true })).toHaveText( - 'Response streamed.', - ) - - const systemMessages = streamRequestBody?.messages?.filter( - (message: ChatRequestMessage) => message.role === 'system', - ) - expect( - systemMessages?.some( - (message: ChatRequestMessage) => - message.content?.includes('- Active tab:') && - message.content?.includes('app.css'), - ), - ).toBe(true) - expect( - systemMessages?.some((message: ChatRequestMessage) => - message.content?.includes('Active tab source:'), - ), - ).toBe(true) - expect( - systemMessages?.some((message: ChatRequestMessage) => - message.content?.includes('Available tab targets (id and path):'), - ), - ).toBe(true) -}) - -test('AI chat streaming text still updates while latest undo actions are visible', async ({ - page, -}) => { - let requestCount = 0 - - await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { - requestCount += 1 - const body = route.request().postDataJSON() as ChatRequestBody | null - - if (requestCount <= 2) { - if (body?.stream) { - await route.fulfill({ - status: 502, - contentType: 'application/json', - body: JSON.stringify({ message: 'force fallback for proposal setup' }), - }) - return - } - - await route.fulfill({ - status: 200, - contentType: 'application/json', - body: JSON.stringify({ - choices: [ - { - message: { - role: 'assistant', - content: 'Prepared updates for styles editor.', - tool_calls: [ - { - id: 'call_styles', - type: 'function', - function: { - name: 'propose_editor_update', - arguments: JSON.stringify({ - target: 'src/styles/app.css', - content: '.button { color: rgb(10 20 30); }', - }), - }, - }, - ], - }, - }, - ], - }), - }) - return - } - - if (body?.stream) { - await route.fulfill({ - status: 200, - contentType: 'text/event-stream', - body: [ - 'data: {"choices":[{"delta":{"content":"Streaming "}}]}', - '', - 'data: {"choices":[{"delta":{"content":"works with undo visible."}}]}', - '', - 'data: [DONE]', - '', - ].join('\n'), - }) - return - } - - await route.fulfill({ - status: 200, - contentType: 'application/json', - body: JSON.stringify({ - choices: [{ message: { role: 'assistant', content: 'fallback text' } }], - }), - }) - }) - - await waitForAppReady(page, `${appEntryPath}`) - await connectByotWithSingleRepo(page) - await setStylesEditorSource(page, '.button { color: red; }') - await connectOpenRouterKey(page) - - await page.getByLabel('Ask AI assistant').fill('Suggest a styles update.') - await page.getByRole('button', { name: 'Send' }).click() - - await expect( - page.getByText('Prepared updates for styles editor.', { exact: true }), - ).toBeVisible() - await page.getByRole('button', { name: 'Apply update to app.css' }).click() - await expect( - page.getByRole('button', { name: 'Undo last apply for app.css' }), - ).toBeVisible() - - await page - .getByLabel('Ask AI assistant') - .fill('Are you still working on that last request?') - await page.getByRole('button', { name: 'Send' }).click() - - await expect(page.getByText('Streaming works with undo visible.')).toBeVisible() -}) - -test('AI chat falls back to non-streaming response when streaming fails', async ({ - page, -}) => { - let streamAttemptCount = 0 - let fallbackAttemptCount = 0 - const attemptedModels: string[] = [] - - await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { - const body = route.request().postDataJSON() as ChatRequestBody | null - if (typeof body?.model === 'string') { - attemptedModels.push(body.model) - } - - if (body?.stream) { - streamAttemptCount += 1 - await route.fulfill({ - status: 502, - contentType: 'application/json', - body: JSON.stringify({ message: 'stream failed' }), - }) - return - } - - fallbackAttemptCount += 1 - await route.fulfill({ - status: 200, - contentType: 'application/json', - body: JSON.stringify({ - rate_limit: { - remaining: 17, - reset: 1704067200, - }, - choices: [ - { - message: { - role: 'assistant', - content: 'Fallback response from JSON path.', - }, - }, - ], - }), - }) - }) - - await waitForAppReady(page, `${appEntryPath}`) - await connectByotWithSingleRepo(page) - await connectOpenRouterKey(page) - - const selectedModel = 'openai/gpt-6-astra' - await page.getByLabel('Chat model').selectOption(selectedModel) - await expect(page.getByLabel('Chat model')).toHaveValue(selectedModel) - - await page.getByLabel('Ask AI assistant').fill('Use fallback path.') - await page.getByRole('button', { name: 'Send' }).click() - - await expect(page.getByText('Fallback response loaded.', { exact: true })).toHaveText( - 'Fallback response loaded.', - ) - await expect(page.getByText('Fallback response from JSON path.')).toBeVisible() - expect(streamAttemptCount).toBeGreaterThan(0) - expect(fallbackAttemptCount).toBeGreaterThan(0) - expect(attemptedModels.length).toBeGreaterThan(0) - expect(attemptedModels.every(model => model === selectedModel)).toBe(true) -}) - -test('clearing chat removes previous conversation context from new request', async ({ - page, -}) => { - const streamBodies: ChatRequestBody[] = [] - - await page.route('https://openrouter.ai/api/v1/chat/completions', async route => { - const body = route.request().postDataJSON() as ChatRequestBody - if (body?.stream) { - streamBodies.push(body) - await route.fulfill({ - status: 200, - contentType: 'text/event-stream', - body: [ - 'data: {"choices":[{"delta":{"content":"ok"}}]}', - '', - 'data: [DONE]', - '', - ].join('\n'), - }) - return - } - - await route.fulfill({ - status: 200, - contentType: 'application/json', - body: JSON.stringify({ - choices: [{ message: { role: 'assistant', content: 'ok' } }], - }), - }) - }) - - await waitForAppReady(page, `${appEntryPath}`) - await connectByotWithSingleRepo(page) - await connectOpenRouterKey(page) - - await page.getByLabel('Ask AI assistant').fill('First conversation prompt') - await page.getByRole('button', { name: 'Send' }).click() - await expect(page.getByText('Response streamed.', { exact: true })).toBeVisible() - - await page.getByRole('button', { name: 'Clear', exact: true }).click() - await expect(page.getByText('Chat cleared.', { exact: true })).toBeVisible() - - await page.getByLabel('Ask AI assistant').fill('Second conversation prompt') - await page.getByRole('button', { name: 'Send' }).click() - await expect(page.getByText('Response streamed.', { exact: true })).toBeVisible() - - expect(streamBodies.length).toBeGreaterThanOrEqual(2) - const latestMessages = streamBodies[streamBodies.length - 1]?.messages ?? [] - const allLatestContent = latestMessages.map(message => message.content ?? '').join('\n') - - expect(allLatestContent).toContain('Second conversation prompt') - expect(allLatestContent).not.toContain('First conversation prompt') -}) - test('BYOT remembers selected repository across reloads', async ({ page }) => { test.setTimeout(90_000) diff --git a/playwright/helpers/app-test-helpers.ts b/playwright/helpers/app-test-helpers.ts index 8e3f0a6..0b7c580 100644 --- a/playwright/helpers/app-test-helpers.ts +++ b/playwright/helpers/app-test-helpers.ts @@ -45,7 +45,7 @@ const isRetryableGotoError = (error: unknown) => { return false } - return /WebKit encountered an internal error|Test timeout/i.test(error.message) + return /WebKit encountered an internal error|page\.goto: Timeout/i.test(error.message) } const navigateToApp = async (page: Page, path: string) => { diff --git a/src/index.html b/src/index.html index a79ebd0..3ce47a4 100644 --- a/src/index.html +++ b/src/index.html @@ -78,8 +78,9 @@