fix(amr): improve unavailable model gating #14650
ci.yml
on: pull_request
Resolve runner profiles
10s
Matrix: playwright_critical
Matrix: playwright_visual
Matrix: ui_p0
Validate Nix flake
3m 32s
Preflight
3m 10s
Workspace unit tests
1m 44s
Windows tools-pack payload tests
2m 36s
Web workspace tests
3m 20s
E2E Vitest
2m 30s
UI P0 smoke
3m 36s
Docker image build
3m 26s
Runtime summary
7s
Annotations
28 errors, 3 warnings, and 2 notices
|
E2E Vitest
Process completed with exit code 1.
|
|
tests/amr/turn.test.ts > AMR chat-run end-to-end > drives /api/runs against vela ACP and the assistant message captures the fake stream:
e2e/lib/vitest/runs.ts#L75
Error: run 7da7bf66-8ace-488d-8739-01dcd8cb15bd reached failed, expected succeeded
β― waitForRunStatus lib/vitest/runs.ts:75:13
β― tests/amr/turn.test.ts:299:27
β― runToolsDevSuite lib/vitest/suite.ts:265:5
β― tests/amr/turn.test.ts:204:5
|
|
tests/amr/relogin-required.test.ts > AMR relogin-required run failures > uses daemon AMR runtime credentials for pre-run login status:
e2e/lib/vitest/runs.ts#L75
Error: run bab40fd4-f52f-45f8-a9ec-ea4e83aa73c3 reached failed, expected succeeded
β― waitForRunStatus lib/vitest/runs.ts:75:13
β― tests/amr/relogin-required.test.ts:137:11
β― runToolsDevSuite lib/vitest/suite.ts:265:5
β― tests/amr/relogin-required.test.ts:107:9
β― Object.withEnv [as env] lib/vitest/suite.ts:202:12
β― tests/amr/relogin-required.test.ts:100:5
|
|
tests/amr/relogin-required.test.ts > AMR relogin-required run failures > uses configured AMR profile env for pre-run login status:
e2e/lib/vitest/runs.ts#L75
Error: run 5ba0b586-eb19-465a-a415-8391b313c4fc reached failed, expected succeeded
β― waitForRunStatus lib/vitest/runs.ts:75:13
β― tests/amr/relogin-required.test.ts:91:9
β― runToolsDevSuite lib/vitest/suite.ts:265:5
β― tests/amr/relogin-required.test.ts:62:7
β― Object.withEnv [as env] lib/vitest/suite.ts:202:12
β― tests/amr/relogin-required.test.ts:60:5
|
|
tests/amr/logout-state-persistence.test.ts > AMR logout state persistence > a previously working AMR session stops working after local logout and requires re-login:
e2e/lib/vitest/runs.ts#L75
Error: run 3ac7bda2-da67-41cb-91ba-240e3a344ae0 reached failed, expected succeeded
β― waitForRunStatus lib/vitest/runs.ts:75:13
β― tests/amr/logout-state-persistence.test.ts:55:9
β― runToolsDevSuite lib/vitest/suite.ts:265:5
β― tests/amr/logout-state-persistence.test.ts:19:7
β― Object.withEnv [as env] lib/vitest/suite.ts:202:12
β― tests/amr/logout-state-persistence.test.ts:18:5
|
|
tests/amr/insufficient-balance.test.ts > AMR insufficient balance run failures > fails the run with a recharge-facing AMR error when fake vela reports insufficient balance:
e2e/tests/amr/insufficient-balance.test.ts#L53
AssertionError: expected 'id: 1\nevent: pipeline_stage_started\β¦' to contain 'AMR_INSUFFICIENT_BALANCE'
- Expected
+ Received
- AMR_INSUFFICIENT_BALANCE
+ id: 1
+ event: pipeline_stage_started
+ data: {"kind":"pipeline_stage_started","runId":"fd9bbe76-8320-41a8-83d3-d005c9d63875","snapshotId":"2c1b0711-cedd-4e71-9c5c-9bd3b0b4fcb7","stageId":"plan","iteration":0,"startedAt":1783583944102}
+
+ id: 2
+ event: pipeline_stage_completed
+ data: {"kind":"pipeline_stage_completed","runId":"fd9bbe76-8320-41a8-83d3-d005c9d63875","snapshotId":"2c1b0711-cedd-4e71-9c5c-9bd3b0b4fcb7","stageId":"plan","iteration":0,"completedAt":1783583944106,"converged":true}
+
+ id: 3
+ event: pipeline_stage_started
+ data: {"kind":"pipeline_stage_started","runId":"fd9bbe76-8320-41a8-83d3-d005c9d63875","snapshotId":"2c1b0711-cedd-4e71-9c5c-9bd3b0b4fcb7","stageId":"generate","iteration":0,"startedAt":1783583944106}
+
+ id: 4
+ event: pipeline_stage_completed
+ data: {"kind":"pipeline_stage_completed","runId":"fd9bbe76-8320-41a8-83d3-d005c9d63875","snapshotId":"2c1b0711-cedd-4e71-9c5c-9bd3b0b4fcb7","stageId":"generate","iteration":0,"completedAt":1783583944106,"converged":true}
+
+ id: 5
+ event: pipeline_stage_started
+ data: {"kind":"pipeline_stage_started","runId":"fd9bbe76-8320-41a8-83d3-d005c9d63875","snapshotId":"2c1b0711-cedd-4e71-9c5c-9bd3b0b4fcb7","stageId":"critique","iteration":0,"startedAt":1783583944107}
+
+ id: 6
+ event: pipeline_stage_completed
+ data: {"kind":"pipeline_stage_completed","runId":"fd9bbe76-8320-41a8-83d3-d005c9d63875","snapshotId":"2c1b0711-cedd-4e71-9c5c-9bd3b0b4fcb7","stageId":"critique","iteration":0,"completedAt":1783583944107,"converged":true}
+
+ id: 7
+ event: start
+ data: {"runId":"fd9bbe76-8320-41a8-83d3-d005c9d63875","agentId":"amr","bin":"amr","streamFormat":"acp-json-rpc","projectId":"d1802a64-9b82-4d90-acaa-fcc3204b9591","cwd":"/home/runner/_work/open-design/open-design/.tmp/e2e/e2e-amr-insufficient-balance-1783583941255-1gyyvh/scratch/data/projects/d1802a64-9b82-4d90-acaa-fcc3204b9591","model":"default","reasoning":null,"toolTokenExpiresAt":"2026-07-09T08:14:04.111Z"}
+
+ id: 8
+ event: agent
+ data: {"type":"status","label":"model","model":"glm-5"}
+
+ id: 9
+ event: agent
+ data: {"type":"status","label":"waiting_for_first_output","elapsedMs":16}
+
+ id: 10
+ event: error
+ data: {"message":"json-rpc id 3: session/set_model must be called before session/prompt"}
+
+ id: 11
+ event: run_retry_finished
+ data: {"page_name":"chat_panel","area":"chat_panel","project_id":"d1802a64-9b82-4d90-acaa-fcc3204b9591","conversation_id":"7e81d19f-110e-44bb-b53d-058f675bfd05","run_id":"fd9bbe76-8320-41a8-83d3-d005c9d63875","retry_of_run_id":"fd9bbe76-8320-41a8-83d3-d005c9d63875","retry_attempt_index":1,"retry_max_attempts":1,"retry_strategy":"same_run_transient","agent_provider_id":"amr","model_id":"default","failure_category":"process_exit","failure_detail":"terminated_unknown","failure_stage":"child_close","error_code":"AGENT_SIGNAL_SIGTERM","retry_result":"suppressed","retry_suppressed_reason":"non_retryable_category"}
+
+ id: 12
+ event: diagnostic
+ data: {"type":"runtime_close","rpc_close_reason":"fatal_rpc_error","status":"failed","exit_code":1,"signal":"SIGTERM"}
+
+ id: 13
+ event: end
+ data: {"code":1,"signal":"SIGTERM","status":"failed","resumable":false}
+
+
β― tests/amr/insufficient-balance.test.ts:53:22
β― runToolsDevSuite lib/vitest/suite.ts:265:5
β― tests/amr/insufficient-balance.test.ts:17:5
|
|
tests/amr/auth-error-convergence.test.ts > AMR auth error convergence > marks the run and assistant message as failed when fake vela returns an auth error during prompt:
e2e/tests/amr/auth-error-convergence.test.ts#L56
AssertionError: expected 'id: 1\nevent: pipeline_stage_started\β¦' to match /AMR_AUTH_REQUIRED/
- Expected:
/AMR_AUTH_REQUIRED/
+ Received:
"id: 1
event: pipeline_stage_started
data: {\"kind\":\"pipeline_stage_started\",\"runId\":\"fef7a1ab-eec5-4abd-a32e-8478ea3483d3\",\"snapshotId\":\"92a635a5-a9fa-419c-b22b-4c0d9b1d16ec\",\"stageId\":\"plan\",\"iteration\":0,\"startedAt\":1783583947624}
id: 2
event: pipeline_stage_completed
data: {\"kind\":\"pipeline_stage_completed\",\"runId\":\"fef7a1ab-eec5-4abd-a32e-8478ea3483d3\",\"snapshotId\":\"92a635a5-a9fa-419c-b22b-4c0d9b1d16ec\",\"stageId\":\"plan\",\"iteration\":0,\"completedAt\":1783583947627,\"converged\":true}
id: 3
event: pipeline_stage_started
data: {\"kind\":\"pipeline_stage_started\",\"runId\":\"fef7a1ab-eec5-4abd-a32e-8478ea3483d3\",\"snapshotId\":\"92a635a5-a9fa-419c-b22b-4c0d9b1d16ec\",\"stageId\":\"generate\",\"iteration\":0,\"startedAt\":1783583947627}
id: 4
event: pipeline_stage_completed
data: {\"kind\":\"pipeline_stage_completed\",\"runId\":\"fef7a1ab-eec5-4abd-a32e-8478ea3483d3\",\"snapshotId\":\"92a635a5-a9fa-419c-b22b-4c0d9b1d16ec\",\"stageId\":\"generate\",\"iteration\":0,\"completedAt\":1783583947627,\"converged\":true}
id: 5
event: pipeline_stage_started
data: {\"kind\":\"pipeline_stage_started\",\"runId\":\"fef7a1ab-eec5-4abd-a32e-8478ea3483d3\",\"snapshotId\":\"92a635a5-a9fa-419c-b22b-4c0d9b1d16ec\",\"stageId\":\"critique\",\"iteration\":0,\"startedAt\":1783583947627}
id: 6
event: pipeline_stage_completed
data: {\"kind\":\"pipeline_stage_completed\",\"runId\":\"fef7a1ab-eec5-4abd-a32e-8478ea3483d3\",\"snapshotId\":\"92a635a5-a9fa-419c-b22b-4c0d9b1d16ec\",\"stageId\":\"critique\",\"iteration\":0,\"completedAt\":1783583947628,\"converged\":true}
id: 7
event: start
data: {\"runId\":\"fef7a1ab-eec5-4abd-a32e-8478ea3483d3\",\"agentId\":\"amr\",\"bin\":\"amr\",\"streamFormat\":\"acp-json-rpc\",\"projectId\":\"9007a7ac-e88d-41c0-8bce-0cb714ff1b6e\",\"cwd\":\"/home/runner/_work/open-design/open-design/.tmp/e2e/e2e-amr-auth-error-convergence-1783583944882-kl5n25/scratch/data/projects/9007a7ac-e88d-41c0-8bce-0cb714ff1b6e\",\"model\":\"default\",\"reasoning\":null,\"toolTokenExpiresAt\":\"2026-07-09T08:14:07.631Z\"}
id: 8
event: agent
data: {\"type\":\"status\",\"label\":\"model\",\"model\":\"glm-5\"}
id: 9
event: agent
data: {\"type\":\"status\",\"label\":\"waiting_for_first_output\",\"elapsedMs\":16}
id: 10
event: error
data: {\"message\":\"json-rpc id 3: session/set_model must be called before session/prompt\"}
id: 11
event: run_retry_finished
data: {\"page_name\":\"chat_panel\",\"area\":\"chat_panel\",\"project_id\":\"9007a7ac-e88d-41c0-8bce-0cb714ff1b6e\",\"conversation_id\":\"0520bd61-d5a2-4c3f-9f8f-4aaefddb7ec2\",\"run_id\":\"fef7a1ab-eec5-4abd-a32e-8478ea3483d3\",\"retry_of_run_id\":\"fef7a1ab-eec5-4abd-a32e-8478ea3483d3\",\"retry_attempt_index\":1,\"retry_max_attempts\":1,\"retry_strategy\":\"same_run_transient\",\"agent_provider_id\":\"amr\",\"model_id\":\"default\",\"failure_category\":\"process_exit\",\"failure_detail\":\"terminated_unknown\",\"failure_stage\":\"child_close\",\"error_code\":\"AGENT_SIGNAL_SIGTERM\",\"retry_result\":\"suppressed\",\"retry_suppressed_reason\":\"non_retryable_category\"}
id: 12
event: diagnostic
data: {\"type\":\"runtime_close\",\"rpc_close_reason\":\"fatal_rpc_error\",\"status\":\"failed\",\"exit_code\":1,\"signal\":\"SIGTERM\"}
id: 13
event: end
data: {\"code\":1,\"signal\":\"SIGTERM\",\"status\":\"failed\",\"resumable\":false}
"
β― tests/amr/auth-error-convergence.test.ts:56:52
β― runToolsDevSuite lib/vitest/suite.ts:265:5
β― tests/amr/auth-error-convergence.test.ts:17:5
|
|
Web workspace tests
Canceling since a higher priority waiting request for ci-5335 exists
|
|
Docker image build
Canceling since a higher priority waiting request for ci-5335 exists
|
|
Docker image build
The operation was canceled.
|
|
Validate Nix flake
Canceling since a higher priority waiting request for ci-5335 exists
|
|
Validate Nix flake
The operation was canceled.
|
|
UI P0 (workspace-restoration)
Canceling since a higher priority waiting request for ci-5335 exists
|
|
UI P0 (workspace-restoration)
The operation was canceled.
|
|
UI P0 smoke
Canceling since a higher priority waiting request for ci-5335 exists
|
|
UI P0 smoke
The operation was canceled.
|
|
UI P0 (project-runtime)
Canceling since a higher priority waiting request for ci-5335 exists
|
|
UI P0 (project-runtime)
The operation was canceled.
|
|
UI P0 (project-workspace)
Canceling since a higher priority waiting request for ci-5335 exists
|
|
UI P0 (project-workspace)
The operation was canceled.
|
|
UI P0 (entry-settings)
Canceling since a higher priority waiting request for ci-5335 exists
|
|
UI P0 (entry-settings)
The operation was canceled.
|
|
Playwright visual (entry-navigation)
Canceling since a higher priority waiting request for ci-5335 exists
|
|
Playwright visual (entry-navigation)
The operation was canceled.
|
|
ui/visual-entry.test.ts:48:1 βΊ [P2] captures the visual home harness:
e2e/lib/playwright/visual.ts#L578
1) ui/visual-entry.test.ts:48:1 βΊ [P2] captures the visual home harness ββββββββββββββββββββββββββ
Error: locator.waitFor: Test timeout of 60000ms exceeded.
Call log:
- waiting for getByText('Loading Open Designβ¦') to be hidden
10 Γ locator resolved to visible <div class="od-loading-shell">Loading Open Designβ¦</div>
at ../lib/playwright/visual.ts:578
576 |
577 | export async function waitForVisualReady(page: Page): Promise<void> {
> 578 | await page.getByText('Loading Open Designβ¦').waitFor({ state: 'hidden', timeout: T.long });
| ^
579 | await expect(page.getByTestId('home-hero')).toBeVisible({ timeout: T.medium });
580 | await expect(page.getByTestId('home-hero-input')).toBeVisible({ timeout: T.medium });
581 | await page.evaluate(async () => {
at waitForVisualReady (/home/runner/_work/open-design/open-design/e2e/lib/playwright/visual.ts:578:48)
at gotoVisualHome (/home/runner/_work/open-design/open-design/e2e/lib/playwright/visual.ts:597:9)
at /home/runner/_work/open-design/open-design/e2e/ui/visual-entry.test.ts:50:3
|
|
ui/visual-entry.test.ts:48:1 βΊ [P2] captures the visual home harness:
e2e/ui/visual-entry.test.ts#L0
1) ui/visual-entry.test.ts:48:1 βΊ [P2] captures the visual home harness ββββββββββββββββββββββββββ
Test timeout of 60000ms exceeded.
|
|
Validate workspace
Process completed with exit code 1.
|
|
ci
Canceling since a higher priority waiting request for ci-5335 exists
|
|
Detect validation scopes
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
Docker image build
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: docker/build-push-action@v6, docker/metadata-action@v5, docker/setup-buildx-action@v3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
Validate Nix flake
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/setup-node@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
π Playwright Run Summary
19 passed (1.7m)
|
|
π Playwright Run Summary
3 passed (1.2m)
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
handoff-report-visual-pr
Expired
|
439 Bytes |
sha256:abac93f6f1c7ed2ee9c2cf5cd22355f3bc3efe4ab805c01d2e9ad89ad1b19513
|
|
|
ui-p0-ci-29002806723-smoke
Expired
|
269 KB |
sha256:3939f84a535c918eadbf0181c1be651ec7b10185c8d1a5215bd2715de9441aae
|
|
|
visual-pr-capture-5335-29002806723-entry-navigation
Expired
|
613 KB |
sha256:e8fdbc6f53e702969beaabe00fe0a36fcd3604838e5958f45d25055cba33dfa1
|
|
|
visual-pr-capture-5335-29002806723-settings-workspace
Expired
|
2.87 MB |
sha256:4cb60f6b0f4d45a96bfe063cd6984a61b7a1de143d07f98c62aa590a1725b9dd
|
|