Skip to content

feat(desktop): add desktop pet window with Synergy state awareness - #1220

Open
yzxoi wants to merge 6 commits into
SII-Holos:devfrom
yzxoi:synergy/session-fe3040
Open

feat(desktop): add desktop pet window with Synergy state awareness#1220
yzxoi wants to merge 6 commits into
SII-Holos:devfrom
yzxoi:synergy/session-fe3040

Conversation

@yzxoi

@yzxoi yzxoi commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Summary

在 Electron shell 中新增桌面宠物(desktop pet)窗口:透明、置顶、不占任务栏,主进程订阅 Synergy 全局事件流(GET /global/event?stream=delta),将 session 事件映射为 7 态动画状态机(idle / working / happy / celebrate / angry / sleepy / dragging),驱动沙箱化的精灵图渲染器,支持拖拽与点击交互。

Changes

  • packages/desktop/src/pet-window.tsDesktopPetWindow 创建 BrowserWindow{ transparent, frame:false, alwaysOnTop, skipTaskbar, sandbox },主进程建立 SSE 连接并推送 pet:state / pet:settings / pet:sprite 到渲染器;pet.poke / pet.dragBy / pet.setDragging / pet.getState IPC 均做 sender 校验。
  • packages/desktop/src/pet-state.ts — 纯 PetStateMachine:session.updated busy → working、session.completion → celebrate、session.error → angry、空闲超时 → sleepy、poke → happy、dragging 覆盖;瞬态情绪到期回落。
  • packages/desktop/src/pet-settings.ts — 版本化 desktop-pet.json(userData),含 enabled / spritePath / width / height / position / idleTimeoutMs / frameMs,缺省回退。
  • packages/desktop/src/pet-sprite.ts — 8:7 精灵图校验(PNG/JPEG/WebP 头部尺寸解析,无完整解码器),转 base64 data URL;无效素材回退 CSS 占位。
  • packages/desktop/src/pet-page.ts + pet-preload.ts — 严格 CSP(default-src 'none')的内联 HTML,canvas 精灵动画 + 拖拽/点击;窄 preload 桥,渲染器不接触服务器 URL/SDK/Node。
  • packages/desktop/src/main.tsinitializePetWindow()(窗口就绪后)、服务器重启 setServerUrlbefore-quit 停止;desktop:build 增加 pet-preload.cjs 产物。
  • 测试test/pet-state / pet-settings / pet-sprite / pet-page / pet-window 五个套件;MockElectronWindow 补充 position/bounds/alwaysOnTop 等方法。

Verification

cd packages/desktop
bun run typecheck          # pass
bun test test/*.test.ts    # 251 pass, 5 skip, 0 fail
bun run desktop:build      # pet-preload.cjs 产物生成
cd ../..
bun run format:check       # pass
bunx oxlint                # 0 warnings/errors
bun run decision:check     # pass
bun run doc:check          # pass
bun run test-layout:check  # pass
bun run quality:quick      # 12/14 pass; 2 个失败为本地缺 gitleaks/actionlint 工具(环境问题,与本次变更无关)

Notes

  • Phase 1(MVP)范围:窗口 + SSE 订阅 + 状态机 + 交互 + 设置持久化 + 精灵图校验/热重载。
  • Phase 2(对话生成精灵图闭环)与 Phase 3(气泡/右键菜单/多角色)为后续迭代,见 docs/research/desktop-pet-proposal.md
  • 决策记录:docs/decisions/implemented/feature/2026-08-20-desktop-pet-window.md

Add a transparent always-on-top pet window to the Electron shell that subscribes to the Synergy global event stream (GET /global/event) in the main process, maps session events to a seven-mood animation state machine (idle/working/happy/celebrate/angry/sleepy/dragging), and drives a sandboxed sprite-sheet renderer with drag and click interaction.

Introduces pet-window/pet-state/pet-settings/pet-sprite/pet-sse/pet-page/pet-preload modules, a versioned desktop-pet.json settings file, 8:7 sprite-sheet validation with data-URL fallback, and bundled pet-preload in desktop:build. Settings persist position, size, idle timeout, and frame rate; server restarts re-point the SSE subscription and quit stops the window.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

The pet page's CSS fallback blob used literal product colors (#7dd3fc / #38bdf8 / #0284c7 and rgba(...)), which violates the frontend semantic color contract enforced by the App color-token-contract test. Inject the desktop shell skin's semantic tokens (markBackground, control, focus, border, text) as CSS variables in the pet document and reference only those in the fallback styles, mirroring how the startup page derives its palette from the theme snapshot.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
@synergy-agent

synergy-agent Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

All checks are done and the review is complete. I have one blocking integration finding (the SSE endpoint the pet subscribes to does not exist server-side), two non-blocking bugs, and a couple of suggestions — all with file:line evidence.


Review — #1220 desktop pet window

Reviewed 1cb7e6e31 against dev. The unit-test surface is solid and every claimed check reproduces locally (bun run typecheck ✓, bun test test/*.test.ts → 251 pass / 5 skip / 0 fail ✓, desktop:build emits dist/pet-preload.cjs ✓, bun run format:check ✓). The sandbox/security posture (sender-checked IPC, data:-only renderer, strict CSP, deny window-open/navigation) is good. But there is one blocking integration defect that the test suite cannot catch, plus two smaller bugs.

Blocking

B1 — The pet subscribes to a non-existent endpoint; it will never receive an event.
packages/desktop/src/pet-window.ts:205 connects to ${base}/global/event?stream=delta, but the server registers no such route. The only event routes are:

  • /global/event/wsWebSocket (packages/synergy/src/server/server.ts:714), used by the Web app (packages/app/src/context/global-sdk.tsx:98)
  • /eventSSE (packages/synergy/src/server/server.ts:1512, operationId: "event.subscribe")

GET /global/event falls through to the SPA fallback (or 404 if mountApp hasn't run), so PetSseClient either sees a 200 HTML body (declares connected, reads no data: lines, stream ends, no reconnect — pet sits on "idle" forever) or loops on 404. The pet's core feature (state awareness) never activates.

The tests can't catch this: installFetchMock (packages/desktop/test/pet-window.test.ts:26-53) returns a canned 200 SSE stream for any URL, so the "connects to the global event SSE stream" test passes vacuously. A test asserting the exact URL against a route table would be needed.

Note the docs encode the same wrong endpoint and need updating with the fix: docs/decisions/implemented/feature/2026-08-20-desktop-pet-window.md:13 and docs/research/desktop-pet-proposal.md:35,72,117.

Two fix options:

  1. Recommended: subscribe to the existing global WebSocket stream /global/event/ws?stream=delta, mirroring global-sdk.tsx:98. This is the only truly global stream: frames are { directory, payload } (server.ts:725-732), so unwrap .payload before feeding PetStateMachine (the SSE /event route sends bare payloads, server.ts:1539-1543, which is why the current client parses {type, properties} directly — a direct switch to /event would also be wrong on frame shape).
  2. Add a real GET /global/event SSE route server-side (new server surface + OpenAPI metadata + docs). More work than option 1.

B1b — even /event (SSE) is not global. provideRequestScope resolves /event with no directory/scopeID to Scope.home() (server.ts:366-372), and Bus.subscribeAll subscribes to that scope's bus (packages/synergy/src/bus/index.ts:203-205); scoped publishes only reach the publisher's scope state (bus/index.ts:106-111). Only the GlobalBus fan-out (bus/index.ts:120, consumed by the WS route at server.ts:749) carries project/channel session events. So even fixing the URL to /event would leave the pet blind to non-home sessions — reinforcing option 1.

Non-blocking

N1 — VP8L (lossless WebP) height is decoded from the wrong bits. packages/desktop/src/pet-sprite.ts:84-89: after the 0x2f signature at byte 20, height-1 is 14 bits at bitstream offset 14 — i.e. ((byte22 >> 6) & 0x3) | (byte23 << 2) | ((byte24 & 0xf) << 10), not readUint14LE(view, 23) (which reads byte 23 as the low byte). Valid lossless WebP sheets will be rejected and fall back to the CSS placeholder. The VP8 and VP8X paths are correct; the test suite has no WebP fixture (only PNG/JPEG, test/pet-sprite.test.ts), which is why it passed. Add a VP8L fixture test with the fix.

N2 — pet.dragBy writes desktop-pet.json on every pointer move. packages/desktop/src/pet-window.ts:276-277 calls savePetSettings per dragBy IPC, and the renderer sends one per pointermove (packages/desktop/src/pet-page.ts:219-225). A drag session can be dozens of synchronous file writes. Persist on drop (after setDragging(false)) or throttle.

Suggestions

  • packages/desktop/src/pet-window.ts:38-40screen.getAllDisplays()[0] is not guaranteed to be the primary display; use screen.getPrimaryDisplay().workArea.
  • The renderer starts with mood "idle" and only changes on pet:state (pet-page.ts:95); the main process broadcasts only on SSE status/events (pet-window.ts:22-30). After the endpoint fix, send an initial broadcastState() right after loadURL so a pet that starts while a session is busy isn't stuck showing idle until the next event.

Synergy: review complete — 1 blocking (dead SSE endpoint), 2 non-blocking, 2 suggestions

The pet window connected to GET /global/event?stream=delta, but the Synergy
server registers the bus event stream as a top-level GET /event SSE route
(server.ts /event, supporting ?stream=delta); /global/event does not exist
and falls through to the SPA fallback, so the pet never received session
events and working/celebrate/angry moods could not trigger. Point
connectSse() at ${base}/event?stream=delta, fix the test assertion that
mirrored the wrong URL, and correct the endpoint references in module
comments, the decision record, and the research proposal.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
@yzxoi

yzxoi commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up fix: pet SSE subscription endpoint

Light Loop review found the pet subscribed to GET /global/event?stream=delta, which does not exist on the server (the bus event stream is the top-level GET /event SSE route, supporting ?stream=delta; /global/event/ws is the WebSocket variant with a {payload:{...}} frame wrapper). The pet therefore never received session events.

Fix in commit 885f31169c:

  • packages/desktop/src/pet-window.tsconnectSse() now connects to ${base}/event?stream=delta.
  • packages/desktop/test/pet-window.test.ts — assertion updated to /event?stream=delta.
  • Comments and docs (pet-sse.ts, pet-state.ts, decision record, research proposal) updated to the real endpoint.

Verified locally: bun run typecheck pass, bun test test/*.test.ts 251 pass / 5 skip / 0 fail. CI on the pushed head: Test, Typecheck, Quality, Desktop Checks, Package Validation, Smoke Test, Secret Scan, Workflow Validation, Windows Checks all pass (Coverage report-only job still running).

yzxoi and others added 3 commits August 20, 2026 13:16
The Coverage CI job failed because src/pet-preload.ts was never loaded by
any test (measured 39, missing 1). Add test/pet-preload.test.ts which loads
the preload through the electron mock and asserts the synergyPet bridge
exposes poke/dragBy/setDragging/getState/onState/onSettings/onSprite with
the correct IPC channels and payloads, plus subscription disposer behavior.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
…poses

The pet page drag gesture invoked bridge.moveBy(dx, dy) while the preload
bridge exposes dragBy(dx, dy), so every drag threw a TypeError and the
window never moved. Add a jsdom-based gesture test that pins the drag
path to dragBy and drop the nonexistent moveBy call.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
Hovering over the pet window fired pointermove with no pressed button, and
the drag threshold was measured against the initial (0,0) origin, so mere
hover instantly entered the dragging state and dragBy chased the cursor,
making the window oscillate rapidly under the pointer. Track a pressed
flag from pointerdown/pointerup and ignore pointermove until a button is
actually pressed.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
@yzxoi

yzxoi commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

CI Test 红不是这个 PR 引入的:根因是 tool-scheduler.test.ts 的 shutdown 测试把全局 ToolScheduler 置为停止态且不恢复,#1219 移动 shard 边界后同进程的 auto-expand 真实 dispatch 全被拒("Tool scheduler is stopping")。dev 已合入修复 cdb24f79arebase 到最新 dev 后 Test 应转绿

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant