Commit 8793664
refactor(galgame): UI cleanup + capture chain (dxcam→mss→pyautogui) + dead-code cleanup (#1191)
* feat(galgame): add PyAutoGuiCaptureBackend, drop PrintWindow from default chain
Refactor screenshot backend chain to dxcam → mss → pyautogui as the new
fallback strategy. PrintWindow is intentionally removed from the default
chain because:
- It's a "render to DC" mechanism that often produces stale frames on
DirectX/Unity games (already documented as `selected_legacy_fallback`
in the previous detail label).
- Slower than BitBlt-based backends (mss/pyautogui).
- Inner implementation already falls back to BitBlt when PrintWindow
itself fails, so its theoretical "capture occluded windows" advantage
doesn't always materialize.
PrintWindow is still kept reachable in two narrower roles:
1. Explicit `selection="printwindow"`: chain becomes
`[printwindow, dxcam, mss, pyautogui]` so power users who actually
need occluded-window capture still have it as the first attempt with
sane fallbacks if PrintWindow returns black/empty.
2. Smart-mode background target: still uses `[printwindow]` only since
it's the only backend that can plausibly capture an occluded window
(others read screen pixels). ocr_reader already emits
`backend_not_suitable_for_background` warning when quality is poor,
so the failure mode stays observable.
PyAutoGuiCaptureBackend is a thin wrapper over `pyautogui.screenshot()`
(already a main dep for brain/computer_use). On Windows it goes through
the same GDI path as mss; on macOS/Linux it shells out to platform
screenshot tools — slower than mss but battle-tested as a defense-in-depth
fallback.
New chain matrix:
| selection | chain |
|-------------|-------------------------------------------------|
| auto / dxcam| dxcam → mss → pyautogui |
| mss | mss → dxcam → pyautogui |
| pyautogui | pyautogui → dxcam → mss |
| printwindow | printwindow → dxcam → mss → pyautogui |
| smart fg | dxcam → mss → pyautogui |
| smart bg | [printwindow] only (occluded-window capture) |
Existing chain ordering tests updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(galgame-ui): remove rapidocr/dxcam install buttons, replace with bundled hint
PR #1188 removed the runtime install machinery for rapidocr/dxcam from
the Python side (HTTP routes, SDK actions, support helpers). The UI
side was left half-done: install buttons + JS handlers + i18n strings
were still in place. This commit completes the cleanup:
- static/main.js:
- Drop RAPIDOCR_INSTALL_URL / DXCAM_INSTALL_URL constants
- OCR_INSTALL_TABS = ['tesseract'] (was: rapidocr/dxcam/tesseract)
- Drop rapidocr/dxcam entries from getInstallUIConfig() registry
- Drop dependencySummaryItem rapidocr/dxcam branches (unreachable)
- Drop installRapidOcr / installDxcam / restoreRapidOcr*Install*State
- Drop case 'install_rapidocr' from action handler
- Drop init promise.all entries for rapidocr/dxcam restore
- Drop click event listeners for rapidocrInstallBtn / dxcamInstallBtn
- activeInstallTab default 'rapidocr' → 'tesseract'
- Add pyautoguiUseBtn click handler (new backend selection)
- static/main.js (renderRapidOcr / renderDxcam):
Rewrite both functions to drop install-button DOM lookups and all
install task state branches. Status banners (rapidocrPrompt,
dxcamPrompt) now show:
- Installed: ready/active state with detected_path + model_cache_dir
- Not installed: bundled hint pointing dev to `uv sync --group galgame`
or telling end-users to reinstall the packaged build
- broken_runtime: "rebuild venv" hint for upgrade users with stale
legacy installs
Also added pyautoguiUseBtn config in renderDxcam alongside other
capture-backend select buttons.
- static/index.html: drop <button id="rapidocrInstallBtn"> and
<button id="dxcamInstallBtn">; add <button id="pyautoguiUseBtn">.
- i18n × 5 locales (zh-CN/en/ja/ko/ru): symmetric -25/+7 — drop
install-action keys (action/retry/queued/success/failure/installing_*/
install_kicker/failed_title/completed_refresh_title/install_first_title/
missing_models_*) for both kinds; add bundled_hint, unavailable_title,
pyautogui.use/.using/.title.
- integration test test_galgame_bridge_ui_routes.py:
Drop assertions for now-removed JS strings (RAPIDOCR_INSTALL_URL,
DXCAM_INSTALL_URL, restoreRapidOcrInstallState, restoreDxcamInstallState)
and i18n key (ui.install.rapidocr.action). Re-target i18n bundle
smoke check at ui.install.tesseract.action which still exists.
Plugin loads cleanly with `--group galgame` synced and without (rapidocr
status reports detail=missing in the latter case, which renders as the
bundled_hint banner instead of an install button). Chain ordering tests
still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(galgame): drop force_reload param + _purge_modules dead helper
Proposal A from PR #1188 cleanup discussion. The runtime install path
that used `force_reload=True` to evict cached `rapidocr_onnxruntime`
sys.modules entries is gone (HTTP routes deleted, install_rapidocr
function deleted). The remaining `inspect_rapidocr_installation`
callsite always passes `force_reload=False`, so the parameter and the
helper it gates are dead code.
- `load_rapidocr_runtime` drops `force_reload: bool = False` param
- `_purge_modules` function deleted
- Sole remaining callsite (inspect_rapidocr_installation) updated
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(galgame): drop dead rapidocr_install_{manifest_url,timeout_seconds} cfg fields
Both fields fed only the runtime install machinery that PR #1188
deleted (HTTP fetch of install manifest JSON / subprocess timeout for
pip install). After install code removal, no consumer reads them.
Removed:
- `GalgameRapidOcrConfig.rapidocr_install_manifest_url`
- `GalgameRapidOcrConfig.rapidocr_install_timeout_seconds`
- Corresponding `_FIELD_MAP` entries in `GalgameConfig`
- Constructor kwargs in `service.py:_build_rapidocr_config`
- Test fixture entries in `test_ocr_pipeline.py`
Backward compat: old user cfg files with these keys → service.py loader
no longer calls `.get()` on them, so the dict entries are silently
ignored. No migration shim needed.
Kept (despite the misleading "install" prefix):
- `rapidocr_install_target_dir`: still actively used by ocr_reader as
the runtime model cache root (rapidocr writes downloaded model files
here). Renaming to `rapidocr_model_cache_root` belongs in a
follow-up — too invasive (touches schema + UI + i18n).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(galgame): address codex P1+P2 on PR #1191
P1: load_rapidocr_runtime caller in ocr_reader.py was missed by the
force_reload removal commit. RapidOcrReader._ensure_runtime still
passed `force_reload=False`, which would raise TypeError on first OCR
runtime init (cache miss / warmup path). Drop the kwarg from the
caller to match the new signature.
P2: OCR_INSTALL_TABS shrinking to ['tesseract'] left two issues:
- HTML still rendered tab buttons for rapidocr/dxcam → clicking them
coerced to tesseract via switchInstallTab fallback (UI inconsistency)
- revealCaptureBackendSettings() called switchInstallTab('dxcam') →
same coercion to tesseract instead of focusing dxcam panel
Drop the rapidocr / dxcam install-tab buttons from index.html (their
banners are always visible now, no tab gating). Drop the dead
switchInstallTab('dxcam') call from revealCaptureBackendSettings —
expandAndScrollTo('dxcamPrompt') alone scrolls correctly without
needing to activate a (now nonexistent) tab.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(galgame-ui): re-route install_rapidocr action to bundled hint banner
Codex P2 on PR #1191: removing `case 'install_rapidocr'` from the
action handler switch left dangling action emitters. `withRapidOcrInstallAction()`
(general diagnosis flow) and the onboarding action button still produce
`install_rapidocr` action IDs. Source-install users without `--group
galgame` would see the action button, click it, and hit the default
"action unavailable" warning — broken UX.
Keep the action ID for backward compat with all emitter sites, but
change semantics + label:
- Re-add `case 'install_rapidocr'` to the action handler — now scrolls
to `rapidocrPrompt` (where bundled_hint copy already explains
reinstall packaged build / `uv sync --group galgame`).
- Action label: "一键安装 RapidOCR" → "查看 RapidOCR 启用提示" (and 5 locales)
- New flash key `ui.flash.rapidocr_hint_revealed` (5 locales) shown when
the action triggers, telling user to follow the on-banner steps.
User who clicks now lands on the actionable hint instead of dead-end
warning. Action ID name "install_rapidocr" is misleading post-change
but fully internal; renaming would touch 5 client-side sites + i18n
key for cosmetic improvement only — not worth the diff.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(galgame-ui): renderPluginUnavailable safe for missing rapidocr/dxcam install config
Codex P1 on PR #1191: renderPluginUnavailable() iterates
['rapidocr', 'dxcam', 'tesseract', 'textractor'] and calls
getInstallConfig(kind) for each. Since rapidocr/dxcam were removed
from getInstallUIConfig() in commit e942d8e, this throws on the
first iteration — exactly the wrong moment, since this fallback
render runs precisely when the plugin is unavailable and diagnostics
are needed most.
Restructure the loop:
- Use a static PROMPT_LABELS table (rapidocr/dxcam labels are just
brand strings, no need to round-trip through the install registry).
- Address banner DOM elements by `${kind}Prompt` directly (the prefix
matches the kind for all 4).
- Guard `card`/`button` updates behind `kind in {'tesseract', 'textractor'}`
— bundled rapidocr/dxcam no longer have install button/card DOM
(removed in HTML cleanup).
- Defensive `if (banner)` early-continue: future-proof against
any banner being removed without breaking the whole render.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(galgame-ui): address coderabbit review on PR #1191 (3 actionable)
3 valid concerns from coderabbit review of commits 99e1c1f + e942d8e:
1. **PyAutoGUI multi-monitor bug** (Major) — PyAutoGuiCaptureBackend
used `pyautogui.screenshot(region=...)` which internally calls Pillow
`ImageGrab.grab()` without `all_screens=True`. PyAutoGUI 0.9.54
doesn't expose that flag, so secondary-monitor or negative-coordinate
windows silently truncate to primary screen → OCR sees garbage.
Switch to direct `ImageGrab.grab(bbox=..., all_screens=True)`.
`is_available()` still gates on `import pyautogui` to keep the
user-facing label honest. Comment explains why we bypass pyautogui's
wrapper. all_screens=True is Windows-only in Pillow but harmlessly
ignored on macOS/Linux.
2. **i18n PyAutoGUI casing** (Minor) — `ui.install.pyautogui.title`
used lowercase `pyautogui` while `.use` and `.using` use brand
`PyAutoGUI`. Was wrong in all 5 locales (en/ja/ko/ru/zh-CN, plus
the main.js fallback string), not just the zh-CN line coderabbit
flagged.
3. **Capture chain copy stale** (Minor) — `ui.install.capture_auto.title`
and `ui.flash.capture_backend_settings_revealed` still listed
"DXcam, MSS, PrintWindow" — missing PyAutoGUI which is now the
third default-chain backend (PrintWindow moved to explicit-only +
Smart background). Same issue across all 5 locales (coderabbit only
flagged ru, but the rest had identical wording). Fold PyAutoGUI into
the chain copy, note PrintWindow's narrower role.
i18n × 5 locale × 3 keys = 15 symmetric updates. ocr_reader.py change
is the substantive fix. Chain ordering tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(galgame): address codex/coderabbit review on PR #1191 (3 actionable)
3 valid follow-ups from review of commit 87e7a65:
1. **Plugin-unavailable fallback leaves bundled banners hidden** (codex P2,
main.js:2973): refreshAll() failure path rewrites RapidOCR/DXcam
banner text but doesn't clear HTML `hidden`. Post-refactor
switchInstallTab() only iterates ['tesseract'], so rapidocrPrompt /
dxcamPrompt never get unhidden in the exact failure mode where the
user needs the bundled-setup hint. Add explicit `banner.hidden = false`
in the loop body.
2. **PyAutoGUI is_available crashes on non-ImportError** (codex P2 +
coderabbit, ocr_reader.py:2505): pyautogui's import touches platform
display state; in headless / WSL / missing-DISPLAY environments it
may raise KeyError or RuntimeError instead of ImportError. With
`except ImportError`, the exception bubbles up and aborts capture
preflight. Widen to `except Exception` so backend probing degrades
cleanly to "unavailable".
3. **Korean common values not localized** (coderabbit, ko.json:1010):
`ui.common.none_value` was English `(none)`. While at it, also
fix `ui.common.auto_value` which had the same issue. Now (없음) /
(자동) — consistent with ja `(なし)/(自動)` and ru `(нет)/(авто)`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(galgame): address coderabbit review on PR #1191 (2 actionable, 1 follow-up test)
1. **Major: Explicit PrintWindow chain semantics broken** (ocr_reader.py)
`selection="printwindow"` chain was [printwindow, dxcam, mss, pyautogui]
for ALL targets. PrintWindow's only reason to exist post-bundling is
capturing occluded / background windows. If PrintWindow failed on a
background target, silent fallback to dxcam/mss/pyautogui reads screen
pixels — usually the occluding window, not the game — and OCR produces
confident garbage from the wrong source.
Fix: split _ordered_backends_for_target by target type, mirroring Smart
mode's pattern:
- Foreground target: keep [printwindow, dxcam, mss, pyautogui] (other
backends also see the right window, fallback safe)
- Background target: [printwindow] only (no silent screen-grab fallback)
- Minimized: raise printwindow:target_window_minimized_for_capture
New test test_win32_capture_backend_printwindow_strict_for_background_target
locks in the foreground/background asymmetry.
2. **Minor: revealCaptureBackendSettings + install_rapidocr action don't
expand collapsed parents** (main.js)
`expandAndScrollTo('dxcamPrompt')` / `expandAndScrollTo('rapidocrPrompt')`
only scroll. Both banners live inside `advancedSettings` + dependency
`<details>` which default to collapsed. After scroll the user lands on
a still-collapsed parent — the action effectively no-ops.
Fix: prefix with navigateToInstallPanel(kind) which expands both
containers (rapidocr/dxcam aren't in OCR_INSTALL_TABS so the tab-switch
side of navigateToInstallPanel is a noop, but the expansion runs).
Also synced JS fallback string for `capture_backend_settings_revealed`
to match the i18n key (already includes PyAutoGUI from prior commit;
the JS fallback was stale).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(galgame-ui): make navigateToInstallPanel scroll opt-out so banner scroll wins
Coderabbit minor on PR #1191: previous fix chained
`navigateToInstallPanel(kind)` + `expandAndScrollTo(banner)` to expand
collapsed parents before scrolling to the specific banner. But
navigateToInstallPanel queues `installSection.scrollIntoView()` in the
next animation frame — that fires AFTER expandAndScrollTo's
synchronous scroll, snapping the viewport back to installSection top
and hiding the banner the caller just landed on.
Add `scrollToSection: true` option to navigateToInstallPanel; the two
banner-targeted callers pass `false` to opt out of the queued scroll
while keeping the Advanced Settings + dependency `<details>`
expansion side effects.
- revealCaptureBackendSettings: navigateToInstallPanel('dxcam', { scrollToSection: false })
+ expandAndScrollTo('dxcamPrompt')
- case 'install_rapidocr': navigateToInstallPanel('rapidocr', { scrollToSection: false })
+ expandAndScrollTo('rapidocrPrompt')
Existing onboarding callers of navigateToInstallPanel keep the default
(scrollToSection=true) — they want the section-top scroll.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Hongzhi Wen <cartabio.coder1@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 8a56ef5 commit 8793664
16 files changed
Lines changed: 374 additions & 389 deletions
File tree
- plugin
- plugins/galgame_plugin
- i18n/ui
- static
- tests
- integration
- unit/plugins
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
145 | 145 | | |
146 | 146 | | |
147 | 147 | | |
148 | | - | |
| 148 | + | |
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
| |||
4055 | 4055 | | |
4056 | 4056 | | |
4057 | 4057 | | |
4058 | | - | |
| 4058 | + | |
4059 | 4059 | | |
4060 | 4060 | | |
4061 | 4061 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
96 | 96 | | |
97 | 97 | | |
98 | 98 | | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | 99 | | |
110 | 100 | | |
111 | 101 | | |
| |||
456 | 446 | | |
457 | 447 | | |
458 | 448 | | |
459 | | - | |
460 | 449 | | |
461 | 450 | | |
462 | 451 | | |
| |||
472 | 461 | | |
473 | 462 | | |
474 | 463 | | |
475 | | - | |
476 | | - | |
477 | | - | |
478 | 464 | | |
479 | 465 | | |
480 | 466 | | |
| |||
483 | 469 | | |
484 | 470 | | |
485 | 471 | | |
486 | | - | |
487 | | - | |
488 | | - | |
489 | | - | |
490 | 472 | | |
491 | | - | |
492 | 473 | | |
493 | 474 | | |
494 | 475 | | |
| |||
499 | 480 | | |
500 | 481 | | |
501 | 482 | | |
502 | | - | |
| 483 | + | |
503 | 484 | | |
504 | 485 | | |
505 | 486 | | |
| |||
533 | 514 | | |
534 | 515 | | |
535 | 516 | | |
536 | | - | |
| 517 | + | |
537 | 518 | | |
538 | 519 | | |
539 | 520 | | |
| |||
548 | 529 | | |
549 | 530 | | |
550 | 531 | | |
551 | | - | |
552 | 532 | | |
553 | 533 | | |
554 | 534 | | |
| |||
561 | 541 | | |
562 | 542 | | |
563 | 543 | | |
564 | | - | |
565 | | - | |
566 | | - | |
567 | | - | |
568 | | - | |
569 | 544 | | |
570 | 545 | | |
571 | 546 | | |
| |||
704 | 679 | | |
705 | 680 | | |
706 | 681 | | |
707 | | - | |
| 682 | + | |
708 | 683 | | |
709 | 684 | | |
710 | 685 | | |
| |||
1032 | 1007 | | |
1033 | 1008 | | |
1034 | 1009 | | |
1035 | | - | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
1036 | 1019 | | |
0 commit comments