Detect backend completion behind stale Thinking UI

Author: TanChuping

README.md

@@ -6,7 +6,7 @@ Local-first stall detection and safe recovery for long-running OpenAI Codex task
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
 [![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://www.python.org/)
 
-**Codex Task Watchdog** is an unofficial, Windows-first **OpenAI Codex watchdog** and installable **Codex skill**. It detects stalled tool calls, missing SSE response activity, and long-running tasks; distinguishes opaque model preparation from completed-command inactivity; diagnoses oversized Codex threads under context pressure; and builds same-task-first recovery plans. It is deliberately conservative: it records evidence and notifies you, but never automatically retries a quota-spending or side-effecting tool, sends a prompt, creates a task, kills Codex, or deletes task data.
+**Codex Task Watchdog** is an unofficial, Windows-first **OpenAI Codex watchdog** and installable **Codex skill**. It detects stalled tool calls, missing SSE response activity, long-running tasks, and backend-completed turns whose UI may still display Thinking; distinguishes opaque model preparation from completed-command inactivity; diagnoses oversized Codex threads under context pressure; and builds same-task-first recovery plans. It is deliberately conservative: it records evidence and notifies you, but never automatically retries a quota-spending or side-effecting tool, sends a prompt, creates a task, kills Codex, or deletes task data.
 
 The project is useful when Codex Desktop remains on “Thinking”, a tool call never returns to the agent, a batch silently stops progressing, or an old task has accumulated too much history to continue safely.
 
@@ -20,6 +20,7 @@ The project is useful when Codex Desktop remains on “Thinking”, a tool call
 | Detects response-stream and tool-completion stalls | Treat every slow model response as a failure |
 | Tracks explicit long operations with unique `arm` / `heartbeat` / `disarm` tags | Guess which parallel call completed |
 | Classifies absence-only evidence as review due and explicitly unsafe to interrupt | Treat missing completed commands or unchanged files as proof that an Agent stopped working |
+| Emits a one-time backend-completed receipt after a reviewed stall | Claim it can inspect or clear the Codex renderer's busy flag |
 | Produces local Windows notifications and bounded incident records | Send telemetry or upload logs |
 | Builds bounded, same-task-first recovery plans and metadata-only manifests | Automatically send wake prompts, create tasks, fork, or compact giant histories |
 | Prunes only watchdog-owned metadata under strict retention rules | Automatically retry tools, spend quota, or repeat side effects |
@@ -33,6 +34,7 @@ Several different failures look identical in the UI:
 - a tool completed but the next model request never began;
 - the Agent is composing code or preparing the next model request, so no completed command or file change is visible yet;
 - the client or app-server disconnected, so a timer cannot wake the active agent;
+- the backend completed, but the client missed the terminal UI update and remains on Thinking;
 - the task is technically alive but its accumulated context is now operationally risky.
 
 Open Codex reports describe related symptoms, including [long periods without child-agent health/progress signals](https://github.com/openai/codex/issues/16900) and [Codex Desktop remaining on Thinking while Stop fails](https://github.com/openai/codex/issues/24287). The watchdog supplies external evidence and a safe recovery boundary; it does not claim to repair the Codex scheduler or network connection itself.
@@ -65,7 +67,7 @@ The default detector thresholds are:
 | Tool started, but no matching completion | 180 seconds | 600 seconds |
 | Explicitly armed operation with no verified progress | — | selected rolling no-progress threshold |
 
-A model-preparation event, new response request, stream event, terminal event, or matching completion clears stale transition state. Parallel calls remain isolated by `call_id` and explicit jobs use unique tagged generations. Absence-only incidents carry `confirmed_failure: false` and `safe_to_interrupt: false`; they notify and record, but do not prove failure or authorize interruption.
+A model-preparation event, new response request, stream event, terminal event, or matching completion clears stale transition state. If a previously reviewed stall later reaches a positive backend completion event, the watchdog emits one `backend_completed_after_stall_review` receipt. Its recovery action is to refresh the client if Thinking remains—not to wait, wake, stop, or retry completed work. Parallel calls remain isolated by `call_id` and explicit jobs use unique tagged generations. Absence-only incidents carry `confirmed_failure: false` and `safe_to_interrupt: false`; they notify and record, but do not prove failure or authorize interruption.
 
 For explicitly armed work, 180 seconds is no longer a universal limit. The main conversation selects a **rolling no-progress threshold** based on the expected silent interval:
 
@@ -182,7 +184,7 @@ Build a bounded recovery decision for one task without waking or stopping it:
 python $Watchdog recover-plan --thread '<thread-id>' --turn '<turn-id>'
 ```
 
-The plan requires the supervising Agent to inspect the live Codex task, including incomplete model output. If the task is active or opaque, it stays untouched. If it is confirmed terminal or idle and unfinished, the preferred recovery is one continuation message to that same task. A small disk handoff and a clean user-visible task are fallbacks for an unrecoverable or critically oversized history. The watchdog itself never sends the continuation or creates a new task.
+The plan requires the supervising Agent to inspect the live Codex task, including incomplete model output. If the task is active or opaque, it stays untouched. If the backend is terminal-completed while the UI still shows Thinking, the plan returns `backend_completed_refresh_client_if_busy`: do not wait or retry; switch tasks and back, then reload Codex if necessary. Recent terminal evidence can be reconstructed from a bounded 1,000-row read-only log query after watchdog cache pruning or restart. If it is confirmed terminal or idle and unfinished, the preferred recovery is one continuation message to that same task. A small disk handoff and a clean user-visible task are fallbacks for an unrecoverable or critically oversized history. The watchdog itself never sends the continuation or creates a new task.
 
 When the user explicitly asks for a new sidebar-visible task, the skill now requires the supervising Agent to use Codex's `list_projects` and `create_thread` tools immediately, pass only the small handoff and exact next action, and return a `::created-thread{threadId="..."}` receipt. A subagent, background worker, Quick Chat, or a promise to create a task does not satisfy the request. If the tool is unavailable, the Agent must report failure honestly and offer `codex://threads/new?prompt=...&path=...` or `Ctrl+N`; the deep link only pre-fills the composer.
 
@@ -254,6 +256,7 @@ This repository focuses on a small Windows-first, local event-state detector plu
 - A quiet model may be actively reasoning or composing a patch rather than stuck; thresholds are review points, not proof or permission to interrupt.
 - It does not repair proxy, VPN, TUN, DNS, provider, or OpenAI service problems.
 - It does not promise exactly-once tool execution after a disconnect.
+- It cannot inspect or mutate the Codex renderer's busy flag; it can only prove backend completion and tell you when a client refresh is the safe recovery.
 - Automatic retries, automatic UI prompts, process killing, and destructive session cleanup are out of scope.
 
 ## Development

llms.txt

@@ -4,7 +4,7 @@ Repository: https://github.com/TanChuping/codex-task-watchdog
 License: MIT
 Status: unofficial community project; Windows-first; local-first; experimental compatibility with current Codex Desktop logs.
 
-Codex Task Watchdog is an installable Codex skill and Python watchdog for stalled tool calls, tool timeouts, missing SSE response activity, long-running Codex tasks, oversized thread diagnosis, session health, agent observability, bounded metadata cleanup, safe handoff recovery, and explicit creation of user-visible sidebar recovery tasks.
+Codex Task Watchdog is an installable Codex skill and Python watchdog for stalled tool calls, tool timeouts, missing SSE response activity, long-running Codex tasks, stale Thinking indicators after backend completion, oversized thread diagnosis, session health, agent observability, bounded metadata cleanup, safe handoff recovery, and explicit creation of user-visible sidebar recovery tasks.
 
 Primary skill: skills/codex-watchdog/SKILL.md
 Watchdog CLI: skills/codex-watchdog/scripts/codex_watchdog.py
@@ -20,6 +20,7 @@ Safety invariants:
 - one opaque tag per long-running attempt and retry generation;
 - caller-selected rolling no-progress thresholds by task class, rather than one universal 180-second cap;
 - threshold expiry triggers main-agent or monitoring-agent review; healthy current evidence can heartbeat and continue;
+- a reviewed stall followed by positive backend completion emits one non-retry UI-refresh receipt;
 - recovery manifests contain metadata only;
 - an explicit request for a new visible task uses `list_projects` and `create_thread`, never a subagent or Quick Chat substitute;
 - successful visible task creation must return a `::created-thread` receipt;

skills/codex-watchdog/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: codex-watchdog
-description: Monitor long-running Codex tools and diagnose stalled or oversized tasks. Use for watchdog, 看门狗, stuck tool calls, tool timeouts, 卡死监控, session health, adaptive no-progress thresholds, bounded cleanup, safe handoff recovery, explicit visible sidebar task creation during recovery, enabling or disabling local monitoring, or work expected to run longer than 30 seconds. Never auto-retry side effects or delete Codex data.
+description: Monitor long-running Codex tools and diagnose stalled, oversized, or UI-stale tasks. Use for watchdog, 看门狗, stuck Thinking indicators after backend completion, stuck tool calls, tool timeouts, 卡死监控, session health, adaptive no-progress thresholds, bounded cleanup, safe handoff recovery, explicit visible sidebar task creation during recovery, enabling or disabling local monitoring, or work expected to run longer than 30 seconds. Never auto-retry side effects or delete Codex data.
 ---
 
 # Codex Watchdog
@@ -44,6 +44,8 @@ Then recover in this order:
 4. After a reconnect, verify actual outputs before any retry. A missing UI notification is not proof that a side effect failed.
 5. Use a small disk handoff and ask for a clean task only when same-task recovery is impossible or thread health is `critical`. Never fork or clone a critical history.
 
+If `recover-plan` reports `backend_completed_refresh_client_if_busy`, the backend has positively completed even if Codex still displays Thinking. Do not wait, wake, stop, or retry that work. Switch to another task and back; reload Codex only if the stale indicator remains. The report reconstructs recent terminal evidence from at most 1,000 matching read-only log rows when the in-memory detector cache was restarted or pruned. The watchdog cannot observe or mutate the renderer's busy flag, so it records `ui_state: unobservable` and emits a one-time completion notification when a previously reviewed stall later reaches a backend terminal event.
+
 Treat `severity: review`, `evidence_class: absence_only`, or `safe_to_interrupt: false` literally. These are review notices, not stall verdicts. The watchdog must remain lower-cost than the work it monitors; avoid repeated broad log scans or diagnostic prompts that preempt normal work.
 
 ## Create a visible sidebar task when explicitly requested

skills/codex-watchdog/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "Codex Watchdog"
-  short_description: "Review stalls and create explicit visible recovery tasks"
-  default_prompt: "Use $codex-watchdog to diagnose without interrupting active work, then use same-task recovery or create a visible sidebar task when explicitly requested."
+  short_description: "Review stalls, stale Thinking UI, and safe recovery"
+  default_prompt: "Use $codex-watchdog to distinguish active work, backend completion with stale UI, and real recovery needs without retrying completed work."

skills/codex-watchdog/references/protocol.md

@@ -69,6 +69,8 @@ After reconnecting, run `status`, `list --limit 50`, and `incidents --limit 20`,
 
 For task recovery, run `recover-plan`, inspect the live task with the Codex thread/task interface, and prefer one continuation message to the same task only after it is confirmed terminal or idle and unfinished. If it is active or opaque, leave it untouched. Do not have the monitoring task redo the target task. Use a small disk handoff and a clean task only when same-task recovery is impossible or health is `critical`.
 
+When a turn that already produced a stall warning/review later emits a positive backend completion event, record one `backend_completed_after_stall_review` incident with `severity: resolved`, `evidence_class: positive_terminal`, and `recommended_action: refresh_client_if_ui_still_busy_do_not_retry`. Notify once that the backend completed. If the UI still shows Thinking, switch away and back or reload the client; never resend, wake, stop, or replay completed work. Keep terminal detector state for six hours so delayed UI-stale reports remain diagnosable. If that cache was restarted or pruned, `recover-plan` reconstructs the requested turn from at most 1,000 matching rows in `logs_2.sqlite` opened read-only. The daemon cannot observe or clear the renderer's busy flag.
+
 ### Visible sidebar task handoff
 
 When the user explicitly requests a new visible task after a recovery decision, execute the request in the same turn: