pipe-pane silently stops forwarding output after alternate-screen redraw burst (pane stays live, piped copy freezes)

[klabulan]

Summary

pipe-pane's forwarded copy of a pane's output can silently stop updating -- the tmux pane
itself keeps rendering correctly (confirmed via tmux capture-pane), but the external cat >> fifo process pipe_pane() starts never receives another byte, so FifoManager's reader (and
everything downstream: status detection, GET /terminals/{id}/output) freezes on stale content
indefinitely. Reproduced once, live, immediately after a burst of alternate-screen-buffer
redraws (a dialog transitioning to a full-screen splash then a ready prompt). Not yet reproduced
on demand / isolated to a minimal repro independent of that specific app-level trigger --
flagging this honestly, same as the original #382 report before its trigger was fully narrowed
down.

Environment

• cli-agent-orchestrator installed from main at commit 01472e274a23df335032eaf37ae9933275127f73
  (the commit that merged fix(fifo): non-blocking reader loop + event-loop-safe session teardown (fixes #382) #383/fixed DELETE /sessions can permanently wedge cao-server (D-state, fifo_open, unkillable) #382) -- so this is not the fifo_open deadlock; that fix
  is confirmed present and working (verified independently, see DELETE /sessions can permanently wedge cao-server (D-state, fifo_open, unkillable) #382's thread).
• tmux 3.6, Ubuntu, terminal_backend: tmux.
• Client: this project's own FastAPI gateway, calling cao-server's HTTP API only.
• Provider: claude_code, launched via the normal POST /sessions flow.

Reproduction

1. Launch a session (POST /sessions?provider=claude_code&agent_profile=developer...), land at
   Claude Code's "Yes, I trust this folder / No, exit" trust dialog (first launch in a new
   working directory).
2. Accept the dialog (POST /terminals/{id}/key?key=Enter). Claude Code transitions: trust
   dialog → full-screen splash box (╭─── Claude Code vX.X.X ───╮ ASCII-art box, "Welcome
   back!") → ready chat prompt, footer reading Not logged in · Run /login for this
   particular (never-authenticated) account.
3. Poll GET /terminals/{id}/output repeatedly over the following ~60s.

Observed: output length stops growing entirely partway through step 2 -- captured content
ends mid-transition, right after "...Yes, I trust this folder ✔", and never advances to show
the splash box, the ready prompt, or the Not logged in footer, even though tmux capture-pane -t <target> -p against the same pane at the same moment shows all of that content rendered
correctly and currently. Sending further input (POST /terminals/{id}/input, e.g. later typing
plain text) also never appears in output, even though tmux capture-pane shows the typed text
landing in the pane normally.

Diagnostic evidence

• The tmux session/window (cao-<name>:<window>) is alive and current the whole time --
  tmux capture-pane -p returns fresh, correct content at every check.
• pane_pipe format variable (tmux list-panes -F "#{pane_pipe}") reports 1 (piped) the
  whole time -- tmux itself believes the pipe is still active.
• The sh -c "cat >> <fifo>" process pipe_pane() starts is alive (not exited, not zombie).
  /proc/<pid>/wchan for its actual cat child reads unix_stream_data_wait -- it's blocked
  waiting for more data on the unix stream socket tmux uses to feed piped commands (ls -la /proc/<cat_pid>/fd/0 shows socket:[...]), and none is arriving.
• FifoManager's own reader thread for this terminal (fifo-<terminal_id>, confirmed by name
  via /proc/<cao_pid>/task/*/comm) is healthy -- wchan reads poll_schedule_timeout,
  matching the fixed non-blocking select()-based loop from fix(fifo): non-blocking reader loop + event-loop-safe session teardown (fixes #382) #383/fifo_reader.py exactly. It is
  correctly waiting for data; there just isn't any to read, because tmux stopped sending it.
• cao-server's own /health stays 200 throughout -- this is not a DELETE /sessions can permanently wedge cao-server (D-state, fifo_open, unkillable) #382 recurrence (no
  full-server wedge, no D-state, no thread leak: fifo-* thread count stayed proportional to
  active terminals, no accumulation observed).

Recovery, confirmed working: manually re-issuing the pipe restores live forwarding
immediately -- confirmed by typing text afterward and watching it appear in output right
away. Two details that mattered in practice:

• pipe_pane()'s tmux command always passes -o (toggle-if-not-already-piped). Since tmux
  still reports pane_pipe=1 for the stalled pane, calling pipe_pane() again just toggles
  the (already-stalled) pipe off instead of restarting it -- confirmed by checking
  pane_pipe immediately after, and had to be corrected with a bare (non--o) pipe-pane
  call to actually re-arm it. A recovery path needs stop_pipe_pane() then pipe_pane(), not
  just pipe_pane() alone.
• Historical output missed during the stall is permanently lost (nothing to retroactively
  recover from -- the bytes tmux failed to forward were never buffered anywhere CAO can reach).
  Only live forwarding going forward is restored.

Suggested direction (a starting point, not a proven fix)

Not proposing a full parallel/fallback poller (constants.py's existing
opencode_inbox_delivery_daemon precedent for OpenCode's own "pipe-pane output can stop
changing once the TUI settles" note is the closest existing pattern, but running a second
continuous data path for every terminal as a permanent workaround felt like treating the
symptom rather than the transport fault). A narrower, self-healing liveness check on
FifoManager seems like a better fit for its existing _lock/_readers bookkeeping:

_LIVENESS_CHECK_INTERVAL_S = 8.0  # cheap: one capture-pane call per active terminal per interval

def _liveness_watchdog_loop(self) -> None:
    while not self._shutdown.is_set():
        time.sleep(_LIVENESS_CHECK_INTERVAL_S)
        for terminal_id in list(self._readers.keys()):
            self._check_pipe_liveness(terminal_id)

def _check_pipe_liveness(self, terminal_id: str) -> None:
    """A stalled pipe-pane forwarder looks identical to a genuinely idle terminal from
    inside the FIFO reader (no bytes to read either way) -- the only way to tell them apart
    is comparing against tmux's own live pane state directly."""
    session_name, window_name = _resolve_window(terminal_id)  # already exists, terminal.py
    current_hash = hash(get_backend().get_history(session_name, window_name, tail_lines=50))
    last_hash, last_changed_at = self._liveness.get(terminal_id, (None, time.monotonic()))
    if current_hash != last_hash:
        self._liveness[terminal_id] = (current_hash, time.monotonic())
        return
    # Pane content is unchanged -- likely genuinely idle, not stalled. Only escalate if the
    # FIFO reader *also* hasn't seen fresh bytes in that same window (a real stall shows
    # divergence between "pane changed" and "FIFO delivered nothing" -- an idle terminal has
    # neither, which must not trigger a needless re-pipe).
    ...

def _rearm_pipe(self, terminal_id: str, session_name: str, window_name: str) -> None:
    fifo_path = str(FIFO_DIR / f"{terminal_id}.fifo")
    get_backend().stop_pipe_pane(session_name, window_name)  # pipe_pane()'s own `-o` would
    get_backend().pipe_pane(session_name, window_name, fifo_path)  # otherwise just toggle off
    logger.warning(f"Re-armed stalled pipe-pane forwarder for terminal {terminal_id}")

Genuinely unsolved in this sketch: distinguishing "pane content is unchanged because the
terminal is legitimately idle" from "pane content changed but the FIFO didn't get it" needs the
FIFO's own last-delivered-byte timestamp threaded in too (not shown above) -- and I don't have a
confirmed root cause for why tmux's own forwarding stalls in the first place (couldn't pin it
to a specific escape sequence or timing window beyond "a burst of alt-screen redraws" empirically
correlating with it once), so this is offered as a direction, not a validated patch. Happy to dig
further / help test candidate fixes against the same repro if useful -- this project can
reproduce a fresh unauthenticated Claude Code session's trust-dialog-to-splash transition on
demand.

[haofeif]

Thanks @klabulan would you like to raise a PR on this ?