Send now can overwrite active turn metadata and leave chat permanently in-flight

[yongkangc]

Summary

Clicking Send now for a queued message can leave a Codex chat permanently marked in-flight even though the native Codex session later emits task_complete. The chat then stops ingesting messages, queue pause requests time out, and recovery requires restarting Garcon.

Observed on deployed main commit 12b166f888ac2c6427ec226711f81ab48392a8f9.

Public incident snapshot: https://garcon-backtest.chiayong.com/shared/LfVl0HTGmFQrsgchgPAa-T0UvzHDdCEF

Reproduction / incident sequence

1. Run a Codex turn.
2. Queue another message while the turn is active.
3. Click Send now (agent-interrupt-and-send).
4. In this incident, Garcon logged an in-flight metadata overwrite while Codex continued running.
5. Native Codex eventually completed, but Garcon did not ingest the remaining messages or clear the active state.
6. A later queue pause failed with Failed to pause queue: signal timed out.

Evidence

Chat: 1784772965538199
Codex session: 019f8cc9-6984-7301-ac75-bca297be7691

UTC timeline:

• 02:36:27.111 — native Codex task_started
• 02:37:15.683 — last activity persisted by Garcon
• 02:37:25 — Garcon log: [agents:event-bus] agents: overwriting in-flight turn metadata for chat 1784772965538199
• 02:39:27.470 — native Codex emitted final agent_message
• 02:39:27.513 — native Codex emitted task_complete
• Garcon shared snapshot created later omitted everything after 02:37:15
• Queue pause subsequently timed out

Garcon itself remained healthy throughout:

• garcon.service active, NRestarts=0
• /api/v1/auth/status returned HTTP 200 in 1–7 ms
• no ENOSPC/OOM

This isolates the failure to per-chat execution/queue ownership, not service or disk health.

Suspected race

Relevant paths:

• server/chat-execution/chat-execution-coordinator.ts
  • #interruptActiveTurn() calls #abortStop(...), then always calls #requestDrain(...) in finally
  • #performStop() coordinates attempt settlement and provider abort
• server/agents/event-bus.ts
  • trackTurn() warns but still overwrites existing active metadata
  • subsequent terminal events only dispatch when identity matches current metadata
• server-agents/codex/src/agents/codex/app-server/runtime.ts
  • abort() interrupts and removes the active runtime session

Likely invariant violation: successor drain/launch can reach trackTurn() before prior turn ownership/event metadata is fully settled. trackTurn() then overwrites the old identity. A late completion from either side can no longer reconcile the correct turn, leaving Garcon stale even though native history has terminal proof.

PR #362 fixed stale client processing events from earlier turns, but this incident appears to be a server-side ownership/transcript reconciliation gap.

Expected behavior

• Send now atomically stops/settles the active attempt before launching the queued successor.
• Exactly one active turn identity exists per chat.
• Native terminal evidence (turn/completed / task_complete) clears matching ownership and is imported even across WebSocket disconnects or interrupt races.
• Queue pause remains responsive.

Acceptance criteria

• Add deterministic regression test: active Codex turn + queued successor + Send now, with delayed/late terminal events from interrupted turn.
• Assert trackTurn() never silently overwrites a live identity.
• Assert successor launches exactly once only after predecessor settlement.
• Assert late predecessor events cannot clear successor state.
• Assert successor final message is persisted and chat becomes idle.
• Assert queue pause returns promptly after completion.
• Add restart/native-history reconciliation test for a terminal native turn missing from Garcon state.

[yongkangc]

Additional live reproduction confirms a per-chat mutation-lock leak, not only stale processing metadata.

I attempted a point fork of the affected chat through the normal API:

POST /api/v1/chats/fork
{
  "sourceChatId": "1784772965538199",
  "chatId": "1784776176060330",
  "upToSeq": 39
}

upToSeq should use Codex's supported fork-at-message-while-running path and bypass whole-head settled-snapshot reconciliation. Instead:

• request remained blocked for >60 seconds with no HTTP response
• no fork target was created
• source nextForkOrdinal remained unchanged
• no fork validation/deferred-settlement log appeared
• Garcon service and unrelated HTTP endpoints remained healthy
• client request was terminated; no files were manually changed

This strongly suggests withChatMutationLocks([sourceChatId, targetChatId], ...) is waiting forever on the source chat lock left by the earlier Send now path. It also explains why queue pause returns client-side signal timed out: both controls wait behind the same leaked lock.

Suggested regression extension:

1. Active turn + queued input.
2. Trigger agent-interrupt-and-send at the problematic interleaving.
3. Assert source mutation lock is released on every success/failure/abort path.
4. Immediately point-fork at a persisted sequence and pause queue.
5. Assert both operations return promptly and no mutex waiter survives request cancellation.