[BUG] Agent stuck status=running after run completes — zombie run not auto-cleared, blocking all subsequent heartbeats

[aduduman031084]

Summary

Paperclip agents intermittently get stuck in status=running after their heartbeat run has already completed successfully. Because maxConcurrentRuns=1, no new heartbeat can start — the agent is silently dead to the platform while looking alive. This is a recurring, high-impact production bug that has now manifested at least 5 times on our instance since June 2026.

Recurrence History (our instance)

Date	Issue	Agent	Impact
2026-06-23	[FAI-5776]	Founding Engineer + DevOps	~90 min zombie, 3 issues 409/403-locked, pricing build blocked
2026-06-23	[FAI-5776]	DevOps Engineer	Same incident, concurrent zombie
2026-06-26	FAI-6087 (this)	Founding Engineer	7h+ zombie, Phase 2 work (FAI-6090) stranded overnight
Earlier	FAI-5355, FAI-5428, FAI-5462, FAI-5686	DevOps Engineer	Multiple silent-run reviews raised
Earlier	FAI-2588	Founding Engineer	"Standing in stale sessions multiple times in one day"

Every one of these was a productivity kill. An agent that should be building or reviewing sits frozen while the board sees a green "running" status and assumes work is progressing.

What Happens

1. Agent runs a heartbeat. Run completes with status=succeeded.
2. Paperclip does not flip agent.status from running → idle.
3. lastHeartbeatAt goes stale while agent.status stays running.
4. All new wake attempts are rejected (maxConcurrentRuns=1 concurrency guard).
5. Assigned issues collect 409/403 checkout conflicts as if the agent is busy.
6. Human or CEO agent must manually intervene: pause → resume (or board-level runtime-state/reset-session).

Root Cause (as far as we can tell)

The run completion callback / state-machine transition from run.status=succeeded → agent.status=idle is not firing reliably. This appears to be a race or a missed event in the agent lifecycle handler, not reproducible on demand but recurring under normal agentic load.

Why This Keeps Hurting Us

• Our zombie-run watchdog (a 15-min cron we built ourselves as a workaround, [FAI-5939]) was paused by the board on 2026-06-24 — meaning the platform's own lifecycle should handle this, not a user-space workaround.
• The runtime-state/reset-session recovery endpoint returns 403 Board access required when called from an agent JWT, so agent self-recovery is impossible. Board manual intervention is the only path.
• There is no platform-native alerting beyond "suspicious after 1h / critical after 4h" stale-run checks — by the time an alert fires, hours of work are lost.

Reproduction Steps

We cannot reproduce this on demand, but the conditions are:

• codex_local adapter
• wakeOnDemand: true, maxConcurrentRuns: 1
• Normal heartbeat run that completes (last observed: run completed ~19:16Z, zombie persisted until ~07:55Z next day — ~12 hours)

Expected Behavior

When a heartbeat run reaches a terminal state (succeeded, failed, cancelled), agent.status must atomically transition to idle (or paused if paused). No manual recovery should be needed.

Requested Fixes / Mitigations

1. Fix the state transition so run.completed → agent.status=idle is atomic and reliable.
2. Expose a self-service recovery endpoint (e.g. POST /api/agents/:id/recover-zombie) callable by agents with their own JWT, not just board-level tokens.
3. Platform-native watchdog built into the scheduler — not a user-space cron we have to maintain and remember to keep unpaused.
4. Alert loudness — a zombie older than 30 min should be auto-reaped or at minimum surfaced as a board-level notification, not just a stale-run evaluation issue.

Business Impact

Each zombie incident costs 2–12 hours of autonomous agent capacity and requires board-level manual intervention. On a team running autonomous agentic workflows 24/7, this is the single biggest reliability drag we have. We have built two generations of internal workarounds ([FAI-1166], [FAI-5778], [FAI-5939]) and the bug still reappears.

Please treat this as a high-priority platform reliability issue. It is seriously impacting our work and overall productivity

[stubbi]

Thanks for the extremely thorough write-up and the recurrence history — this is clearly a real, high-impact reliability problem and the expected behavior you describe (terminal run state must atomically flip agent.status back to idle/paused) is correct.

Triage: this is a control-plane / application bug, not an operator bug. This repo (paperclip-operator) only manages the Kubernetes resources that run the Paperclip app — StatefulSet, Service, Ingress, PVCs, credential Secrets. It has no visibility into or control over the agent run lifecycle, the heartbeat scheduler, agent.status, run.status, maxConcurrentRuns, or the runtime-state/reset-session endpoint. All of that logic lives in the Paperclip server process itself (the paperclipinc/paperclip application), not in the operator. Nothing the operator renders can fix or work around the missing run.completed -> agent.status=idle transition.

Concretely, all four of your requested fixes are application-layer changes:

1. The atomic state transition on terminal run status — server state machine.
2. A self-service POST /api/agents/:id/recover-zombie endpoint (currently reset-session is board-only) — server API + authz.
3. A scheduler-native watchdog to auto-reap zombies — heartbeat scheduler.
4. Zombie-age alerting/auto-reap thresholds — server.

I've labeled this upstream and am routing it to the Paperclip platform team's backlog (it lines up with the internal FAI tickets you referenced). Public issues are disabled on paperclipinc/paperclip, so I'm keeping this issue open here as the public tracking anchor and will update it as the platform-side fix lands.

If it's useful in the meantime: the safest operator-side stopgap is not a pod restart (as you note, that risks CrashLoopBackOff and doesn't clear a zombie) — keep your 15-min watchdog cron unpaused until the platform fix ships. Sorry that regressed on you.

Apologies for the impact — this is understood as high priority.