Add specific exit code or recovery mechanism when agent dies with active worker pods in Kubernetes

[Deddinho23]

Currently, when using Woodpecker in Kubernetes mode, if an agent (master) pod dies unexpectedly, all its associated worker pods also terminate with exit code 0.

This behavior creates a problem for our external automated systems that monitor pipeline status: since the exit code is 0, it is interpreted as a successful execution, making it impossible to detect and handle this scenario properly.

---

Why this matters

In many cases, the failure is transient and could be resolved by simply restarting the pipeline. However, without a distinct error code or recovery mechanism, we cannot reliably identify this condition.

---

Proposed Solutions

1. Introduce a specific exit code in case of agent failure.
   • This would allow external systems to detect and handle the situation properly.
2. Implement a recovery mechanism, such as:
   • When an agent dies while managing active worker pods, transfer ownership of these pods to another running agent replica.
   • Alternatively, store the state so that a new agent can resume unfinished work.

---

Current Workaround

We are considering running agents on more reliable nodes (e.g., on-demand instances) to minimize the risk, but this does not fully solve the problem.

---

Benefits

• Improves reliability and fault tolerance in Kubernetes deployments.
• Enables better automation and error handling for CI/CD pipelines.
• Reduces manual intervention and downtime.

---

Would this feature be feasible?
It would greatly improve resilience and observability for Kubernetes users of Woodpecker!