Jobs stuck in "Running" state, how to properly upgrade?

[enzo-pauvy]

Hello,

I have 3 dkron instances running on 3 different servers.
I recently upgraded from dkron 4.0.4 to 4.0.9, after which many jobs were stuck in the "Running" state.

This is not the first time it has happened. I believe it usually occurs after a restart or something similar. From what I can see, the only workaround is to recreate the job or restore a backup (#1261).

I was wondering if this is a known issue, and what the proper way to upgrade or restart a node is.

For example:

• Should I upgrade the leader first or last?
• Should I just stop the node before upgrading?
• Should I expect to restore a backup after each upgrade?

Edit: It might not be related to the upgrade itself, but to version (4.0.9). Some jobs ended up stuck in "Running" without restarting or showing any errors.

Best regards,
Enzo

[sguilly]

<!> This analysis was performed with Claude.ai and confirmed with GPT and Gemini <!>

Title: Jobs with concurrency: forbid can get permanently stuck after a failed execution stream

Description:

A job configured with concurrency: forbid can become permanently blocked ("ghost job") if a gRPC streaming
failure occurs during execution. The job will never run again until its executions are manually purged via
the API.

Root cause

When a job execution starts, the execution record is persisted early in the Raft log with StartedAt set and
FinishedAt zero. If the gRPC stream between the leader and the agent breaks (network issue, timeout, node
restart, leader election…) and the subsequent ExecutionDone call also fails, the execution remains in the
store permanently with FinishedAt zero.

The concurrency check in IsRunnable queries GetRunningExecutions, which considers any execution with a
non-zero StartedAt and a zero FinishedAt as "running". Since an orphaned execution matches this condition,
every future scheduling tick is blocked and the job never runs again.

How to reproduce

1. Create a job with concurrency: forbid (HTTP executor jobs are most susceptible)
2. Trigger a network partition or node failure during job execution, such that both the gRPC stream and the
   fallback ExecutionDone RPC fail
3. Observe that the execution record in the store has StartedAt set but FinishedAt is zero
4. The job will no longer be scheduled — logs show "job: Skipping concurrent execution (found running
   executions in storage)"

Impact

• The job is silently dead — no error is surfaced, only an Info-level log line on each skipped tick
• There is no self-healing mechanism: no TTL, no timeout-based reaper, no periodic cleanup of stale
  executions
• The only workaround is a manual DELETE /v1/jobs/{jobName}/executions API call, which requires the
  operator to notice the problem first

Affected code path

• store.go — GetRunningExecutions: filters on !StartedAt.IsZero() && FinishedAt.IsZero()
• job.go — IsRunnable: blocks scheduling when running executions are found
• grpc_client.go — agentRunAttempt: two code paths (broken stream and normal EOF) where ExecutionDone can
  fail silently, leaving the orphaned record behind

[vcastellm]

Fixed in #1957