[BUG] Task remains assigned on engine after worker heartbeat timeout interrupts execution loop

[matheusgr]

Describe the issue

Hatchet engine assigns a task to an worker that miss a heartbeat due to DEADLINE EXCEEDED.

Hatchet thinks that worker is running this task, but the worker itself doesn't not assign or acknowledge the task as executing.

Environment

• SDK: python 1.17.1
• Engine: Cloud

Expected behavior

Task should be assigned or fail as the heartbeat fails.

Code to Reproduce, Logs, or Screenshots

Relevant workflow description on hatchet cloud:

2025-10-16T16:56:23.218 - Task queued
2025-10-16T16:56:23.230 - Assigned to worker
2025-10-16T17:11:23.711 - Execution timed out (Task exceeded timeout of 15m)

Relevant log on worker:

2025-10-16T16:56:30.206699778Z - [WARNING]\t🪓 -- 2025-10-16 16:56:30,206 - interrupted read_with_interrupt task of action listener

debug_error_string = \"UNKNOWN:Error received from peer  {grpc_message:\"Deadline Exceeded\", grpc_status:4, created_time:\"2025-10-16T16:56:30.205401749+00:00\"
details = \"Deadline Exceeded\"
status = StatusCode.DEADLINE_EXCEEDED

grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
    raise _create_rpc_error("}
  File \"/app/lib/python3.11/site-packages/grpc/aio/_call.py\", line 327, in __await__
    await self.aio_client.Heartbeat(
  File \"/app/lib/python3.11/site-packages/hatchet_sdk/clients/dispatcher/action_listener.py\", line 110, in heartbeat
Traceback (most recent call last):

2025-10-16T16:56:30.206659795Z - [ERROR]\t🪓 -- 2025-10-16 16:56:30,205 - failed to send heartbeat

2025-10-16T16:56:23.195498103Z - [INFO]\t🪓 -- 2025-10-16 16:56:23,195 - finished step run: XXXXXXXXXXX

Additional context

It seems that the assigned action’s execution path is never completed because the heartbeat failure triggers an interrupt that prematurely cancels the action listener loop.

When a DEADLINE_EXCEEDED occurs during the Heartbeat() RPC:

1. The heartbeat() coroutine catches the error and calls self.interrupt.set().
2. The main _generator() loop (which is awaiting self.interrupt.wait()) wakes up.
3. It sees that its read_with_interrupt task isn’t done yet, so it:
   • Cancels the read task (t.cancel()),
   • Cancels the gRPC listener (listener.cancel()),
   • Breaks out of the loop.
4. This stops the action listener coroutine entirely, so the currently assigned action never finishes execution or reports completion.
5. Meanwhile, the server still thinks the worker is alive and running the action, since no error or unsubscribe message is sent.

Maybe hatchet should avoid interrupting the listener on transient heartbeat failures.

[deepakstwt]

Hi Sir 👋,

I’d like to work on this issue
From what I understand, the task remains assigned on the engine even after a worker heartbeat timeout interrupts the execution loop.

I can investigate and propose a fix. Could you please assign this to me?

[mrkaye97]

hey @matheusgr - thanks for the report here! I'll take a look

[mrkaye97]

hey @matheusgr - I'm following up here a bit! one thing I'm a bit unclear on is this:

so the currently assigned action never finishes execution

one important distinction is that the actions run in a separate loop than the action listener (it's called the action runner), which means that breaking out of the action listener loop for a failed heartbeat has no effect on the ability of an action to finish running. also, we don't explicitly send an unsubscribe since missing heartbeats imply the worker is dead, but if we miss (iirc) 4 heartbeats in a row, we declare the worker dead.

it'd be really helpful if you could confirm for a single task that you see the assignment, then the missed heartbeat, and then the incorrect behavior. I'm trying to reproduce this locally but am struggling (I think the behavior right now is correct, at least as far as I'm understanding).

Here's some example logs. In this case, we assign a task, it starts, we stop heartbeating, and the task still completes and reports completion to the engine which is registered properly, even when the worker is not heartbeating. Let me know if this situation does not match what you were reporting:

[DEBUG] 🪓 -- 2025-10-31 20:26:08,730 - sending heartbeat
[DEBUG] 🪓 -- 2025-10-31 20:26:08,733 - acquired action listener: 0645326e-9a6e-44f6-8199-e405a787e840
[DEBUG] 🪓 -- 2025-10-31 20:26:08,733 - sending heartbeat
[INFO]  🪓 -- 2025-10-31 20:26:11,690 - rx: start step run: eaa49aed-e676-4204-9375-459ede0623d2/simple:simple
[DEBUG] 🪓 -- 2025-10-31 20:26:11,690 - tx: event: simple:simple/1
[INFO]  🪓 -- 2025-10-31 20:26:11,691 - run: start step: simple:simple/eaa49aed-e676-4204-9375-459ede0623d2
[DEBUG] 🪓 -- 2025-10-31 20:26:11,691 - tx: event: simple:simple/1
[DEBUG] 🪓 -- 2025-10-31 20:26:11,691 - start time: 0.00057220458984375
[DEBUG] 🪓 -- 2025-10-31 20:26:12,734 - sending heartbeat
[DEBUG] 🪓 -- 2025-10-31 20:26:12,736 - sending heartbeat
[DEBUG] 🪓 -- 2025-10-31 20:26:16,742 - sending heartbeat
[ERROR] 🪓 -- 2025-10-31 20:26:17,739 - failed to send heartbeat
Traceback (most recent call last):
  File "/Users/matt/Documents/GitHub/hatchet/sdks/python/hatchet_sdk/clients/dispatcher/action_listener.py", line 110, in heartbeat
    await self.aio_client.Heartbeat(
  File "/Users/matt/Library/Caches/pypoetry/virtualenvs/hatchet-sdk-7ExloYWe-py3.11/lib/python3.11/site-packages/grpc/aio/_call.py", line 327, in __await__
    raise _create_rpc_error(
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
        status = StatusCode.DEADLINE_EXCEEDED
        details = "Deadline Exceeded"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:4, grpc_message:"Deadline Exceeded"}"
>
[WARNING]       🪓 -- 2025-10-31 20:26:17,741 - interrupted read_with_interrupt task of action listener
[DEBUG] 🪓 -- 2025-10-31 20:26:20,752 - sending heartbeat
[INFO]  🪓 -- 2025-10-31 20:26:21,692 - finished step run: simple:simple/eaa49aed-e676-4204-9375-459ede0623d2
[DEBUG] 🪓 -- 2025-10-31 20:26:21,692 - tx: event: simple:simple/2
[DEBUG] 🪓 -- 2025-10-31 20:26:21,741 - sending heartbeat
[INFO]  🪓 -- 2025-10-31 20:26:21,745 - listener established
[WARNING]       🪓 -- 2025-10-31 20:26:21,745 - time since last successful heartbeat: 13.01s, expects 4s
[DEBUG] 🪓 -- 2025-10-31 20:26:25,746 - sending heartbeat

[matheusgr]

Hi @mrkaye97

This log corresponds to the worker allocated for the task, but I don’t see any entry indicating that the related task was actually assigned to the worker.

I don’t have access to the full workflow execution, but if you can share the taskExternalId of this failed task, I can look it up in our logs to confirm this and investigate further.

( workflow: 692b1e41-a683-486f-97ab-dfadfae69841 actionId: documentworkflow:procreate)

[mrkaye97]

hey @matheusgr, sorry for just circling back. I'm not sure I understand this bit:

This log corresponds to the worker allocated for the task, but I don’t see any entry indicating that the related task was actually assigned to the worker.

Do you mean that the task showed up on the worker but there was no assignment log? What do you see in the dashboard?