Task run modes (in-loop/thread/subprocess) & hard timeout enforcement

[guy-dono]

Problem

The execution_timeout and cancellation features work well as long as a task yields back to the event loop. But if the event loop gets blocked - for example, by a bad condition in a while loop - the timeout can’t be enforced. In that case the task hangs forever, and the worker stays stuck until it’s manually restarted. In larger codebases, especially with many contributors, this scenario is almost inevitable.

Proposed solution

Support the following executor modes, and allow tasks to specify them, so developers can choose the right level of isolation:

• inloop (default, runs in the event loop)
• subprocess (runs in a separate process, can be force-killed on timeout)
• thread (optional, fits the general feature but less relevant here. Can be used to run synchronous code, from what I found in this discussion I guess it's already implemented just auto-detected)

@hatchet.task(name="my_task", execution_timeout=timedelta(seconds=30), executor="subprocess")
async def my_task():
  while True:
    print("kill me")

Considerations

• Subprocesses add overhead but they're optional, and very useful for larger systems.
• On-Failure tasks already exist but subprocesses introduce a separation between "soft kill" and "hard kill" signals. That might require some thinking:
  • adding a dedicated "soft kill handler" and "hard kill handler" for subprocess tasks.
  • adding separate soft & hard timeouts
  • distinct end states (soft terminated vs hard terminated)
• Limit resources per subprocess (cpu time, memory)
• Capturing subprocess logs won't work natively because we configure the logger at the module level when initiating hatchet in the parent process, which means the logger won't be available in the subprocess. It's possible to get around this by streaming stderr from the subprocesses to the logger manually.

[mrkaye97]

hi @guy-dono - thanks for the suggestion! I think we've already implemented some of this functionality, and other parts we likely won't / can't do. here's some notes:

1. sync tasks are run in a thread (as you mention), so we have an implicit inloop / thread switch as you're describing. it's implicit, but IMO that's a good thing. it doesn't make sense to e.g. be able to define a task as inloop, but also make it sync with def. Note that this is also how FastAPI handles execution, so if you had an endpoint in FastAPI that was async and blocked, then your API would be blocked just the same as your Hatchet worker.
2. It's not feasible, at least for now, to run tasks in separate processes. This has been asked for by a handful of people, but the problem is that doing that would require things like the client and context to be serializable, which they are not. So it's a no-op for now AFAICT.

My opinion is that developers, regardless of the size of the codebase, should follow the advice of FastAPI:

If you just don't know, use normal def.

Ultimately, it's up to the developer to ensure they're using async / await properly. And while I agree that some blocking is likely, there are other things, such as running separate workers for separate teams (akin to microservices) that can work as a codebase scales.

I'm going to close this since I don't think there's more to do here, but hopefully that helps!