Task timeout support

[kbakk]

Is your feature request related to a problem? Please describe.
If a task takes longer than 10 seconds to complete (by default, can be changed), The Zeebe broker will mark the task as timed out. However, the task may continue to progress on the worker. In this case, as the broker will allow other workers to start the task, it will lead to the task being processed twice.

To prevent this, having a way to cancel the task (e.g. with SIGINT), would prevent the task from being worked on by multiple instances.

Describe the solution you'd like
ZeebeWorker could get an argument cancel_on_timeout that means that the task will be cancelled if it runs longer than the timeout allows.

Describe alternatives you've considered
The user could also implement such a functionality within the task. However, it's not clear what should happen in case it times out, as failing the task (letting an exception bubble up) would mark the task as failed on the broker. Maybe the fact that the worker should allow the task to time out (i.e. not send anything back to the broker) could be indicated with a TimeOutException being raised in the user's code.

[Chadys]

I prefer the alternative you provided, it makes the solution more generic since it can also be used for tasks that shouldn't be stopped by external code. The tasks needs to be aware of the "at least once" policy of Zeebe anyway to deal with idempotency and such, so making them responsible of correctly reacting to timeouts seams not to be such a stretch. Might be great however to add a very simple way for a task to check "have I timed out?".
We should probably also add a "Timeout" to pyzeebe.JobStatus enum.
From the documentation, it's not clear whether a timed out task that fail has any impact (it only says that a success is not relevant any more), so depending on that a special treatment by Pyzeebe of the TimeOutException will be needed or not (but I like the idea of an explicit exception in any case).

[kbakk]

There seems to be a feature in asyncio that can be used for this: https://docs.python.org/3/library/asyncio-task.html#timeouts

Relates to #143

[DanielEidlin]

Hi, I want to open a PR for this issue and want to make sure we're all on board with the changes.
I want to apply the changes suggested in previous comments:

• Raising TimeOutException when a task timeouts
• Adding Timeout to pyzeebe.JobStatus

Extra info: So digging around Zeebe's docs, it only mentions what happens when trying to send a complete job from a worker with a timed out job, but from what I've understood, because the job is reassigned, all commands are treated the same and are met with a NOT FOUND error. So that's applied to fail job as well. Anyway I don't think we should fail the job when TimeOutException is raised because it's both side-effectish and has no actual effect as I've just explained. I think we should keep it only as a way for developers to know their task has timed out and handle it.

Let me know if that seems good and I'll start working on the PR.
Thanks.

[DanielEidlin]

For async functions I'm using asyncio.wait_for. For sync functions it's a little trickier. There are some solutions but they aren't cross platform. I found this package that seems to support both unix and windows but it isn't supported by a large number of devs. Let me know what you think about using it.

[dimastbk]

Hi. Maybe add warning "timeout is not supported for sync handlers"?

[johannesloibl]

There's actually a reference to update_job_timeout: https://camunda-community-hub.github.io/pyzeebe/zeebe_adapter_reference.html#pyzeebe.grpc_internals.zeebe_adapter.ZeebeAdapter.update_job_timeout
But this is missing in the code. Is this related?
I was also looking for a way to increase the timeout dynamically but it seems the actual function is missing.

[rb83]

Hm, we're already doing this today. Of course, we're using the async decorators (that allow declare a custom timeout), and with custom exception handlers we're modifying the timeouts dynamically, even doing exponential back-off in some cases.