Testing hatchet tasks/workflows

[TheCaffinatedDeveloper]

We are strongly considering Hatchet as an alternative to our current distributed framework we built on top of Celery. The two contenders are Hatchet (a favorite since its closest to our current home grown solution) and Temporal(io).

With celery, we take advantage of it's "eager" mode which essentially gives us an in memory execution environment of our existing framework and allows us to do more unit level testing of input/output from a workflow perspective. We also leverage integration tests for well...integration.

Looks like Temporal has some testing utilities as well - time skip, sandbox, pytest examples etc.

What I don't see is testing documentation of any kind for Hatchet (or examples 😬) - I hope I'm wrong here. Can you guide me/the community on some utils, examples, and or documentation towards testing what we build with Hatchet?

[mrkaye97]

Hey @TheCaffinatedDeveloper - great questions here! We have a ton of integration / E2E tests in Pytest (we rely heavily on these tests for both testing the Python SDK itself as well as for testing new features of the engine).

The TL;DR here is we don't have any of this documented officially and are planning to work on that soon, but we have some well-worn E2E testing patterns that are working well for us. Unit testing is a bit harder given the amount of internal stuff we'd likely need to mock to get a unit test environment working, but it's on our radar to figure out how to make this more sane. Some thoughts on E2E testing below:

For our own CI / local testing, we have a test suite running against local Hatchet (you could use Hatchet lite for this). We basically have two different patterns:

1. A large worker with a number of test workflows registered on it
2. An "on-demand" worker, which can be requested by individual tests

Both workers run in the background in the test suite using a Pytest fixture, so they run similarly, but the mechanics are slightly different. For the "large" one, we just start it at the start of our test run and shut it down at the end, and for the on-demand one, it can be requested by individual tests as needed.

For individual tests, we've basically been combining the run methods and the API to run the tasks and then assert that whatever properties we're expecting actually occurred. For instance, that test I linked makes sure that tasks are not retried if we raise a non-retry exception. I've been quite happy with this - the API has enough flexibility that we can make lots of different types of assertions depending on what we need in any given test.

One thing we've been struggling with is that tests like this can be slow and racy, since they often rely on sleeps to allow things to process within Hatchet, etc. (this was expected from my PoV, so we just retry and it's pretty reliable but not 100%).

Another thing is isolation between tests, which you mentioned. This one I haven't figured out yet - you could probably achieve this in a bunch of ways though, like maybe prepending some test id to your workflows so that they don't leak between different tests (basically namespacing). Happy to discuss more here though, haven't found a miracle cure yet.

Misc other notes:

• I've considered + recommended (in some cases) using testcontainers for running the workers - tradeoffs there, but has some nice properties re: portability and isolation