Ensured reliability of job enqueueing?

[netsensei]

Hi,

I'm exploring solutions that allow me to orchestrate database transactions while reliably enqueuing jobs. I'm currently looking at River which leverages transactional enqueuing, bundling inserting a new job with changes made to data during a database transaction. This is possible because River's data model is entirely integrated within the application database, making River an integral part of the application. If a transaction is rolled back, the job goes with it.

Hatchet has a different architecture using an independent engine service, with it's own data store, that takes care of all the orchestration. I was wondering: To what extent would Hatch guarantee reliable worker registration (and returning errors) during an ongoing database transaction in a way that's comparable to the level of consistency that River offers through transactional safety?

Hatchet's documentation seems to hint at encapsulating database actions as well as long running processes as child tasks in a single Hatchet workflow. So, given a DAG workflow, how would I roll back a database transaction in a parent task, if a child task fails or times out, or vice versa? But would that be the correct approach?

Alternatively, suppose I start a database transaction in my application and then register a new job in Hatchet, triggering a workflow, Would it be possible to either reliably rollback the transaction if the job fails, or avoid the job from running / being registered at all if the transaction fails? Barring scenario's where intermittent failures on the wire leave the application and Hatchet in an inconsistent state (i.e. job registered, but the return ack message never reaches the application), that is.

I was wondering which insights the Hatchet team might offer on these questions. Thank you!

[abelanger5]

Hey @netsensei these are good questions! Unfortunately we don't currently have a mechanism to enqueue jobs as part of the same transaction as your core application (though we are designing a helper library to better support 2PC in Hatchet + Postgres setups). In this sense, the enqueueing model for Hatchet is no different from using a queue/database combination which doesn't support two-phase commit. As a result, for this type of consistency your best option is to create an outbox table in your application database which gets written to the same way you would with River, and periodically flush jobs from the outbox to Hatchet.

So, given a DAG workflow, how would I roll back a database transaction in a parent task, if a child task fails or times out, or vice versa? But would that be the correct approach?

I wouldn't recommend this approach, because all tasks in Hatchet are distributed between workers by default, so they may not be running on the same worker in between tasks.

More broadly, I'm doubtful that keeping a transaction open while processing work is the best approach here. I think this is only correct if:

1. All work is done in-process (i.e. no external API calls)
2. The work is done very quickly (<10ms)

Which is rare for things that run in the background (and if that is the case, you could very likely just put them in the same tx as your initial database write!). The reason for this is that you generally don't want more transactions going at a time than (very roughly) the number of cores on your database in a busy system - and if your transactions are often idle, modeling the load on your database becomes harder to reason about. And some other problems, like interfering with vacuum operations, min xids, that sort of thing.

So I would think about modeling your application state in a way that doesn't require a rollback of the transaction to handle errors, perhaps a sequence of local transactions like https://microservices.io/patterns/data/saga.html ?

With a durableTask, you can model a SAGA workflow through a sequence of try...catch blocks or equivalent, which would allow you to rollback work in the way that you're expecting.

[netsensei]

Hi @abelanger5

Thank you so much for your response. I have follow-up question:

As a result, for this type of consistency your best option is to create an outbox table in your application database which gets written to the same way you would with River, and periodically flush jobs from the outbox to Hatchet.

This makes sense. So, your suggestion would imply a model of eventual consistency. The Outbox table would act like an intermediate queue.

Before flushing a job from the Outbox queue, I would like to know if the job really was registered with Hatchet itself. Looking at the Go SDK, the RunNoWait() method, for instance, returns an err and a RunID. The error would allow trivial checking if anything went wrong while sending the job to Hatchet. But I was wondering what the semantics were behind returning the RunID. Did Hatchet actually persist the job in the PostgreSQL database with a hard promise to pass it onto an available worker at some point in time?

So I would think about modeling your application state in a way that doesn't require a rollback of the transaction to handle errors, perhaps a sequence of local transactions like https://microservices.io/patterns/data/saga.html ?

I can see how keeping a transaction open across distributed workers wouldn't work well. It sounds like the correct approach really is dividing the workflow into atomic tasks within which transactions are completed, and implementing the SAGA pattern explicitly adding compensating actions instead of relying on the underlying DBMS's rollback functionality.

[github-actions]

This issue has been stale for 30 days. Please update the issue or comment to keep it active. Otherwise, it will be closed in 5 days.