-
Notifications
You must be signed in to change notification settings - Fork 304
Description
Describe the issue
When a workflow is configured with multiple concurrency rules, each task experiences significant queue time (1-10 seconds), even when running a single isolated workflow with no concurrency contention.
The overhead scales with the number of tasks in the workflow. A 4-task DAG workflow that should complete in ~15s takes ~60s due to queue overhead.
Environment
- SDK: TypeScript v5.9.3
- Engine: v0.77.30 Cloud (but also observed in self-hosted)
Expected behavior
Multiple concurrency rules should not add significant overhead when there's no actual concurrency contention. A single workflow running in isolation should have minimal queue time regardless of how many concurrency rules are configured.
Code to Reproduce, Logs, or Screenshots
export const workflow = hatchet.workflow({
name: "test-concurrency",
concurrency: [
{
expression: "input.key1",
maxRuns: 1,
limitStrategy: ConcurrencyLimitStrategy.CANCEL_IN_PROGRESS,
},
{
expression: "input.key2",
maxRuns: 5,
limitStrategy: ConcurrencyLimitStrategy.GROUP_ROUND_ROBIN,
},
],
})// 4 sequential tasks: A → B → C → D
In this simple isolated run it already adds significant overhead. On real tasks on production, we have seen even more seconds.
Additional context
Obviously this could be fixed by leaving just one concurrency strategy, but having it defined at workflow level instead of application level makes it much easier to work with. e.g. de-duplicating tasks with CANCEL_IN_PROGRESS while also preventing resource hogging by user with GROUP_ROUND_ROBIN.