Skip to content

[BUG] Multiple concurrency rules cause significant task queue overhead #2866

@linardsblk

Description

@linardsblk

Describe the issue
When a workflow is configured with multiple concurrency rules, each task experiences significant queue time (1-10 seconds), even when running a single isolated workflow with no concurrency contention.

The overhead scales with the number of tasks in the workflow. A 4-task DAG workflow that should complete in ~15s takes ~60s due to queue overhead.

Environment

  • SDK: TypeScript v5.9.3
  • Engine: v0.77.30 Cloud (but also observed in self-hosted)

Expected behavior
Multiple concurrency rules should not add significant overhead when there's no actual concurrency contention. A single workflow running in isolation should have minimal queue time regardless of how many concurrency rules are configured.

Code to Reproduce, Logs, or Screenshots

  export const workflow = hatchet.workflow({
    name: "test-concurrency",
    concurrency: [
      {
        expression: "input.key1",
        maxRuns: 1,
        limitStrategy: ConcurrencyLimitStrategy.CANCEL_IN_PROGRESS,
      },
      {
        expression: "input.key2",
        maxRuns: 5,
        limitStrategy: ConcurrencyLimitStrategy.GROUP_ROUND_ROBIN,
      },
    ],
  })

// 4 sequential tasks: A → B → C → D

In this simple isolated run it already adds significant overhead. On real tasks on production, we have seen even more seconds.

Image

Additional context
Obviously this could be fixed by leaving just one concurrency strategy, but having it defined at workflow level instead of application level makes it much easier to work with. e.g. de-duplicating tasks with CANCEL_IN_PROGRESS while also preventing resource hogging by user with GROUP_ROUND_ROBIN.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions