Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/cloud/metrics/openmetrics/metrics-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -518,6 +518,14 @@ These metrics could have high cardinality depending on number of task queues pre

The approximate number of tasks pending in a task queue. Started Activities are not included in the count as they have been dequeued from the task queue.

:::note Known accuracy limitations
This metric is approximate.
It can overcount because invalid or expired Tasks (for example, from cancelled, terminated, completed, or timed out Workflows) remain in the count until they reach the head of the queue and are processed and discarded.
It can also reset to zero on an idle Task Queue. If no Worker polls, no new Tasks are added, and no other Task Queue calls occur (such as `DescribeTaskQueue` or `UpdateTaskQueueConfig`) for approximately 5 minutes, the Task Queue is unloaded from memory.
Infrequent metadata updates and database time-to-live settings can also cause this metric to drift at a smaller magnitude.
See [backlog accuracy limitations](/develop/worker-performance#backlog-accuracy-limitations) for details.
:::

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
Expand Down
17 changes: 14 additions & 3 deletions docs/develop/worker-performance.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -720,9 +720,20 @@ The age is based on the creation time of the Task at the head of the queue.

You can rely on both these counts when making scaling decisions.

Please note: [Sticky queues](https://docs.temporal.io/sticky-execution) will affect these values, but only for a few seconds.
That's because Tasks sent to Sticky queues are not included in the returned values for `ApproximateBacklogCount` and `ApproximateBacklogAge`.
Inaccuracies diminish as the backlog grows.
#### Known accuracy limitations {#backlog-accuracy-limitations}

These values are approximate.
The most common sources:

- **Overcount from invalid or expired Tasks**: Tasks belonging to cancelled, terminated, completed, or timed out Workflows (and Activities) stay in the count until they reach the head of the queue and are processed and discarded.
An invalid or expired Task at the head is removed quickly, so it rarely holds up the count for long.
A valid Task at the head can stay there longer when there aren't enough Workers to dispatch it; while it sits there, invalid or expired Tasks queued behind it cannot be removed.
Invalid and expired Tasks are eventually accounted for, but the count may not return to a fully accurate value because of other sources of discrepancy (such as infrequent metadata updates and database row expirations).
- **Reset to zero on idle Task Queue unload**: If a Task Queue sees no activity for approximately 5 minutes - no Worker polls, no new Tasks added, and no other Task Queue calls (`DescribeTaskQueue`, `UpdateTaskQueueConfig`, etc.) - the Temporal Service unloads it from memory.
When this happens, `ApproximateBacklogCount` reports zero until the Task Queue is reloaded by the next Worker poll, new Task, or Task Queue API call.
An idle Task Queue with a backlog but no active Workers can therefore temporarily report zero even though there are Tasks waiting to be processed.
- **Sticky queue exclusion**: [Sticky queues](/sticky-execution) are not included in these values.
Because Sticky queue Tasks only remain valid for a few seconds, this inaccuracy diminishes as the backlog grows.

### `TasksAddRate` and `TasksDispatchRate` {#TasksAddRate-and-TasksDispatchRate}

Expand Down
Loading