Commit edd021d
authored
Fix partitioned queue worker concurrency tracking (#1272)
## Summary
Fixes partitioned queue dispatch so workflows started from persisted
queue rows preserve their stored `queuePartitionKey` when they are
registered in the worker's in-memory running workflow map.
Without this, `workerConcurrency` can be ineffective for partitioned
queues: DBOS dequeues workflows for a specific partition, but then
starts them without carrying that partition key into
`registerRunningWorkflow`. The running workflow is tracked under
`queuePartitionKey = undefined`, while the next dequeue for the real
partition counts only running workflows with that partition key. As a
result, the worker can repeatedly think there are zero running workflows
for the partition and mark more workflows `PENDING` on every polling
tick.
## Context
We discovered this while running a single DBOS worker with a partitioned
queue configured with only local worker concurrency:
```ts
await DBOS.registerQueue('my-queue', {
partitionQueue: true,
workerConcurrency: 1000,
});
```
We enqueued a large batch of workflows using the same
`queuePartitionKey` and expected at most ~1000 workflows in `PENDING`
for that worker/partition. Instead, `PENDING` grew far above the worker
concurrency limit. The queue row was correct, the app version was
current, and there was only one worker process.
The issue appears to be that `executeWorkflowId` loads
`wfStatus.queuePartitionKey` from the database, but does not pass it
back into the internal workflow params. Later, `internalWorkflow`
registers the running workflow using
`params.enqueueOptions?.queuePartitionKey`, so the running workflow is
counted under `undefined` instead of the actual partition.
## Workaround
A workaround is to set global `concurrency` as well as
`workerConcurrency`:
```ts
await DBOS.registerQueue('my-queue', {
partitionQueue: true,
workerConcurrency: 1000,
concurrency: 1000,
});
```
That works because global `concurrency` is enforced from Postgres state
and the SQL filter includes `queue_partition_key`. However, it has
additional overhead: DBOS must count currently pending workflows in the
database, use stricter transaction behavior, and coordinate through the
database even when there is only one worker. We would prefer to avoid
that overhead when `workerConcurrency` alone is sufficient.
## Fix
This PR preserves `wfStatus.queuePartitionKey` when queue-dispatched or
recovered workflows are re-executed from stored workflow status. That
ensures `registerRunningWorkflow` records the correct partition key and
`countRunningWorkflowsForQueue(queue, partitionKey)` enforces
`workerConcurrency` per partition as intended.
Also adds a regression test for a partitioned queue with
`workerConcurrency: 1` and no global `concurrency`. The first workflow
blocks across multiple polling ticks, and the test verifies that no
second workflow in the same partition starts until capacity is
available.
## Tests
```bash
./node_modules/.bin/jest tests/wfqueue.test.ts --runInBand --testNamePattern="partitioned-queue-worker-concurrency"
npm run build
```1 parent c612cfe commit edd021d
2 files changed
Lines changed: 55 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1148 | 1148 | | |
1149 | 1149 | | |
1150 | 1150 | | |
| 1151 | + | |
| 1152 | + | |
1151 | 1153 | | |
1152 | 1154 | | |
1153 | 1155 | | |
| |||
1157 | 1159 | | |
1158 | 1160 | | |
1159 | 1161 | | |
| 1162 | + | |
1160 | 1163 | | |
1161 | 1164 | | |
1162 | 1165 | | |
| |||
1189 | 1192 | | |
1190 | 1193 | | |
1191 | 1194 | | |
| 1195 | + | |
1192 | 1196 | | |
1193 | 1197 | | |
1194 | 1198 | | |
| |||
1228 | 1232 | | |
1229 | 1233 | | |
1230 | 1234 | | |
| 1235 | + | |
1231 | 1236 | | |
1232 | 1237 | | |
1233 | 1238 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1565 | 1565 | | |
1566 | 1566 | | |
1567 | 1567 | | |
1568 | | - | |
| 1568 | + | |
1569 | 1569 | | |
1570 | 1570 | | |
1571 | 1571 | | |
| |||
1792 | 1792 | | |
1793 | 1793 | | |
1794 | 1794 | | |
| 1795 | + | |
| 1796 | + | |
| 1797 | + | |
| 1798 | + | |
1795 | 1799 | | |
1796 | 1800 | | |
1797 | 1801 | | |
| |||
1809 | 1813 | | |
1810 | 1814 | | |
1811 | 1815 | | |
| 1816 | + | |
| 1817 | + | |
| 1818 | + | |
| 1819 | + | |
| 1820 | + | |
| 1821 | + | |
| 1822 | + | |
| 1823 | + | |
| 1824 | + | |
| 1825 | + | |
| 1826 | + | |
| 1827 | + | |
| 1828 | + | |
1812 | 1829 | | |
1813 | 1830 | | |
1814 | 1831 | | |
| |||
1871 | 1888 | | |
1872 | 1889 | | |
1873 | 1890 | | |
| 1891 | + | |
| 1892 | + | |
| 1893 | + | |
| 1894 | + | |
| 1895 | + | |
| 1896 | + | |
| 1897 | + | |
| 1898 | + | |
| 1899 | + | |
| 1900 | + | |
| 1901 | + | |
| 1902 | + | |
| 1903 | + | |
| 1904 | + | |
| 1905 | + | |
| 1906 | + | |
| 1907 | + | |
| 1908 | + | |
| 1909 | + | |
| 1910 | + | |
| 1911 | + | |
| 1912 | + | |
| 1913 | + | |
| 1914 | + | |
| 1915 | + | |
| 1916 | + | |
| 1917 | + | |
| 1918 | + | |
| 1919 | + | |
| 1920 | + | |
| 1921 | + | |
| 1922 | + | |
1874 | 1923 | | |
1875 | 1924 | | |
1876 | 1925 | | |
| |||
0 commit comments