Open
Description
Jobs sometimes executed multiple times.
Issue
I have found a concurrency safety issue.
A dispatched job will be dequeued twice with same jobID and payload.
Reproducing
This is a reproducing repository: https://github.com/sidepelican/QueuesFluentDriverMultipleExecution
This repository dispatches a simple job several times, and automatically detects when the same job launched multiple times.
$ swift run
...
2022-12-19T19:08:24+0900 info main : job_id=940F558A-EC00-47B9-8935-45D884281708 p_id=51675C99-B581-4555-9072-A376D1E95770 [App] EchoJob!
2022-12-19T19:08:24+0900 info main : job_id=940F558A-EC00-47B9-8935-45D884281708 p_id=51675C99-B581-4555-9072-A376D1E95770 [App] EchoJob!
p_id=51675C99-B581-4555-9072-A376D1E95770 is multiple executed!
Cause
- Queues calls
Queue.set
andQueue.push
inQueue.dispatch
.
FluentQueue.set
save aJobModel
.JobModel.state
has.pending
as initial state.
FluentQueue.push
writes the job's state topending
. The default value ofstate
is.pending
, so this operation is seemingly meaningless.
- The jobs set in 2 is ready for the workers to dequeue. What happens if a worker dequeues a job set in 2 between 2 and 3? The worker set the
state
to.processing
and then it is overridden to.pending
in 3. - The state is
.pending
so another worker can dequeue the job. Incident happens.
How to fix?
I think there are two ways.
One is to add .initialized
to QueuesFluentJobState
and use it as an initial value of JobModel.state
.
The other is to do nothing in FluentQueue.push
.