[WIP] Task Processing Optimization #7332
Conversation
tubignat
requested review from
3vilhamster,
Groxx,
Shaddoll,
davidporter-id-au,
demirkayaender,
dkrotx,
jakobht,
neil-xie,
sankari165,
shijiesheng and
taylanisikdemir
as code owners
| DefaultValue: false, | ||
| }, | ||
| TimerProcessorInMemoryQueue: { | ||
| KeyName: "history.timerProcessorInMemoryQueue", |
There was a problem hiding this comment.
I prefer to rename it history.enableTimerProcessorInMemoryQueue to make it a boolean explicitly.
There was a problem hiding this comment.
We're using pending task tracker to track all the pending tasks in-memory and uses this component to get the ack level of history queues. The in-memory tasks introduced in this PR are NOT tracked by that component, so it's possible that the ack level becomes larger than a pending task.
There was a problem hiding this comment.
And RemoveScheduledTasksAfter should be implemented in a way that invokes Cancel method of task interface on tasks in pending task tracker.
| } | ||
| } | ||
|
|
||
| func (r *reschedulerImpl) RemoveScheduledTasksAfter(t time.Time) { |
There was a problem hiding this comment.
This method is not necessary. Task has Cancel method. If it's canceled, it will be ignored by rescheduler.
| ts := info.Tasks[i].GetVisibilityTimestamp() | ||
| if ts.Before(nextTime) { | ||
| nextTime = ts | ||
| readLevel := q.base.newVirtualSliceState.Range.InclusiveMinTaskKey.GetScheduledTime() |
There was a problem hiding this comment.
This is the current max read level. The current read level is in virtual slice component. If virtual queue is still trying to read tasks from the database, the task can be submitted twice.
There was a problem hiding this comment.
These changes don't seem to be necessary.
2 participants
What changed?
Timer task queue now keeps incoming tasks in an in-memory queue until the next execution iteration
Why?
To reduce load on DB by avoiding agressively polling it for next timer tasks
How did you test it?
[WIP]
Potential risks
[WIP]