Explore: Adjust the number of workers based on generation #22
Open
Description
This remain to be seen whether we'll actually need this, but would be worth exploring.
The assumption is that the first few generation will invalidate shared pages rather quickly, as such the peak memory usage at first might get quite high.
A possible solution to this would be to start with a lower number of workers to compensate for the lower CoW effectiveness, and then increase the number of workers of processes with each new generation.
This would mean pitchfork needs a bit of a warm up before achieving peak throughput.
Metadata
Assignees
Labels
No labels