Parallel Processing with Pool

[sencer4898]

Hi. I am trying to run Neural Prophet model for different, similar entities. It is parallelized, however model progress is too slow/close to none. This is the part where I do the parallelizing:

model_array is the array that keeps models for each entity. Here is the part where the progress gets stuck:

It progresses at times (discrete time progress) but it is 10x slower than the serial processing, therefore the problem. Also, it uses all of the cores of local computer (88 cores), even though I specify it to be 4. What could be the problem?

[ourownstory]

Hi @sencer4898
We have also observed some odd behavior depending which type of multiprocessing function we used. (In particular, combining ThreadPools and regular Pools lead to odd behavior.)
The one setup that worked reliably on all our machines thus far was to use map_async
You can see two examples here:

neural_prophet/neuralprophet/benchmark.py

Line 435 in da31717

with Pool(self.num_processes) as pool:

and here:

neural_prophet/neuralprophet/benchmark.py

Line 516 in da31717

pool.map_async(self._run_exp, args_list, callback=self._log_result)

Please let me know if this approach works for you and share your updated code snippet.
Thank you,
Oskar

[sencer4898]

Hi @ourownstory . Thanks for the quick reply, much appreciated.

I tried to run the code the way you suggested. However, the problem still continues. Its been 10 minutes and the script is still at this stage shown below, with no progress at all.

These are the main and callback functions:

This is the array that I keep all the models (different model for each entity):

Also, on a side note, I am working on the local server of my company. This is the only configuration attribute that I set up:
os.environ['NUMEXPR_MAX_THREADS'] = '40'

I can give additional information if you need any. Thanks a lot.

Mehmet

[sencer4898]

I noticed that I return a variable called 'i', which doesn't exist, in the function. However, it should not be an issue because I couldn't complete the model fitting stage anyways.

[ourownstory]

Hi Mehmet I am not sure why the same method is not working for you. I wish I understood mp better, to help you better.
As of now, my best guess is that in my use of mp, I instantiate the model in the parallelized function, while you instantiate it before.
I suggest creating a minimal setup to re-create the problem with the least amount of confounding factors to better identify the root cause.

Please do let us know if you find a solution. I am sure other people will appreciate it!