Hi all,
I realized the multi-threading performance is good, but probably can be even better for local machines. I noticed that my CPU cores are not engaged fully so a simple solution that I usually use and found to be helpful here is joblib parallel processing.
At its core, all it needs is something like this:
results = Parallel(n_jobs=-1)(
delayed(network_analysis.analyse_single_target)(
settings=settings, data=data,target=node) for node in range(n_nodes))But of course, it can be more user-friendly if this is wrapped in a function, something like an interface where we just tell how many jobs, what to do, some kwargs for the function and potentially some kwargs for the parallel processing backend. This way, each core is occupied with one single_target analysis, so as in my case, it can help a lot with the performance.
About joblib, we used it in our own library and compared to some fancier things like dask and ray it actually is a lot better and less pain! So far, it never broke anything for us.
Hi all,
I realized the multi-threading performance is good, but probably can be even better for local machines. I noticed that my CPU cores are not engaged fully so a simple solution that I usually use and found to be helpful here is
joblibparallel processing.At its core, all it needs is something like this:
But of course, it can be more user-friendly if this is wrapped in a function, something like an interface where we just tell how many jobs, what to do, some
kwargsfor the function and potentially somekwargsfor the parallel processing backend. This way, each core is occupied with onesingle_targetanalysis, so as in my case, it can help a lot with the performance.About
joblib, we used it in our own library and compared to some fancier things likedaskandrayit actually is a lot better and less pain! So far, it never broke anything for us.