Test performance with parallel jobs

[nbokulich]

Improvement Description
If extra jobs are being called, setting n_jobs separately for the estimator and CV could be the culprit.

References
forum xref

[jakereps]

Anecdotally, sklearn's n_jobs parameter almost never listens to what it is given. I've even put n_jobs=1 and it still used dozens of CPUs.

[nbokulich]

interesting, thank you for that information @jakereps . So the reported issue might be rooted in sklearn (or even one of its dependencies)

[nick-youngblut]

This is a big issue for running qiime sample-classifier classify-samples on a cluster. I've been trying to run a classify job on a SGE cluster with 24 threads. However, the qsub always dies because qiime sample-classifier classify-samples keeps trying to use more threads than 24. It seems to be trying to use all CPUs on the particular node that it's running on, regardless of how many threads I designate with --p-n-jobs.

So, due to this issue, I really can't run qiime sample-classifier classify-samples on my cluster, which is especially a big issue when running a large set of qiime2 commands in a pipeline running on a cluster (the pipeline always dies during classification).

[nbokulich]

so far I have been unable to replicate this behavior; all tests have resulted in the expected number of jobs being called. I have not had any issues with multithreading on the SGE cluster I use. Based on @jakereps's anecdote above, I wonder if this is an issue with sklearn.

regardless of how many threads I designate with --p-n-jobs.

what about n_jobs=1?

[nbokulich]

Here's an interesting test @nick-youngblut : what happens if you use the parameters --p-no-parameter-tuning and/or --p-no-optimize-feature-selection (these are disabled by default but I assume you may have enabled these in your jobs). Would you mind testing and letting me know? (I've tested this but again can't replicate this issue on my system)