Hello,
I am currently running a Python script (SCCA_hyper_param.py) which performs hyperparameter tuning using GridSearchCV from the cca_zoo package (version: 2.3.2). Even though I have set n_jobs=-1 to allow full parallelism, and I have allocated 8 CPU cores in my Slurm script, only one core and one thread are being utilized when I monitor the execution using htop.
My setup:
I’m using SCCA_PMD from the cca_zoo.linear module with GridSearchCV.
The job is submitted using a Slurm script with 8 cores allocated and 256GB memory.
I’ve confirmed that joblib is installed, and n_jobs=-1 is passed to GridSearchCV.
Despite this setup, I’m seeing only one CPU core in use.
Code snippet from the Python script (SCCA_hyper_param.py):
sparse_CCA = GridSearchCV(SCCA_PMD(latent_dimensions=1), param_grid=param_grid, cv=5, verbose=2, scoring=scorer_cov, n_jobs=-1).fit([cca_x, cca_y])
Slurm script:
`#!/bin/bash
#SBATCH -t 1:00:00
#SBATCH -N 1
#SBATCH -c 8
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=1
#SBATCH --chdir=/data/connectome/kimbo/project_comm
#SBATCH --nodelist=gpu-1
#SBATCH --output=logs/R-%j-%x.out
#SBATCH --error=logs/R-%j-%x.out
#SBATCH --job-name=cca_w1
#SBATCH --mem=256GB
source /etc/profile.d/miniconda-setup.sh
conda activate bct_cca_py39
python SCCA_hyper_param.py --domain w1 --dataset w1train --exp_tag 1835.27
`
Problem:
Even though I have specified n_jobs=-1 in GridSearchCV and allocated 8 CPU cores in my Slurm script, htop shows that only one thread is active and only one core is being utilized.
My question:
Why is GridSearchCV not utilizing all 8 cores, and how can I configure it to ensure all the CPU cores allocated by Slurm are used?
I would greatly appreciate any guidance on resolving this issue.
Thank you in advance for your help!
Best regards,
Bogyeom Kim
Hello,
I am currently running a Python script (SCCA_hyper_param.py) which performs hyperparameter tuning using GridSearchCV from the cca_zoo package (version: 2.3.2). Even though I have set n_jobs=-1 to allow full parallelism, and I have allocated 8 CPU cores in my Slurm script, only one core and one thread are being utilized when I monitor the execution using htop.
My setup:
I’m using SCCA_PMD from the cca_zoo.linear module with GridSearchCV.
The job is submitted using a Slurm script with 8 cores allocated and 256GB memory.
I’ve confirmed that joblib is installed, and n_jobs=-1 is passed to GridSearchCV.
Despite this setup, I’m seeing only one CPU core in use.
Code snippet from the Python script (SCCA_hyper_param.py):
sparse_CCA = GridSearchCV(SCCA_PMD(latent_dimensions=1), param_grid=param_grid, cv=5, verbose=2, scoring=scorer_cov, n_jobs=-1).fit([cca_x, cca_y])Slurm script:
`#!/bin/bash
#SBATCH -t 1:00:00
#SBATCH -N 1
#SBATCH -c 8
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=1
#SBATCH --chdir=/data/connectome/kimbo/project_comm
#SBATCH --nodelist=gpu-1
#SBATCH --output=logs/R-%j-%x.out
#SBATCH --error=logs/R-%j-%x.out
#SBATCH --job-name=cca_w1
#SBATCH --mem=256GB
source /etc/profile.d/miniconda-setup.sh
conda activate bct_cca_py39
python SCCA_hyper_param.py --domain w1 --dataset w1train --exp_tag 1835.27
`
Problem:
Even though I have specified n_jobs=-1 in GridSearchCV and allocated 8 CPU cores in my Slurm script, htop shows that only one thread is active and only one core is being utilized.
My question:
Why is GridSearchCV not utilizing all 8 cores, and how can I configure it to ensure all the CPU cores allocated by Slurm are used?
I would greatly appreciate any guidance on resolving this issue.
Thank you in advance for your help!
Best regards,
Bogyeom Kim