cpu-bind error

[areiser2-tamu]

I currently am following along with the quick start guide for E3SM and am currently using an account on Perlmutter. I am able to get the template script to run, build, and submit the job. However, when the job actually runs it errors immediately. I checked the log file in the run directory and I get the only error of "srun error: Invalid numeric value "--cpu-bind=threads" for --cpus-per-task." I have tried this with v3 and v2.1 and both get the exact same error. I am unsure of how to change this value. I asked a peer and they said it should be in my case.run script but I do not have that in my case_scripts directory. Any guidance would be greatly appreciated

[whannah1]

@areiser2-tamu Can you share more details about your run, such as which grid you are using and how many nodes?
I might helpful to just post your run script.

[areiser2-tamu]

Here is the output

Results in group mach_pes
NTASKS: ['CPL:1024', 'ATM:1024', 'LND:1024', 'ICE:1024', 'OCN:1024', 'ROF:1024', 'GLC:1', 'WAV:1', 'IAC:1', 'ESP:1']
NTHRDS: ['CPL:1', 'ATM:1', 'LND:1', 'ICE:1', 'OCN:1', 'ROF:1', 'GLC:1', 'WAV:1', 'IAC:1', 'ESP:1']

Results in group mach_pes_last
MAX_TASKS_PER_NODE: 256
MAX_MPITASKS_PER_NODE: 128

[whannah1]

ok, you're using 8 nodes, so that all looks ok.

I'm even more confused about your issue though, because the use of MAX_MPITASKS_PER_NODE: 128 should not have resulted in --cpu-bind=threads.

Can you show me the result of grep "srun" .case.run.sh from the case directory?

[areiser2-tamu]

This what it outputs

srun --label -n 1024 -N 8 -c --cpu_bind=threads -m plane=Python 3.8 is recommended to run CIME. You have 3.6. 128 /pscratch/sd/a/areiser2/E3SMv2/v2_test/build/e3sm.exe

[whannah1]

well the part about "Python 3.8 is recommended to run CIME. You have 3.6" is certainly part of the issue.

Are you loading the NERSC python module? That is recommended and should give you version 3.11.7 by default. Delete the current case and try creating another one from scratch after loading the python module.

I'm still puzzled about why you're getting --cpu_bind=threads instead of --cpu_bind=cores, but maybe changing the python version will fix that.

[areiser2-tamu]

I guess I have not been loading python. I will do that and let you know the results in the morning

[areiser2-tamu]

It does appear that not having up to date python was the issue. It is interesting that python version would change how the cpu-bind is set. Thank you for the assistance.

[whannah1]

Glad we solved it. I'm also perplexed at how the python version was so impactful here, but I've run into similar problems in the past, so I'm not too surprised.

[whannah1]

FYI case.run now exists as .case.run so it is hidden by a normal ls command. But either way, you shouldn't edit it or run it directly.