No indication of walltime/core queue limit

[minghangli-uni]

Submitting a large job via payu run fails with subprocess.CalledProcessError: ... returned non-zero exit status 32 with no explanatory message. This sometimes makes it a bit hard to diagnose when the request exceeds queue limits (walltime vs ncpus thresholds on Gadi).

qsub -q normalsr -P tm70 -l walltime=36000 -l ncpus=4472 -l mem=21500GB -l jobfs=10GB -N panan_4km_prepr -l wd -j n -v PAYU_PATH=/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20251015T221553Z-462fc45/bin,PAYU_FORCE=True,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/vk83/prerelease/modules:/etc/scl/modulefiles:/etc/scl/modulefiles:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -l storage=gdata/tm70+gdata/vk83 -- /g/data/vk83/prerelease/./apps/conda_scripts/payu-dev-20251015T221553Z-462fc45.d/bin/launcher.sh /g/data/vk83/prerelease/./apps/base_conda/envs/payu-dev-20251015T221553Z-462fc45/bin/python /g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20251015T221553Z-462fc45/bin/payu-run
Traceback (most recent call last):
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20251015T221553Z-462fc45/bin/payu", line 7, in <module>
    sys.exit(parse())
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20251015T221553Z-462fc45/lib/python3.10/site-packages/payu/cli.py", line 49, in parse
    run_cmd(**args)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20251015T221553Z-462fc45/lib/python3.10/site-packages/payu/subcommands/run_cmd.py", line 116, in runcmd
    job_id = cli.submit_job('payu-run', pbs_config, pbs_vars)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20251015T221553Z-462fc45/lib/python3.10/site-packages/payu/cli.py", line 173, in submit_job
    result = subprocess.run(shlex.split(cmd), capture_output=True, check=True)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20251015T221553Z-462fc45/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['qsub', '-q', 'normalsr', '-P', 'tm70', '-l', 'walltime=36000', '-l', 'ncpus=4472', '-l', 'mem=21500GB', '-l', 'jobfs=10GB', '-N', 'panan_4km_prepr', '-l', 'wd', '-j', 'n', '-v', 'PAYU_PATH=/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20251015T221553Z-462fc45/bin,PAYU_FORCE=True,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/vk83/prerelease/modules:/etc/scl/modulefiles:/etc/scl/modulefiles:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles', '-l', 'storage=gdata/tm70+gdata/vk83', '--', '/g/data/vk83/prerelease/./apps/conda_scripts/payu-dev-20251015T221553Z-462fc45.d/bin/launcher.sh', '/g/data/vk83/prerelease/./apps/base_conda/envs/payu-dev-20251015T221553Z-462fc45/bin/python', '/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20251015T221553Z-462fc45/bin/payu-run']' returned non-zero exit status 32.

[aidanheerdegen]

Good point. Do you have a proposed solution?