NLDAS grb forcing on Discover is very slow

[slabhan1]

Hi all,

I'm trying to spin up a subset of NLDAS domain using NLDAS2 grb files on Discover. The output for a 12 hour runtime is only about 60 days (whereas I'm looking for a 20-30 year spin up).
I requested many nodes and changed configurations of processors on x and y, but the runs only end up producing 60-80days per 12 hour runtime. It appears that I'm spending too much time on I/O for the NLDAS2 grib forcing files from the LIS log and job summary.
Do you have any ideas to get around this?

Thanks in advance!
Sakshi

[dmocko]

Hi @slabhan1

Yes, I do have some ideas. Since you are running on Discover, it might be best for you to point me to the file locations of your lis.config file, your LISF/lis/make/configure.lis file, and your SLURM job script. Please respond here with the paths/filenames, after ensuring that "other" will have read/access permissions to see the files.

[slabhan1]

Hi David,
Thanks so much. Please find the addresses below.

lis config: /discover/nobackup/slabhane/wd/nu_wrf_test/exp1/LISexp/dveg4/lis_config_rst
LIS make/configure: /discover/nobackup/slabhane/wd/nu_wrf_test/nu-wrf-dev/LISF/lis/make/configure.lis
slurm log: /discover/nobackup/slabhane/wd/nu_wrf_test/exp1/LISexp/dveg4/lisjob.out

[dmocko]

Hi @slabhan1

I don't have permissions to get into your "nu_wrf_test" directory. Can you please open, and ensure the sub-directories are open as well for me to get to those files?

[slabhan1]

Hi David, I hope I've opened up all the relevant directories now. Thank you for your patience w me!

…

________________________________ From: David Mocko ***@***.***> Sent: Wednesday, July 16, 2025 3:35 PM To: NASA-LIS/LISF ***@***.***> Cc: Sakshi Labhane ***@***.***>; Mention ***@***.***> Subject: Re: [NASA-LIS/LISF] NLDAS grb forcing on Discover is very slow (Discussion #1720) External Email - Use Caution Hi @slabhan1<https://github.com/slabhan1> I don't have permissions to get into your "nu_wrf_test" directory. Can you please open, and ensure the sub-directories are open as well for me to get to those files? — Reply to this email directly, view it on GitHub<#1720 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AXQLRRBGCU4MVGQXFLHC73L3I2SPZAVCNFSM6AAAAACBTC7GY6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGNZYGE3DKOA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[dmocko]

Still not quite yet. I need execute permissions for directories, and read permissions for files.

/discover/nobackup/slabhane/wd> ll -d nu_wrf_test/
drwxr--r-- 4 slabhane s2915 4096 Jul  9 16:44 nu_wrf_test//
/discover/nobackup/slabhane/wd> chmod go+rx nu_wrf_test

And so on for all sub-directories until getting to where the files are.

[dmocko]

I still can't get into this sub-directory. If you can't figure out how to open permissions, perhaps you can attach the files.

Please attach them individually as text files (not in a zip file).

configure.lis, lisjob.out, lis_config_rst, and lislog.0000 please.

Also, you can get some speed-up by changing the deflate level during the configure step.

Try going into LISF/lis/make, run "gmake realclean", then run "configure" again, choosing this option as "1":

NETCDF use deflate level? (1 to 9-yes, 0-no, default = 9): 1

[slabhan1]

Hi! Thanks for the tip! I've changed the permissions now-- sorry about the delay!

[dmocko]

Hi @slabhan1

I can read your files now. If you are using "9" for the deflate level, then I definitely recommend the above steps,
and re-configuring choosing "1" for this option, and then re-compiling.

I see the you are writing output daily. You mentioned that you are doing a "spin-up". Do you need the output this
frequently for the spin-up? We often only write output one time per month "1mo" when doing a spin-up, to see the
progress of the state variables. Reducing the output interval and the number of variables being written out will help
to speed up your spin-up run.

I also notice that you are only using 32 tasks. You should use at least the 126 CPUs on each node, but perhaps for
your domain you can use 252 CPUs (aka, 2 Milan nodes). Please try that and see if the run speeds up at all. You'll
need to change the job scripts as well as these two lines in your lis.config:

Number of processors along x:
Number of processors along y:

The product of these two values should equal 252, using the above example.