Issue with NoahMP-4.0.1 Parallel Computing in GRACE TWS DA runs: Vertical Strips in Output

[ThangjamSatrajitSingh]

Hi,
I am Satrajit from the Hydrosense Group at IIT Delhi. We are running GRACE TWS data assimilation (DA) using the NoahMP-4.0.1 model and have encountered an issue with parallel run.

Details:

• In open-loop runs, using multiple processors works without any issues.
• However, during GRACE TWS DA runs with parallel computing in ensemble smoother runmode, the output shows abnormal vertical strips of single pixels.
• The number of vertical strip segments matches the number of processors used along the x-dimension.
• For example: With a 6x6 processor grid (36 processors total), the output is divided into 6 parts, each with vertical single-pixel strips. And with 4x4 processor grid, the output is divided into 4 parts as shown below:

• Also, the issue disappears with a 1x36 processor grid.
• The strips start appearing from April 2002, which coincides with the availability of the first GRACE satellite data.

So, most probably the problem is with how the model handles parallelization across processors, particularly during GRACE TWS DA in the ensemble smoother runmode.

Has anyone else encountered this issue with parallel runs in NoahMP-4.0.1 during GRACE TWS DA runs before?

PS: The LIS executable is compiled from the master version with optimization level = 3.

Thank you!

[dmocko]

Hi @ThangjamSatrajitSingh

Thank you for introducing yourself and for your question.

What version of the LIS code are you using?

Can you please share/attach your LISF/lis/make/configure.lis file as well as your lis.config file?

[ThangjamSatrajitSingh]

Hi @dmocko
Thank you for your response.
I am using the LIS master version. Below is the lis.config file as requested:
configure.lis.zip
Please let me know if you need any additional details.

Best regards,
Thangjam Satrajit Singh

[ThangjamSatrajitSingh]

Hi
Just a quick update. The issue is resolved if we use a processor layout of 1 processor along x-axis (used 1x36 instead of previous 6x6 for the output below).

Please let me know if there is anything I can do on my end.

Thanks!

[msaharia]

Hi, @dmocko - as you can see above, the stripes are fixed if we run it for a 1x36 processor grid. Do you have any suggestions how to fix this and what is causing this?

Much appreciated!

• Manab

[dmocko]

Hi @msaharia and @ThangjamSatrajitSingh

Thanks for your patience. I don't know why this is happening for your run. Have you updated your master branch?

What does git branch -vv show for your code?

Perhaps my colleagues @KathyNie or @jvgeiger are familiar with this problem and can provide some assistance.

[ThangjamSatrajitSingh]

Hi @dmocko ,

Here’s the output from git branch -vv on my end:

master 741ecd7 [origin/master] Merge pull request #1613 from jvgeiger/fix/gefs_testcase

As i have checked, there is a newer version out:

master 20ac11f [origin/master] Merge pull request #1676 from emkemp/feature/lisf77_pathlength

I will get back on the runs with the newer version.

Thanks!

[ThangjamSatrajitSingh]

Hi @dmocko
Even after updating to the latest master branch, I’m still encountering the same issue, as shown below:

The current git branch used for this run is:

 master 20ac11f [origin/master] Merge pull request #1676 from emkemp/feature/lisf77_pathlength

[ThangjamSatrajitSingh]

In addition to this, I’m facing a couple of other issues that may or may not be related:
1. Data Assimilation Run Not Completing
When running the data assimilation setup, the model consistently stops after 12–14 months, even though the job is configured for 160 hours. The actual runtime tends to be around 30 hours before it exits.
I’ve already allocated an additional 180 GB of memory, so I don’t believe it’s hitting any obvious time or memory limits. The standard error file shows the following:

forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
LIS_-2             000000000919ADC4  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4D6CECE630  Unknown               Unknown  Unknown
LIS_-2             000000000203DA7B  noahmp401_main_          1192  NoahMP401_main.F90
LIS_-2             000000000163252A  lsmrun_                   476  LIS_lsm_FTable.c
LIS_-2             00000000015EB616  lis_lsmmod_mp_lis         619  LIS_lsmMod.F90
LIS_-2             0000000001923764  lis_surfacemodelm         248  LIS_surfaceModelMod.F90
LIS_-2             00000000088205E4  smootherda_runmod         148  smootherDA_runMod.F90
LIS_-2             00000000018A2E08  lisrun_                   202  LIS_runmode_FTable.c
LIS_-2             0000000004D4B7DC  MAIN__                     70  lisdrv.F90
LIS_-2             0000000000416672  Unknown               Unknown  Unknown
libc-2.17.so       00002B4D6D3FF555  __libc_start_main     Unknown  Unknown
LIS_-2             0000000000416569  Unknown               Unknown  Unknown

I’m still unsure of what is causing the model to terminate like this.

2. Repeated Glacier Melt Warnings in Log Files
I’m also seeing the following warning repeatedly in the log files:

Noah-MP.4.0.1 (wrf_debug): 
  GLACIER HAS MELTED AT:           1           1  ARE YOU SURE THIS SHOULD BE A 
 GLACIER POINT?

From what I can tell, this is referring to the top-left grid cell. I'm unsure if this is something to be concerned about — apart from increasing the size of the log files, I haven't noticed any major issues in the output so far.

I’d really appreciate your help in figuring out what might be going wrong.

Thanks,
Thangjam Satrajit Singh

[dmocko]

Hi @ThangjamSatrajitSingh

I'm really not sure why the job would be stopping there. It appears to be at this line in the code:
https://github.com/NASA-LIS/LISF/blob/master/lis/surfacemodels/land/noahmp.4.0.1/NoahMP401_main.F90#L1192

Are you outputting the "RSSUN" variable (in the MODEL OUTPUT TBL as "RsSunlit"?

If so, do you need this variable and can you try to run with it off? But I suspect the code might just stop elsewhere.

Please look at the end of ALL of your lislog files (not just lislog.0000), as there might be more information there.

---

Regarding the glacier melt warning - it's not the top left grid cell, or cell (1,1), despite the mis-leading error message.

This is a Noah-MP warning (not a LIS warning), but it may be safely ignored. We see this often, especially along coastlines.
What happens is that glacier will be the dominant land type (thus the grid point is glacier), but most of the sub-grid fraction
of different types is not-glacier (e.g., glacier=25%, water=20%, forest=20%, grassland=20%, bare soil=15%).

[ThangjamSatrajitSingh]

Hi @dmocko
Thanks for the quick response.

1. The strip issue is fixed after making the suggested changes from @KathyNie
2. I am not outputting the RSSUN variable in the MODEL OUTPUT TBL. For reference, I’ve attached my MODEL_OUTPUT_TBL file.
   MODEL_OUTPUT_LIST.zip
3. I’ve checked all of the lislog files of my latest run, and the logs end in the same way across all processors:

[INFO] Reading GRACE data ./OUTPUT/GRACE_obs_202010.bin
[INFO] read GRACE data
[INFO] Assimilating Observations using GMAO EnKS

4. Regarding the glacier melt warning — thank you for the clarification! I’ll safely ignore it going forward.

About the model run ending prematurely, the issue is still there. Please let me know if you spot anything unusual in the attached files or have any suggestions.

Best regards,
@ThangjamSatrajitSingh

[KathyNie]

Hi @ThangjamSatrajitSingh

Regarding the simulation, if the error is not from the model output table side, I was wondering if you could do the following tests to trace down the potential bug:

1. take a look at the GRACE_obs_202010.bin file and see if there is any abnormally large values in the file. This might be a potential issue as sometimes LDT may generate large unrealistic values for coastal grid cells in the GRACE obs file, and if so, consider truncate those into a reasonable value and see if that help pass the crashing point.

2. Another suggestion is to restart your simulation at Nov 1st 2020, just to see if the model crash only occurs due to the assimilation of Oct 2020 values.

[ThangjamSatrajitSingh]

Hi @KathyNie

Thanks for the suggestions!

1. Regarding the GRACE_obs_202010.bin file – since it's a binary file, I just wanted to clarify if you have a recommended way to interpret or visualize it to check for any abnormally large values? I assume it would need to be read with a specific tool or script, so let me know if there's one you'd suggest for this.

2. As for the restart idea – yes, I’ve been restarting the simulation the way you suggested. When I restart from the last ending month, the model runs successfully for about a year and a few months, but then it always stops right after generating the first day of the last generated month. For example, in the most recent case, it generated Nov 1st, then stopped. So restarting does help temporarily. I’ll try running it from Oct 1st next, to see if the crash is specifically related to the assimilation of Oct 2020 values and update you with any findings.

Best regards,
@ThangjamSatrajitSingh

[KathyNie]

Hi @ThangjamSatrajitSingh

Regarding the strip issue, we've seen such behavior for some configurations with GRACE-DA. Here is a quick fix. You may modify the following part in subroutine llij_latlon in lis/interp/map_utils.F90:

Instead of:

IF (deltalon .LT. 0) deltalon = deltalon + 360.

Please change it to:

IF (deltalon .LT. -0.001) deltalon = deltalon + 360.
IF (deltalon .LT. 0) deltalon = Abs(deltalon)

[ThangjamSatrajitSingh]

Hi @KathyNie

Thanks a lot for the fix!

I made the suggested changes and it worked perfectly — the striping issue is no longer there.

Really appreciate your help with this!

Best regards,
@ThangjamSatrajitSingh