Performance degradation in develop branch: ~2x computational cost compared to v6.07

[leahcicon]

@aliabdolali @aronroland

I'm experiencing significant performance degradation after updating from WW3 v6.07 to the latest develop branch.

Issue Summary

Running regional wave models with unstructured grids using the implicit solving scheme now requires approximately double the computational nodes to achieve similar runtime as v6.07. The physic and model set up are unchanged between versions (from v6.07 to latest develop) except I am using the new feature ICNUMERICS = T. But I did a test with the latest version with ICNUMERICS = F and saw similar runtimes so unlikely to be the culprit.

Questions

1. Are there known performance regressions from v6.07?
2. Are there additional optimization flags or settings I should configure for the new version?
3. Has anyone else experienced similar performance issues with implicit solver with unstructured grids?

Any guidance on troubleshooting or optimizing performance would be greatly appreciated.

[aronroland]

If u want to share both setups i can take a look ...

[leahcicon]

That would be great, thank you.

ww3_grid.nwaunstr_IC4.txt
The ww3.grid file is the same for both setups minus the ICNUMERICS = T.

Switches v6.07: F90 NOGRB NC4 DIST MPI PR3 UQ FLX0 LN1 ST4 STAB0 NL1 BT1 DB1 MLIM TR1 BS0 IC4 IS0 REF1 XX0 WNT1 WNX1 CRT1 CRX1 PDLIB ECD O1 O2 O7
Switches latest branch: NOGRB DIST MPI PDLIB METIS PR3 UQ FLX0 LN1 ST4 STAB0 NL1 BT1 DB1 MLIM TR1 BS0 IC4 IS0 REF1 WNT1 WNX1 CRT1 CRX1 O1 O2 O7
Switches v6.07: F90 NOGRB NC4 DIST MPI PR3 UQ FLX0 LN1 ST4 STAB0 NL1 BT1 DB1 MLIM TR1 BS0 IC4 IS0 REF1 XX0 WNT1 WNX1 CRT1 CRX1 PDLIB ECD O1 O2 O7