SUBBASINFLAG with vector-based domain, possibly with `pad_outlets`

[dprincz]

From @mee067 (commented on #10, closed)

I ran successfully with the SUBBASINFLAG before but this time, I am getting an issue from the start. It counted 512 land tiles and is failing on tile 513!!

0BAD ITERATION TEMPERATURE 513 59 9.00 12 0
0.0000 0.0000 NaN NaN -19007.4980 NaN NaN
0******** END TSOLVE ************************************************************************ -1

I am running from scratch (i.e. RESUMEFLAG 0).

Is that related to the basin outlet padding? The error is the same for 1860_ME_ZT and MESH_1.5.5

[dprincz]

I'm not sure it's related to pad_outlets if you observed the same error with 1860_ME_ZT. Is there a way you can share the setup or one that replicates this issue so I can debug what's happening?

[mee067]

I will share it with you by email; I will have to extract the forcing for day 1 to reduce size.

[dprincz]

The error can be reproduced with the sample with MPI using 32 cores. It doesn't trigger with 16 cores, so the problem likely comes from when it nudges the indexing (to make sure all tiles of a single grid reside on the same core/node).

With debug symbols, the index-out-of-bounds happens right away during initialization:

forrtl: severe (408): fort: (2): Subscript #1 of the array ILMOS has value 513 which is greater than the upper bound of 512

Image              PC                Routine            Line        Source
mpi_sa_mesh.mpifo  000000000040DB74  mpi_utilities_mp_          46  mpi_utilities.f90
mpi_sa_mesh.mpifo  000000000131341D  read_initial_inpu         753  read_initial_inputs.f90
mpi_sa_mesh.mpifo  00000000016514D1  MAIN__                    244  MESH_driver.f90
mpi_sa_mesh.mpifo  000000000040D70D  Unknown               Unknown  Unknown
libc.so.6          00007F0EBBA4E94A  Unknown               Unknown  Unknown
libc.so.6          00007F0EBBA4EA05  __libc_start_main     Unknown  Unknown
mpi_sa_mesh.mpifo  000000000040D621  Unknown               Unknown  Unknown

The setup has 85 cells active with SUBBASINFLAG enabled. For this setup specifically, it might make sense to use a lesser number of cores. In the longer-term, the nudging in mpi_utilities should be checked if it can be adapted to avoid the error.