MultiRegion with NetCDF output

[mclaret]

Hi, I ran the HydrostaticFreeSurfaceModel using one GPU successfully saving model output with the following command:

outputs=(; u,v,w,b)
simulation.output_writers[:fields] = NetCDFOutputWriter(model, outputs; filename,
schedule = IterationInterval(48), #dt=300
overwrite_existing = true)

I've recently switched to using 2 GPUs with MultiRegion with the help of @francispoulin and I am getting the following error:

ERROR: LoadError: MethodError: no method matching construct_output(::Field{…}, ::Tuple{…})

I've looked at the test cases with MultiRegion but couldn't find any output using NetCDF. Any guidance would be much appreciated.

Thanks, Mariona

[glwagner]

Hi @mclaret @francispoulin ! MultiRegion isn't supported right now, which is why there are no examples! However, we've put a lot of effort into making Distributed architectures work. To use output with Distributed(GPU()), you have to append the name of the output writer with the arch.local_rank so that different processes don't overwrite each other's files. Let us know if you can figure this out or if we can help more.

cc @simone-silvestri

[francispoulin]

Thanks @glwagner for pointing this out. I was the one that had suggested we use MultiRegion but happy to use Distributed instead. I see there are some examples in the validation folder. I'll give them a try and we can go from there.

[glwagner]

This might be what you want: https://github.com/CliMA/Oceananigans.jl/blob/main/validation/distributed_simulations/distributed_nonhydrostatic_turbulence.jl

Curious what you find.

@jagoosw and @simone-silvestri have used it

[francispoulin]

Thanks @glwagner . I tried running this script (and all the other distributed examples) and none of them run successfully for me. The one that gets closest is the hydrostatic one. I created an issue @211, to get that fixed.

Next on the list I will look at the nonhydrostatic example, which has a very different error.

[glwagner]

I think the hydrostatic script is likely just a physics problem because the problem is weird. Maybe test it in serial first to confirm that it makes sense.