ROMS package compilation options

[helenmacdonald]

This issue is about the recently added ancoms-roms package

ROMS sets a lot of user defined options pre – compilation and I think the current compilation does not have the best options activated.

The user defined options are set in a .h file (see here for examples) and usually each ROMS configuration has its own executable. This is also true for each of the ROMS test cases.

My spack understanding is still rudimentary but I believe that in package.h we have set the default executable to be for the benchmark testcase, whilst we are setting up the example configuration for an upwelling testcase. The upwelling case is the better example to use for an initial first test case (community convention is to start on this one). The upwelling test case is listed as a spack package variant so I am hoping (but please correct me) that an initial short-term fix may be as easy as changing default=’benchmark’ to default=’upwelling’.

For access-om3 I have access to several variants of the executable to get a mom6 standalone or a mom6+CICE on prerelease:

$ module use /g/data/vk83/prerelease/modules
$ module load access-om3/pr75-1
$ which access-om3-MOM6
/g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/access-om3-nuopc-git.0.4.1_0.4.1-mdludj4azhygq3p2mrfytygz5lgzintj/bin/access-om3-MOM6
$ which access-om3-MOM6-CICE6
/g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/access-om3-nuopc-git.0.4.1_0.4.1-mdludj4azhygq3p2mrfytygz5lgzintj/bin/access-om3-MOM6-CICE6

(pr75-1 is here)

Long-term, is there any way we can do something like this with ROMS? I believe that we could cover most Australian users with a finite number of executables (3-6 ish). Initially (within the next 6 months) we probably only need to have an upwelling test case and @phellipecouto's EAC variant.

Note that the compiled roms executable (romsM) is named the same regardless of which application it was compiled for (@aidanheerdegen mentioned that this may mean it is a little harder to do multiple variants).

[aidanheerdegen]

The upwelling test case is listed as a spack package variant so I am hoping (but please correct me) that an initial short-term fix may be as easy as changing default=’benchmark’ to default=’upwelling’.

Yes, but rather than make the change in the spack package we can set it in the spec like so:

diff --git a/spack.yaml b/spack.yaml
index 428ab48..de44877 100644
--- a/spack.yaml
+++ b/spack.yaml
@@ -10,7 +10,8 @@ spack:
     # the first element of the require is only a version
     ancoms-roms:
       require:
-        - '@4.2'
+        - '@4.2' 
+        - roms_application='upwelling'
     openmpi:
       require:
         - '@4.1.4'

I'll give that a go and see how it works.

[aidanheerdegen]

Currently the package supports only two "applications": benchmarkand upwelling

https://github.com/ACCESS-NRI/spack-packages/blob/main/packages/ancoms-roms/package.py#L35-L41

There are of order 30 applications in the code

https://github.com/ACCESS-Community-Hub/roms/tree/develop/ROMS/Include

we may need only 3-6 applications, but if we're designing something it may as well be flexible enough to allow any application to be built.

Ideally we should

1. Allow any application to specified in the application variant
2. Allow multiple values for the application variant
3. Build all variants
4. Add some namespace that reflects the application variant name to be able to select the required variant executable

I think 1-3 we do already for ACCESS3 (ACCESS-OM3)

https://github.com/ACCESS-NRI/spack-packages/blob/main/packages/access3/package.py

In that case the multiple application build is handled by Make. In this case I think we'd need to create some logic in the spack package to handle this.

As part of this we would introduce some namespace to separate out the executables. A couple of obvious options:

1. Add application to the install directory tree
2. Add application to the executable file name

Option 1

End up with paths like:

/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.8.0/ancoms-roms-4.2-qshh2xkc6q76twvvcwcmeqxlevg2gu6h/upwelling/bin/romsM
/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.8.0/ancoms-roms-4.2-qshh2xkc6q76twvvcwcmeqxlevg2gu6h/benchmark/bin/romsM

Pros: clean, executable name is the same as user are familiar with
Cons: using a module and payu it is difficult to select for the required romsM executable. Would be multiple executables with the same name in the $PATH

Option 2

End up with paths like:

/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.8.0/ancoms-roms-4.2-qshh2xkc6q76twvvcwcmeqxlevg2gu6h/bin/romsM-upwelling
/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.8.0/ancoms-roms-4.2-qshh2xkc6q76twvvcwcmeqxlevg2gu6h/romsM-benchmark

Pros: using a module and payu it is easy to select for the required romsM executable as they are uniquely named
Cons: executable name is not the same as user are familiar with

From these two Option 2 seems the obvious choice when using module load to inject executable paths into the payu run environment.

Are there any other ways to do this? Could we have multiple specs in the deployment environment, each specifying a different variant? It would automatically separate them out, but there would not be an obvious correspondence between the spack build hash and variant.

[helenmacdonald]

Thanks @aidanheerdegen. I like option 2 as well - the executable name is still recognisable as romsM and it is common enough for ROMS users to change the name of the executable anyway so the con is not a big issue.