compilation flag -fPIC for oasis and fesom

[a270105]

Describe the problem you are facing
I can't compile awiesm-2.6 with recom using my esm-tools branch 'awiesm-2.6-recom'. The error occurs when linking shared libraries during fesom compilation, e.g.:

ld: /work/ab0995/a270105/esm_code/awiesm-2.6-recom/oasis/build/lib/psmile/libpsmile.a(mod_oasis_getput_interface.F90.o): relocation R_X86_64_32 against `__STRLITPACK_135' can not be used when making a shared object; recompile with -fPIC

To Reproduce

cd esm_tools
git checkout awiesm-2.6-recom
esm_master install-awiesm-2.6-recom

Dirty solution
'-fPIC' is included in oasis3mct.env.yaml. So I don't really understand why this needs to be added.
I tried to add the flag in CMake files. But it didn't work. At the end I added cmake option directly in comp-fesom... and comp-oasis...shell scripts and recompiled all model components by running shell scripts.

New problems
After successfully compiling the model, I got errors by running experiment:

Traceback (most recent call last):
  File "/home/a/a270105/.local/bin/esm_runscripts", line 33, in <module>
    sys.exit(load_entry_point('esm-tools', 'console_scripts', 'esm_runscripts')())
  File "/home/a/a270105/esm_tools/src/esm_runscripts/cli.py", line 289, in main
    setup()
  File "/home/a/a270105/esm_tools/src/esm_runscripts/sim_objects.py", line 139, in __call__
    self.prepcompute()
  File "/home/a/a270105/esm_tools/src/esm_runscripts/sim_objects.py", line 230, in prepcompute
    self.config = prepcompute.run_job(self.config)
  File "/home/a/a270105/esm_tools/src/esm_runscripts/prepcompute.py", line 26, in run_job
    config = evaluate(config, "prepcompute", "prepcompute_recipe")
  File "/home/a/a270105/esm_tools/src/esm_runscripts/helpers.py", line 71, in evaluate
    config = esm_plugin_manager.work_through_recipe(
  File "/home/a/a270105/esm_tools/src/esm_plugin_manager/esm_plugin_manager.py", line 159, in work_through_recipe
    config = getattr(submodule, workitem)(config)
  File "/home/a/a270105/esm_tools/src/esm_runscripts/filelists.py", line 1395, in report_missing_files
    config = _check_fesom_missing_files(config)
  File "/home/a/a270105/esm_tools/src/esm_runscripts/filelists.py", line 1429, in _check_fesom_missing_files
    if not os.path.exists(path):
  File "/sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/lib/python3.10/genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not Namelist
Traceback (most recent call last):
  File "/home/a/a270105/.local/bin/esm_runscripts", line 33, in <module>
    sys.exit(load_entry_point('esm-tools', 'console_scripts', 'esm_runscripts')())
  File "/home/a/a270105/esm_tools/src/esm_runscripts/cli.py", line 289, in main
    setup()
  File "/home/a/a270105/esm_tools/src/esm_runscripts/sim_objects.py", line 125, in __call__
    self.config = prepexp.run_job(self.config)
  File "/home/a/a270105/esm_tools/src/esm_runscripts/prepexp.py", line 28, in run_job
    evaluate(config, "prepexp", "prepexp_recipe")
  File "/home/a/a270105/esm_tools/src/esm_runscripts/helpers.py", line 71, in evaluate
    config = esm_plugin_manager.work_through_recipe(
  File "/home/a/a270105/esm_tools/src/esm_plugin_manager/esm_plugin_manager.py", line 159, in work_through_recipe
    config = getattr(submodule, workitem)(config)
  File "/home/a/a270105/esm_tools/src/esm_runscripts/prepexp.py", line 261, in call_esm_runscripts_from_prepexp
    _call_esm_runscripts_internally(config, new_command, scriptsdir)
  File "/home/a/a270105/esm_tools/src/esm_runscripts/prepexp.py", line 194, in _call_esm_runscripts_internally
    subprocess.check_call(command.split(), cwd=exedir)
  File "/sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['esm_runscripts', 'PI_2.6_2p1z1d_tp.yaml', '-e', 'PI_2.6_2p1z1d_tp', '--open-run', '-c', '--no-motd', '--last-jobtype', 'prepcompute', '-t', 'prepcompute']' returned non-zero exit status 1.
[a270105@levante6 esm_runscripts]$ ls -l /home/a/a270105/esm_tools/configs/runscript
ls: cannot access '/home/a/a270105/esm_tools/configs/runscript': No such file or directory

What do these messages mean? Is this related to the problem with -fPIC?

System (please complete the following information):

• Supercomputer: levante
• Version [6.58.0]

[JanStreffing]

Any chance we can tackle this issue in time for the FESOM 2.7 release? More or less today or tomorrow?

[mandresm]

yes! @a270105, I might need to ask some things to you in an hour or so

[chrisdane]

For the fpic problem, it seems that

cd configs # 79333a18 on branch awiesm-2.6-recom
grep -ir cflags
...
components/oasis3mct/oasis3mct.env.yaml:                OASIS_CFLAGS: '"-fPIC"'
components/oasis3mct/oasis3mct.env.yaml:                CFLAGS: '"-fPIC"'
components/oasis3mct/oasis3mct.env.yaml:                CCFLAGS: '"-fPIC"'
components/oasis3mct/oasis3mct.env.yaml:                FCFLAGS: '"-fPIC"'
...

the *CFLAGS never make it into any build cmd (so confirmed for install-awiesm-2.6-recom and also for install-awiesm-2.6). Is that intended?

[mandresm]

@a270105, I am investigating the first problem, but I'm pretty sure the second is unrelated. But to be sure, I need the path to your runscript. Can you share it?

[a270105]

You can find the runscript: /home/a/a270105/esm_runscripts/PI_2.6_2p1z1d_tp.yaml

[mandresm]

After looking at it with @a270105, the problem number 2 was in the runscript, inside the namelist_changes:

        namelist.config:
            paths:
                ClimateDataPath: "/home/a/a270105/initial_files/pi_init/"
                ForcingDataPath: remove_from_namelist
                OpbndPath: remove_from_namelist
                TideForcingPath: remove_from_namelist
                run_config:
                    num_fesom_groups: 1

run_config was included inside the paths and ESM-Tools was trying to check whether num_fesom_groups: 1 (which at that point was a namelist object) existed as a path, and throwing an error type error.

I'll add an error handling in the _check_fesom_missing_files function, so that the next time this happens the error is clear.

[mandresm]

This is where the problem 1 is coming from:

esm_tools/configs/setups/awiesm/awiesm.yaml

Line 521 in a6af359

include_env_from_component_files: False

That parameter, if set to false, ignores any environment definition defined in one of the component files (as described here).

Why do we have this here?

I think it is here because those flags do not apply to all versions of the AWIESM setup, and when I reworked the choose_blocks + environment, non of the versions of AWIESM tested required these flags, so I added this line, to reproduce the exact old environment. Can someone clarify under which versions of oasis/fesom these are needed? Then I can put this parameter under a choose_block, and the problem would be solved

[a270105]

I adapted Miguel's fix for problem 1 in my esm-tools branch and confirm that awiesm-2.6-recom can be installed now.

[mandresm]

Luckily @a270105 is unblocked now, but this issue cannot be truly solved until the following question is answered:

Can someone clarify under which versions of oasis/fesom these are needed?

Then we can take action and fix awiesm.yaml, either remove this line for all versions, of this means problems for earlier versions of AWIESM, we would need a choose block.